[00:00:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.43, 11.68, 10.83 [00:01:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.86, 8.25, 8.01 [00:01:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.77, 4.51, 3.48 [00:01:42] [02miraheze/MirahezeMagic] 07The-Voidwalker pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/c0c78a125d31...ce8a79900c8e [00:01:43] [02miraheze/MirahezeMagic] 07The-Voidwalker 03ce8a799 - remove option -C from tar command [00:01:45] [02MirahezeMagic] 07The-Voidwalker synchronize pull request 03#419: add script to download/upload swift containers for DataDump compatibi… - 13https://github.com/miraheze/MirahezeMagic/pull/419 [00:03:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.74, 8.00, 7.96 [00:04:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.81, 12.73, 11.17 [00:05:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.27, 8.28, 8.05 [00:05:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.48, 3.89, 3.50 [00:07:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.56, 7.89, 7.93 [00:07:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.67, 3.37, 3.35 [00:08:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.58, 11.70, 11.33 [00:10:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.47, 13.02, 11.84 [00:11:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.34, 11.97, 11.64 [00:11:54] miraheze/MirahezeMagic - The-Voidwalker the build passed. [00:13:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.64, 7.79, 7.80 [00:13:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.59, 12.03, 11.67 [00:14:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 11.95, 11.76 [00:15:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.07, 11.52, 11.55 [00:17:52] [02MirahezeMagic] 07The-Voidwalker closed pull request 03#419: add script to download/upload swift containers for DataDump compatibi… - 13https://github.com/miraheze/MirahezeMagic/pull/419 [00:17:56] [02miraheze/MirahezeMagic] 07The-Voidwalker pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/MirahezeMagic/compare/774f72970c7a...eda7502c8380 [00:17:57] [02miraheze/MirahezeMagic] 07Reception123 03eda7502 - add script to download/upload swift containers for DataDump compatibi… (#419) [00:17:58] [02miraheze/MirahezeMagic] 07The-Voidwalker deleted branch 03Reception123-patch-1 [00:18:01] [02MirahezeMagic] 07The-Voidwalker deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/MirahezeMagic [00:18:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.73, 12.33, 11.85 [00:19:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.10, 12.85, 12.03 [00:19:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.10, 5.07, 3.88 [00:23:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 11.80, 11.86 [00:24:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.15, 10.98, 11.54 [00:24:50] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user=Paperforth (END - exit=0) [00:24:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.46, 3.69, 3.73 [00:26:14] miraheze/MirahezeMagic - The-Voidwalker the build passed. [00:27:24] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.90, 3.45, 3.33 [00:29:14] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/5eb484ce4be9...28fe9796647e [00:29:17] [02miraheze/mediawiki] 07The-Voidwalker 0328fe979 - update MirahezeMagic [00:29:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.37, 2.68, 3.06 [00:29:27] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.24, 3.25, 3.49 [00:31:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.94, 2.64, 3.24 [00:32:27] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.21445328 secs [00:33:18] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.19, 3.93, 2.77 [00:33:42] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.94, 9.35, 5.99 [00:37:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.86, 3.45, 3.13 [00:37:25] [02mw-config] 07The-Voidwalker closed pull request 03#5247: use new swiftDump.php script - 13https://github.com/miraheze/mw-config/pull/5247 [00:37:27] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/29c32b8df444...851f354ef0eb [00:37:30] [02miraheze/mw-config] 07The-Voidwalker 03851f354 - use new swiftDump.php script (#5247) [00:37:32] [02miraheze/mw-config] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [00:37:34] [02mw-config] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/mw-config [00:38:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.32, 11.18, 11.02 [00:38:27] miraheze/mw-config - The-Voidwalker the build passed. [00:38:31] !log [void@mwtask141] starting deploy of {'config': True, 'world': True, 'extensionlist': True} to all [00:38:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.86, 11.61, 11.20 [00:40:38] [02mw-config] 07brandon-wm closed pull request 03#5246: Remove CosmicAlpha and Agent Isai - 13https://github.com/miraheze/mw-config/pull/5246 [00:41:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 11.28, 11.11 [00:41:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.27, 2.61, 2.89 [00:43:07] !log [@mwtask141] starting deploy of {'config': True} to all [00:43:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:15] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [00:43:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:29] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.44, 6.70, 6.52 [00:43:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.73, 7.36, 7.93 [00:44:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 10.54, 10.92 [00:44:59] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.07, 3.34, 3.24 [00:47:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 10.75, 10.79 [00:47:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.62, 8.22, 8.11 [00:48:46] !log [void@mwtask141] finished deploy of {'config': True, 'world': True, 'extensionlist': True} to all - SUCCESS in 615s [00:49:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.24, 11.50, 11.09 [00:52:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 10.87, 10.91 [00:52:27] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.02145147324 secs [00:55:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 11.37, 11.46 [00:57:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.12, 11.83, 11.62 [00:57:14] !log [@test131] starting deploy of {'config': True} to all [00:57:15] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [00:57:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:57:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.57, 11.22, 11.41 [00:59:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.61, 7.47, 8.00 [01:01:54] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/851f354ef0eb...4708e1d9765c [01:01:55] [02miraheze/mw-config] 07The-Voidwalker 034708e1d - allow generation of image dumps [01:02:12] !log [void@mwtask141] starting deploy of {'config': True} to all [01:02:22] !log [void@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [01:02:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.88, 13.36, 11.76 [01:02:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:49] miraheze/mw-config - The-Voidwalker the build passed. [01:03:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.05, 12.75, 11.85 [01:03:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.01, 8.11, 8.10 [01:05:19] !log [void@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [01:05:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:31] !log [void@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [01:05:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.37, 11.83, 11.55 [01:06:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.91, 3.51, 2.88 [01:07:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.62, 11.58, 11.65 [01:08:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.49, 12.23, 11.73 [01:09:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.26, 7.39, 7.86 [01:10:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.21, 3.43, 3.06 [01:11:57] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=tali64wiki (END - exit=0) [01:12:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.22, 3.11, 2.99 [01:15:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 11.60, 11.51 [01:16:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.18, 11.56, 11.87 [01:17:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 10.93, 11.28 [01:18:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.72, 12.30, 12.10 [01:19:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.12, 7.91, 7.77 [01:22:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.16, 10.95, 11.67 [01:26:45] !log [@test131] starting deploy of {'config': True} to all [01:26:47] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [01:26:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.32, 12.15, 11.84 [01:30:18] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:30:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.99, 11.88, 11.78 [01:35:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.08, 10.86, 10.74 [01:36:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.39, 3.38, 2.80 [01:37:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.02, 10.41, 10.58 [01:38:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.05, 3.17, 2.79 [01:39:16] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.38, 3.29, 2.57 [01:41:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.52, 3.51, 2.75 [01:42:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.17, 12.87, 11.85 [01:43:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.59, 3.31, 2.77 [01:47:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.63, 4.76, 3.39 [01:47:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.26, 12.33, 11.22 [01:49:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.47, 7.15, 7.85 [01:51:39] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.41, 8.74, 6.78 [01:51:42] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:43] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 13.81, 10.21, 7.29 [01:52:55] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 15.78, 11.10, 7.94 [01:53:39] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.56, 10.51, 7.65 [01:54:43] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 11.90, 10.87, 7.90 [01:55:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.10, 3.77, 3.52 [01:55:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.87, 7.67, 7.80 [01:56:39] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 2 backends are down. mw121 mw122 [01:56:43] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 17.07, 13.31, 9.16 [01:57:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.94, 7.30, 7.64 [01:57:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.403 second response time [01:58:04] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:34] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 14.54, 13.59, 9.59 [01:58:41] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [01:59:29] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:33] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:56] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:00:01] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.56, 5.12, 4.13 [02:02:21] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:02:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.52, 3.70, 3.73 [02:03:57] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.405 second response time [02:04:19] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.244 second response time [02:04:22] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 3 backends are down. mw121 mw122 mw142 [02:04:39] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 3 backends are down. mw121 mw122 mw131 [02:04:41] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 3 backends are down. mw121 mw132 mw141 [02:04:44] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 3 backends are down. mw121 mw132 mw141 [02:05:28] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.485 second response time [02:05:30] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.222 second response time [02:05:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.04, 8.02, 7.67 [02:06:09] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.268 second response time [02:06:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.483 second response time [02:06:51] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.23, 2.67, 3.33 [02:08:22] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [02:08:39] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [02:08:41] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [02:08:43] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 4.34, 10.48, 11.15 [02:08:44] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [02:08:55] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.96, 10.11, 11.08 [02:09:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.12, 7.61, 7.60 [02:09:39] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 6.17, 11.30, 11.72 [02:10:11] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.64, 10.98, 11.44 [02:12:55] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.27, 8.28, 10.08 [02:14:43] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 7.60, 8.32, 9.89 [02:15:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.03, 7.78, 7.65 [02:15:59] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.66, 8.47, 10.14 [02:17:39] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.41, 9.36, 10.38 [02:19:39] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.53, 8.93, 10.14 [02:20:00] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:59] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.411 second response time [02:25:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.78, 7.49, 7.81 [02:30:15] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:30] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:33:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.51, 7.52, 7.54 [02:36:23] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.575 second response time [02:37:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.53, 7.79, 7.67 [02:43:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.41, 7.75, 7.61 [02:45:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.66, 7.68, 7.59 [02:46:12] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 10.90, 4.25, 1.93 [02:47:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.86, 8.24, 7.79 [02:48:11] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.66, 3.42, 1.92 [02:50:11] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.17, 2.56, 1.78 [02:51:25] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [02:54:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.29, 4.61, 3.00 [02:56:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.89, 3.70, 2.86 [02:58:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.79, 3.21, 2.77 [02:59:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.25, 7.93, 7.99 [02:59:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.23, 3.32, 2.25 [03:01:08] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:01:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.19, 2.53, 2.09 [03:01:42] RECOVERY - db142 Backups SQL on db142 is OK: FILE_AGE OK: /var/log/sql-backup.log is 100 seconds old and 275 bytes [03:03:28] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=abungawiki --no-updates --username-prefix=w /home/agent/ImportDump/abungawiki-20230606075825.xml (START) [03:03:30] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=abungawiki --no-updates --username-prefix=w /home/agent/ImportDump/abungawiki-20230606075825.xml (END - exit=0) [03:03:31] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=abungawiki (START) [03:03:33] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=abungawiki (END - exit=0) [03:03:34] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=abungawiki --active --update (END - exit=0) [03:03:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:01] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=pitthrsswiki --no-updates --username-prefix=mh:patmos /home/agent/ImportDump/pitthrsswiki-20230606035547.xml (START) [03:04:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:11] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=pitthrsswiki --no-updates --username-prefix=mh:patmos /home/agent/ImportDump/pitthrsswiki-20230605235939.xml (START) [03:04:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.45, 4.60, 3.45 [03:04:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.29, 6.19, 4.19 [03:04:44] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=pitthrsswiki --no-updates --username-prefix=mh:patmos /home/agent/ImportDump/pitthrsswiki-20230606035547.xml (END - exit=0) [03:04:45] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=pitthrsswiki --no-updates --username-prefix=mh:patmos /home/agent/ImportDump/pitthrsswiki-20230605235939.xml (END - exit=0) [03:04:46] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=pitthrsswiki (START) [03:04:47] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=pitthrsswiki (START) [03:04:48] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=pitthrsswiki (END - exit=256) [03:04:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:49] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=pitthrsswiki --active --update (END - exit=0) [03:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.00, 3.94, 3.35 [03:08:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.79, 3.24, 3.42 [03:08:48] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=pitthrsswiki (END - exit=0) [03:08:49] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=pitthrsswiki --active --update (END - exit=0) [03:08:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.91, 3.09, 3.15 [03:13:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.66, 7.60, 7.38 [03:14:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.87, 2.56, 3.15 [03:15:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.65, 10.32, 11.69 [03:17:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.55, 7.67, 7.46 [03:19:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.34, 11.89, 11.94 [03:19:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.49, 8.13, 7.66 [03:20:42] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:25:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.30, 7.78, 7.67 [03:29:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.45, 7.82, 7.70 [03:31:41] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.84, 10.98, 8.85 [03:32:49] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.85, 7.67, 7.70 [03:33:36] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.53, 9.79, 8.67 [03:34:46] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.136 second response time [03:39:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.25, 7.56, 7.62 [03:41:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.17, 7.00, 7.41 [03:45:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.03, 8.01, 7.67 [03:47:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.16, 7.58, 7.55 [03:49:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.98, 8.15, 7.74 [03:49:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.84, 4.02, 3.14 [03:49:57] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:50:27] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2009858191 secs [03:51:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.25, 3.40, 3.02 [03:55:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.52, 7.93, 7.85 [04:02:20] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:02:27] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.004225045443 secs [04:05:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.03, 7.47, 7.61 [04:07:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.13, 7.29, 7.52 [04:15:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.26, 10.32, 11.55 [04:18:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.82, 11.07, 11.95 [04:20:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.11, 11.90, 12.15 [04:21:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.95, 11.04, 11.37 [04:21:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.45, 7.50, 7.32 [04:22:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.01, 11.46, 11.98 [04:23:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.07, 10.37, 11.10 [04:23:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.69, 7.59, 7.38 [04:29:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.96, 8.92, 10.11 [04:32:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.99, 8.28, 9.87 [04:35:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.81, 12.18, 11.09 [04:35:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.82, 7.84, 7.47 [04:36:53] PROBLEM - mw141 APT on mw141 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates). [04:37:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.33, 11.45, 10.95 [04:37:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.66, 7.76, 7.49 [04:37:40] PROBLEM - mw132 APT on mw132 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [04:39:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.05, 8.09, 7.63 [04:41:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.21, 10.94, 10.55 [04:41:32] PROBLEM - mwtask141 APT on mwtask141 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (1 critical updates). [04:42:15] PROBLEM - mw131 APT on mw131 is CRITICAL: APT CRITICAL: 54 packages available for upgrade (1 critical updates). [04:42:52] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [04:43:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.59, 11.51, 11.04 [04:45:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.16, 10.61, 10.77 [04:45:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.79, 7.95, 7.73 [04:45:34] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ac570cbb2d06...5a3281ed8bd1 [04:45:36] [02miraheze/ssl] 07MacFan4000 035a3281e - remove miraheze.ga redirect [04:45:47] PROBLEM - mail121 APT on mail121 is CRITICAL: APT CRITICAL: 51 packages available for upgrade (1 critical updates). [04:47:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 7.65, 8.03, 7.80 [04:51:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 11.69, 11.13 [04:51:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.55, 7.96, 7.85 [04:53:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.28, 10.95, 10.91 [04:54:02] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.68, 8.66, 5.91 [04:55:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.44, 7.70, 7.74 [04:57:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.52, 7.46, 7.64 [04:57:45] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:57:47] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:59:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.47, 11.51, 10.94 [05:00:27] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [05:01:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.03, 11.00, 10.83 [05:01:35] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [05:02:30] PROBLEM - mw142 APT on mw142 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [05:03:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.96, 11.30, 10.93 [05:03:20] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:04:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.14, 12.47, 11.27 [05:05:24] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 11.47, 8.63, 8.00 [05:05:50] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:06:39] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [05:06:42] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:06:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.72, 11.70, 11.19 [05:08:01] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:10:06] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:10:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.28, 10.90, 11.06 [05:10:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.54, 11.99, 11.40 [05:11:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.51, 7.88, 7.87 [05:12:24] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:12:41] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:12:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.95, 11.84, 11.41 [05:13:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.15, 8.05, 7.93 [05:14:22] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.79, 2.92, 2.03 [05:14:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.41, 11.89, 11.47 [05:15:54] [02mw-config] 07brandon-wm opened pull request 03#5248: rm Agent Isai from MirahezeStaffAccessIDs per resignation - 13https://github.com/miraheze/mw-config/pull/5248 [05:15:58] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:16:20] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.21, 2.25, 1.89 [05:16:22] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:16:52] miraheze/mw-config - brandon-wm the build passed. [05:17:56] [02mw-config] 07brandon-wm edited pull request 03#5248: rm Agent Isai from MirahezeStaffAccessIDs per resignation - 13https://github.com/miraheze/mw-config/pull/5248 [05:18:08] [02mw-config] 07brandon-wm edited pull request 03#5248: rm Agent Isai and Universal Omega from MirahezeStaffAccessIDs per resignations - 13https://github.com/miraheze/mw-config/pull/5248 [05:18:19] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:18:34] I haven't left yet, BrandonWM lol [05:18:46] neither has UO [05:18:58] [02mw-config] 07brandon-wm synchronize pull request 03#5248: rm Agent Isai and Universal Omega from MirahezeStaffAccessIDs per resignations - 13https://github.com/miraheze/mw-config/pull/5248 [05:19:13] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [05:19:54] miraheze/mw-config - brandon-wm the build passed. [05:20:17] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [05:20:22] Agent: yeah I know haha, if you see the PR description you'll see it says to not merge until you've officially resigned [05:20:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.07, 11.12, 11.33 [05:21:10] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [05:22:11] I don't know it's necessary for you to start access removal PRs, BrandonWM. SRE is fully capable of doing that, and can just push the commit directly :) [05:23:04] maybe, not sure, this is just how it's always been done as far as I know [05:25:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.59, 7.45, 7.84 [05:25:34] Not really. Reception123 basically has a checklist of workflow tasks to work through, and likely has templates for such things. He's very likely not going to be looking for open PRs, so in any event, your outstanding PR will just be closed as a duplicate. I mean, it's not the end of the world, but it's time wasting basically [05:28:15] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [05:28:59] Indeed, I'm on a break from non-urgent tasks at the moment so unfortunately I won't be for the time being. [05:29:33] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.55, 3.21, 2.15 [05:31:31] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.13, 2.86, 2.15 [05:33:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.20, 7.58, 7.65 [05:33:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.13, 10.94, 10.82 [05:33:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.54, 6.60, 7.96 [05:34:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.87, 9.04, 10.15 [05:35:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.18, 11.49, 11.07 [05:37:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.51, 7.69, 7.70 [05:41:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.37, 7.78, 7.70 [05:41:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 7.47, 7.96 [05:41:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.45, 4.44, 6.46 [05:43:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.54, 8.21, 8.15 [05:49:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.09, 10.82, 10.61 [05:51:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.59, 9.83, 10.29 [05:51:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.11, 10.39, 10.27 [05:53:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.96, 11.00, 10.66 [05:55:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.30, 10.21, 10.41 [05:57:52] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:01:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.98, 7.60, 7.91 [06:01:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.13, 10.65, 10.36 [06:02:37] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.56, 7.15, 6.30 [06:03:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.25, 11.85, 10.99 [06:03:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.92, 10.59, 10.40 [06:04:36] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.47, 6.66, 6.20 [06:05:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.50, 11.38, 10.72 [06:07:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.94, 7.48, 7.63 [06:09:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.86, 11.22, 10.80 [06:09:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 11.18, 11.11 [06:11:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.48, 11.73, 10.99 [06:12:30] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.66, 7.23, 6.65 [06:16:27] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.06, 6.52, 6.60 [06:19:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.71, 11.64, 11.08 [06:21:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.05, 7.95, 7.93 [06:21:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.70, 11.23, 10.98 [06:23:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.43, 11.45, 11.43 [06:23:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.00, 8.10, 8.00 [06:24:21] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 13.91, 9.38, 7.53 [06:28:37] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.82, 3.93, 2.60 [06:29:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.08, 7.71, 7.88 [06:29:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 11.43, 11.11 [06:29:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:31:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.82, 11.40, 11.16 [06:32:33] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.42, 2.85, 2.47 [06:33:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.97, 12.98, 11.76 [06:36:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.59, 11.84, 11.49 [06:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 11.97, 11.70 [06:38:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.40, 11.84, 11.56 [06:39:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.17, 13.19, 12.17 [06:40:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.59, 12.03, 11.63 [06:43:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.50, 7.68, 7.69 [06:46:08] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.54, 6.98, 7.84 [06:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.83, 7.49, 7.64 [06:49:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.17, 7.81, 7.74 [06:52:18] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 11.35, 6.63, 3.56 [06:53:20] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:54:03] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.54, 7.11, 7.36 [06:55:11] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:20] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [06:55:26] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:02] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.87, 5.81, 6.85 [06:57:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.367 second response time [06:57:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.481 second response time [07:00:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.38, 5.79, 6.65 [07:00:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.62, 3.30, 3.24 [07:03:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.82, 7.60, 7.94 [07:05:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.04, 7.97, 8.02 [07:06:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.09, 10.08, 11.84 [07:07:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.65, 7.93, 8.00 [07:13:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.18, 10.38, 11.79 [07:25:06] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:26:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.73, 3.91, 2.94 [07:27:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.42, 10.76, 10.85 [07:28:36] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:29:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.42, 10.71, 10.84 [07:30:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.49, 9.39, 9.99 [07:30:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.51, 3.31, 2.93 [07:32:56] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [07:33:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.19, 10.81, 10.77 [07:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.77, 10.34, 10.71 [07:41:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.89, 9.03, 10.08 [07:42:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.75, 11.26, 10.39 [07:43:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.07, 6.88, 7.89 [07:46:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.45, 10.16, 10.17 [07:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.30, 7.26, 7.20 [07:47:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.28, 7.81, 8.02 [07:51:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.60, 7.53, 7.38 [07:59:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.31, 6.49, 7.73 [08:03:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.53, 11.05, 10.30 [08:03:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.68, 8.44, 7.54 [08:03:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.95, 8.86, 8.37 [08:05:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.56, 11.14, 10.27 [08:07:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.62, 12.12, 10.81 [08:07:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.72, 11.06, 10.36 [08:08:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.73, 3.27, 2.18 [08:10:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.12, 2.74, 2.12 [08:17:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.71, 7.93, 7.94 [08:17:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.51, 11.15, 10.58 [08:19:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.92, 8.24, 8.05 [08:19:44] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [08:21:43] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [08:23:44] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 18.56, 11.90, 8.19 [08:24:16] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 18.34, 11.54, 7.77 [08:25:41] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.011 second response time [08:25:55] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 18.39, 12.87, 8.31 [08:25:58] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 19.25, 14.26, 9.52 [08:26:00] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.011 second response time [08:27:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.44, 7.60, 7.96 [08:27:45] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.117 second response time [08:29:49] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 11.87, 11.65, 8.86 [08:31:58] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.27, 10.27, 9.47 [08:32:06] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.686 second response time [08:32:16] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 5.31, 10.25, 9.23 [08:33:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.19, 6.84, 7.35 [08:33:43] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 7.97, 9.44, 8.60 [08:33:44] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 8.29, 10.80, 10.03 [08:33:47] @Orduin thank you so much, I could log in again ❤️ [08:33:58] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.71, 9.41, 9.22 [08:34:16] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.78, 9.08, 8.92 [08:35:44] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.87, 9.66, 9.73 [08:39:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.27, 7.50, 7.54 [08:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.33, 7.84, 7.60 [08:55:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.80, 9.77, 11.71 [08:58:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.79, 9.07, 11.62 [08:59:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.24, 3.06, 2.37 [08:59:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.80, 7.58, 7.74 [08:59:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.01, 6.05, 3.99 [09:01:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.00, 8.25, 7.95 [09:03:13] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.75, 3.03, 2.59 [09:03:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.80, 8.00, 7.90 [09:04:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.19, 9.71, 10.95 [09:05:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.55, 9.83, 10.55 [09:06:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.33, 10.38, 11.05 [09:07:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.43, 7.98, 7.89 [09:07:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.08, 9.14, 10.21 [09:07:38] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.50, 2.68, 1.43 [09:09:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.72, 7.93, 7.89 [09:09:37] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.02, 3.49, 1.87 [09:10:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.64, 11.31, 11.19 [09:11:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.89, 7.23, 6.41 [09:13:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.60, 9.35, 10.01 [09:13:37] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.15, 2.81, 2.02 [09:13:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.58, 5.31, 5.81 [09:14:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.49, 11.16, 11.21 [09:16:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.25, 11.97, 11.47 [09:16:21] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.88, 4.63, 2.65 [09:18:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.34, 11.41, 11.31 [09:18:20] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.23, 3.35, 2.43 [09:19:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.58, 8.74, 8.12 [09:20:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.32, 11.61, 11.38 [09:23:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.02, 7.92, 7.93 [09:25:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.93, 10.59, 10.22 [09:28:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.20, 11.54, 11.48 [09:30:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.10, 12.29, 11.75 [09:31:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.11, 12.10, 10.86 [09:31:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.48, 8.32, 8.03 [09:37:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.38, 7.97, 7.98 [09:38:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 11.60, 11.28 [09:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.37, 7.56, 7.61 [09:49:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.99, 7.46, 7.57 [09:51:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 11.15, 9.11, 8.19 [09:57:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.61, 10.79, 10.64 [10:15:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.03, 7.42, 7.92 [10:19:26] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:21:36] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [10:23:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.32, 7.69, 7.81 [10:23:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.34, 10.97, 11.98 [10:25:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.92, 7.78, 7.84 [10:27:51] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.76, 3.05, 2.42 [10:29:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.37, 8.04, 7.92 [10:29:50] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.88, 2.73, 2.39 [10:31:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 8.00, 7.96, 7.90 [10:33:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.43, 8.50, 8.10 [10:33:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.20, 13.23, 12.38 [10:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.14, 7.72, 7.99 [10:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.65, 7.98, 8.04 [10:48:47] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [10:50:45] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [11:01:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.45, 7.06, 7.84 [11:05:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.61, 7.47, 7.78 [11:18:46] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [11:20:45] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [11:43:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.13, 6.17, 3.44 [11:49:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.11, 5.60, 4.27 [11:54:46] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:00:29] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [12:12:20] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:17] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.891 second response time [12:15:36] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.50, 3.78, 2.24 [12:17:35] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.03, 2.74, 2.04 [12:45:49] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 1.79, 1.26, 0.75 [12:47:49] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.47, 0.93, 0.69 [13:02:01] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:03:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:47] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.155 second response time [13:09:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.04, 6.68, 3.56 [13:11:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.46, 5.02, 3.34 [13:19:28] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [13:21:29] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [13:22:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:24:30] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.04, 3.31, 2.57 [13:29:36] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [13:31:17] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.94, 2.88, 2.51 [13:32:12] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:44:18] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:12] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 3.59, 4.89, 2.93 [13:46:20] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.534 second response time [13:48:11] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.17, 3.52, 2.66 [13:50:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:50:11] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.80, 2.60, 2.42 [13:58:50] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.49, 4.96, 3.15 [13:59:13] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:01:11] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [14:01:34] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:02:47] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.25, 3.24, 2.90 [14:10:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:16:04] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.78, 3.70, 2.74 [14:18:03] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.25, 2.76, 2.51 [14:24:08] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'deadprem.wiki' expires in 15 day(s) (Fri 23 Jun 2023 14:09:38 GMT +0000). [14:26:16] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:29:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a3281ed8bd1...a887cbb30fcc [14:29:26] [02miraheze/ssl] 07MirahezeSSLBot 03a887cbb - Bot: Update SSL cert for deadprem.wiki [14:30:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.73, 4.04, 2.92 [14:31:30] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [14:31:40] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - vedopedia.witches-empire.com All nameservers failed to answer the query. [14:32:18] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.28, 3.38, 2.83 [14:33:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.42, 5.53, 2.83 [14:35:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.64, 5.26, 3.08 [14:37:27] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'deadprem.wiki' expires in 15 day(s) (Fri 23 Jun 2023 14:09:38 GMT +0000). [14:41:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:44:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:00] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.15, 4.41, 2.88 [14:46:38] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.281 second response time [14:48:00] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.62, 3.27, 2.64 [14:48:26] PROBLEM - wiki.arsrobotics.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.arsrobotics.org' expires in 15 day(s) (Fri 23 Jun 2023 14:44:32 GMT +0000). [14:48:40] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.86, 3.54, 2.95 [14:51:30] PROBLEM - hkrail.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hkrail.wiki' expires in 15 day(s) (Fri 23 Jun 2023 14:40:41 GMT +0000). [14:53:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a887cbb30fcc...8603ba807825 [14:53:31] [02miraheze/ssl] 07MirahezeSSLBot 038603ba8 - Bot: Update SSL cert for wiki.arsrobotics.org [14:55:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8603ba807825...7b95bcdb2939 [14:55:23] [02miraheze/ssl] 07MirahezeSSLBot 037b95bcd - Bot: Update SSL cert for hkrail.wiki [14:56:32] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.04, 3.29, 3.08 [14:58:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.47, 3.65, 2.63 [15:00:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.28, 2.97, 2.51 [15:01:16] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: SSL OK - vedopedia.witches-empire.com reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:04:10] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:06:21] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.47, 3.22, 3.09 [15:07:16] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Tue 05 Sep 2023 13:29:06 GMT +0000. [15:08:19] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.78, 3.12, 3.07 [15:11:43] PROBLEM - kombinat.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kombinat.eu.org' expires in 15 day(s) (Fri 23 Jun 2023 14:48:08 GMT +0000). [15:13:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7b95bcdb2939...e7c578245deb [15:13:07] [02miraheze/ssl] 07MirahezeSSLBot 03e7c5782 - Bot: Update SSL cert for kombinat.eu.org [15:18:14] RECOVERY - wiki.arsrobotics.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.arsrobotics.org' will expire on Tue 05 Sep 2023 13:53:20 GMT +0000. [15:20:28] RECOVERY - hkrail.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hkrail.wiki' will expire on Tue 05 Sep 2023 13:55:12 GMT +0000. [15:23:25] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Tue 05 Sep 2023 13:29:06 GMT +0000. [15:24:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.63, 3.46, 2.70 [15:26:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:28:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.07, 2.73, 2.63 [15:32:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:32] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:34:17] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.574 second response time [15:41:04] RECOVERY - kombinat.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kombinat.eu.org' will expire on Tue 05 Sep 2023 14:12:55 GMT +0000. [15:42:43] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:47] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.190 second response time [15:46:01] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.26, 3.42, 2.14 [15:48:00] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.39, 3.18, 2.22 [15:49:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:10] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.354 second response time [15:56:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:01:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.53, 7.54, 4.32 [16:03:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.27, 7.53, 4.75 [16:05:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.34, 5.68, 4.39 [16:17:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:18:57] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:11] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [16:21:01] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.646 second response time [16:21:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.634 second response time [16:21:12] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [16:40:39] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:38] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.100 second response time [16:43:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.53, 6.97, 8.00 [16:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.21, 7.49, 8.06 [16:47:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 4.00, 3.36, 2.67 [16:47:12] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.73, 7.53, 8.00 [16:48:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.27, 3.14, 2.68 [16:49:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.61, 8.16, 8.16 [16:55:49] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 2.13, 1.37, 0.78 [16:56:24] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:57:50] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 1.60, 1.44, 0.88 [16:58:32] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [16:59:34] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2085563838 secs [17:03:42] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.958 second response time [17:16:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:24:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.43, 7.10, 3.68 [17:29:34] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.02530229092 secs [17:29:56] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.97, 5.81, 4.49 [17:34:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.95, 3.09, 2.52 [17:36:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.23, 3.44, 2.70 [17:36:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.01, 3.23, 2.65 [17:38:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 2.87, 2.58 [17:42:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:52] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.341 second response time [17:46:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.32, 3.56, 2.98 [17:46:40] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.82, 3.42, 2.79 [17:47:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:48:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.70, 2.91, 2.82 [17:48:40] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 3.04, 2.73 [17:49:38] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:51:36] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [17:53:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:26] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.810 second response time [18:29:34] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1727108359 secs [18:34:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.99, 4.17, 3.24 [18:36:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.17, 3.85, 3.25 [18:40:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.08, 2.67, 2.93 [18:45:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.98, 3.69, 2.87 [18:45:37] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 30.68, 12.74, 5.38 [18:47:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 2.91, 2.68 [18:49:05] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.65, 5.70, 4.20 [18:49:35] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.19, 3.31, 2.22 [18:51:04] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.98, 5.45, 4.28 [18:51:35] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.89, 3.15, 2.30 [18:53:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.27, 4.60, 3.30 [18:55:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.29, 3.63, 3.10 [18:55:35] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.91, 2.83, 3.64 [18:57:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.22, 3.20, 3.01 [18:57:34] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.84, 2.16, 3.29 [19:01:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.48, 3.81, 3.39 [19:01:34] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07016190886 secs [19:05:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.03, 3.36, 3.28 [19:07:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.25, 2.92, 3.13 [19:07:34] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2047596574 secs [19:11:52] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.betaheze.org' expires in 15 day(s) (Fri 23 Jun 2023 18:56:37 GMT +0000). [19:15:30] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.10, 3.85, 2.79 [19:17:29] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.51, 2.91, 2.58 [19:26:17] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.95, 3.56, 3.06 [19:28:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.12, 2.92, 2.88 [19:31:34] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.03413000703 secs [19:35:24] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.54, 3.89, 2.99 [19:37:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:39:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 2.88, 2.81 [19:42:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:13] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.493 second response time [19:49:02] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:51:01] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [19:52:56] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [19:54:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.18, 3.43, 2.66 [19:56:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.43, 3.10, 2.63 [20:01:22] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [20:15:17] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.77, 2.88, 1.76 [20:17:16] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.45, 2.94, 1.94 [20:18:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.81, 3.85, 3.09 [20:19:36] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.18, 3.14, 1.85 [20:19:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.48, 8.42, 5.01 [20:20:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.60, 3.77, 3.15 [20:20:56] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:21:35] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.62, 2.99, 1.95 [20:21:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.87, 7.67, 5.18 [20:23:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:23:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.49, 6.16, 4.93 [20:24:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.80, 4.02, 3.39 [20:25:04] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.46, 3.61, 2.86 [20:26:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.40, 3.67, 3.35 [20:28:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.39, 3.36, 3.28 [20:29:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.39, 2.84, 2.75 [20:30:43] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [20:31:02] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:43:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:47:11] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.16, 4.62, 2.66 [20:47:50] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:48:34] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1408532262 secs [20:49:11] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.19, 3.77, 2.60 [20:49:48] RECOVERY - cp22 APT on cp22 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [20:50:58] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.91, 4.70, 3.55 [20:51:10] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.63, 2.66, 2.33 [20:56:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.82, 3.48, 3.37 [20:58:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.13, 3.22, 3.27 [21:00:24] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:01:03] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:03:33] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.010 second response time [21:03:38] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.43, 5.26, 2.98 [21:05:28] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.236 second response time [21:05:57] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [21:06:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.91, 3.51, 3.21 [21:06:54] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.09, 4.77, 3.88 [21:07:34] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.24, 3.79, 2.97 [21:08:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.53, 3.19, 3.13 [21:08:54] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.21, 3.84, 3.65 [21:09:33] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.86, 2.77, 2.69 [21:12:34] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0903865695 secs [21:12:53] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.06, 2.82, 3.28 [21:37:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.96, 6.84, 7.86 [21:39:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.95, 8.19, 8.21 [21:39:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:40:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.81, 3.63, 3.26 [21:44:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.05, 2.85, 3.02 [21:44:34] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1026194692 secs [21:46:34] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08507600427 secs [21:46:56] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.25, 4.80, 2.72 [21:47:50] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:49:48] RECOVERY - cp22 APT on cp22 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [21:50:18] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:50:55] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.12, 3.48, 2.69 [21:52:55] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.32, 2.67, 2.48 [21:55:12] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [21:56:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.32, 3.85, 3.21 [21:58:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.62, 3.37, 3.11 [22:02:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.27, 4.46, 3.61 [22:03:45] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.010 second response time [22:04:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [22:05:40] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.254 second response time [22:08:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.97, 3.96, 3.71 [22:14:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.10, 2.55, 3.21 [22:16:16] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:18:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.77, 3.43, 3.41 [22:19:51] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:20:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.20, 3.00, 3.25 [22:21:50] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [22:24:40] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:25:22] [02dns] 07MacFan4000 opened pull request 03#415: add ACME challenges for orain.org - 13https://github.com/miraheze/dns/pull/415 [22:25:56] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.010 second response time [22:26:08] paladox: @Orduin could one of you please merge https://github.com/miraheze/dns/pull/415 ? [22:27:52] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.367 second response time [22:28:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.41, 3.79, 2.47 [22:29:58] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [22:30:13] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [22:30:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.64, 3.28, 2.44 [22:33:38] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [22:35:06] [02dns] 07paladox closed pull request 03#415: add ACME challenges for orain.org - 13https://github.com/miraheze/dns/pull/415 [22:35:07] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c84204a28faa...01e254b45427 [22:35:09] [02miraheze/dns] 07MacFan4000 0301e254b - add ACME challenges for orain.org (#415) [22:35:25] I’m mobile so you will have to wait for puppet to run but I’ve merged it [22:35:34] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [22:35:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.26, 3.48, 2.94 [22:37:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.37, 2.91, 2.79 [22:45:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.06, 4.26, 3.38 [22:47:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.81, 4.08, 3.03 [22:49:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.72, 3.64, 2.99 [22:51:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.45, 3.89, 3.13 [22:51:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.69, 3.78, 3.50 [22:53:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.66, 3.29, 3.00 [22:53:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.70, 4.22, 3.69 [22:56:17] OOJSPlus seems to be a horrible mess for deployment. [22:59:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.84, 3.28, 3.51 [23:01:15] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:01:31] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.05, 4.04, 3.76 [23:02:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.99, 3.13, 2.14 [23:02:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e7c578245deb...0d07f1bc1954 [23:02:23] [02miraheze/ssl] 07MirahezeSSLBot 030d07f1b - Bot: Update SSL cert for orain.org [23:02:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.68, 4.44, 3.49 [23:03:13] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [23:03:31] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.37, 3.26, 3.50 [23:04:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.05, 2.86, 2.17 [23:05:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.84, 3.09, 3.40 [23:08:24] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [23:08:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.89, 3.64, 3.50 [23:12:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.53, 3.82, 3.58 [23:13:34] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1551918685 secs [23:14:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.09, 3.34, 3.44 [23:16:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.37, 3.13, 3.34 [23:19:34] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07320067286 secs [23:22:49] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Tue 05 Sep 2023 22:02:13 GMT +0000. [23:24:14] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [23:26:09] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.238 second response time [23:27:22] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Tue 05 Sep 2023 22:02:13 GMT +0000. [23:28:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.28, 2.66, 1.74 [23:30:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.78, 2.75, 1.90 [23:31:50] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:35:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.90, 4.81, 2.55 [23:35:53] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Tue 05 Sep 2023 22:02:13 GMT +0000. [23:36:44] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [23:37:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.53, 4.55, 2.76 [23:38:02] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [23:42:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.99, 3.55, 3.14 [23:44:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 3.48, 3.15 [23:50:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.33, 3.64, 3.43 [23:52:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.39, 3.36, 3.06 [23:52:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.22, 3.18, 3.28 [23:54:57] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.37, 2.61, 1.86 [23:55:26] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:56:00] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:56:18] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.71, 3.55, 3.21 [23:56:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.37, 3.37, 2.28 [23:58:05] RECOVERY - swiftobject122 APT on swiftobject122 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [23:58:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.03, 3.40, 3.19 [23:58:19] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates).