[00:01:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.25, 4.78, 3.54 [00:01:20] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.096 second response time [00:02:04] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [00:02:48] RECOVERY - os141 Disk Space on os141 is OK: DISK OK - free space: / 35362 MB (17% inode=99%); [00:02:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.99, 3.74, 2.91 [00:04:22] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.07, 3.66, 1.93 [00:06:44] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.68, 7.00, 3.87 [00:07:46] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.06717339158 secs [00:08:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.61, 3.29, 3.53 [00:12:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.17, 2.51 [00:12:44] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.03, 7.29, 5.30 [00:14:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.82, 3.14, 2.59 [00:14:21] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.51, 2.94, 2.74 [00:14:42] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [00:14:44] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.20, 5.54, 4.90 [00:14:54] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1261279881 secs [00:14:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.87, 3.96, 3.81 [00:14:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.15, 4.00, 3.78 [00:17:54] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:18:10] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:42] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [00:18:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.88, 2.76, 3.34 [00:20:44] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.56, 6.99, 5.47 [00:21:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 6.92, 6.47 [00:22:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.49, 3.34, 3.47 [00:23:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.62, 6.80, 6.49 [00:24:44] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.99, 7.56, 6.08 [00:26:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:26:20] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.484 second response time [00:26:44] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.75, 5.35, 5.45 [00:30:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.88, 3.82, 3.64 [00:31:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:31:33] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [00:32:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.14, 3.51, 3.54 [00:34:54] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.02205052972 secs [00:36:40] !log [www-data@test131] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [00:36:41] !log [www-data@test131] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [00:36:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.18, 2.98, 3.30 [00:36:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.58, 3.17, 3.94 [00:41:55] !log [www-data@test131] starting deploy of {'world': True} to all [00:42:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:44:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 4.00, 3.57, 3.36 [00:44:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.05, 3.74, 3.86 [00:46:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.57, 8.45, 7.22 [00:46:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.17, 3.49, 3.34 [00:46:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.51, 3.41, 3.74 [00:48:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.26, 7.48, 7.01 [00:48:31] !log [www-data@test131] finished deploy of {'world': True} to all - SUCCESS in 396s [00:48:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.20, 3.60, 3.54 [00:54:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.01, 2.91, 3.37 [00:57:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.11, 6.42, 6.78 [00:58:44] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.42, 6.30, 3.97 [01:00:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.52, 3.94, 3.65 [01:01:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.31, 6.76, 6.86 [01:02:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.12, 3.32, 3.46 [01:02:44] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.90, 5.68, 4.38 [01:03:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.66, 6.43, 6.72 [01:04:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.70, 5.35, 4.14 [01:10:24] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 3.33, 2.10, 1.19 [01:10:37] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:10:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.93, 3.46, 3.39 [01:12:05] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:20] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.78, 1.94, 1.23 [01:12:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.02, 4.44, 3.75 [01:13:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [01:14:06] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.226 second response time [01:14:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.92, 3.89, 3.63 [01:16:26] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [01:18:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.93, 3.25, 3.76 [01:20:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.92, 3.07, 3.35 [01:24:30] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.40, 2.64, 3.30 [01:24:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.12, 3.65, 3.52 [01:25:49] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.61, 1.69, 1.51 [01:26:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.27, 3.12, 3.34 [01:29:41] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.06, 1.88, 1.63 [01:32:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:32:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.34, 7.35, 6.86 [01:33:33] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.82, 1.90, 1.70 [01:35:32] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.03, 1.95, 1.74 [01:36:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.89, 6.45, 6.61 [01:37:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.54, 4.16, 3.66 [01:40:19] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:43:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.05, 3.87, 3.87 [01:43:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:44:05] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:07] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.282 second response time [01:46:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 3.21, 2.93 [01:47:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.56, 3.65, 3.72 [01:47:32] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.83, 2.00, 1.92 [01:49:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.37, 3.57, 3.68 [01:49:32] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.03, 2.01, 1.93 [01:50:23] PROBLEM - phab121 Disk Space on phab121 is WARNING: DISK WARNING - free space: / 3415 MB (7% inode=91%); [01:50:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.32, 3.00, 2.93 [01:51:43] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:51] !log [www-data@test131] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [01:51:52] !log [www-data@test131] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [01:52:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:23] RECOVERY - phab121 Disk Space on phab121 is OK: DISK OK - free space: / 13516 MB (30% inode=91%); [01:52:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:53:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.25, 3.48, 3.58 [01:53:47] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.136 second response time [01:55:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.30, 2.97, 3.37 [01:56:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.91, 5.57, 3.96 [01:57:32] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.55, 1.85, 1.92 [02:02:57] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [02:03:00] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/e01047db149a...ce4a39bc5c54 [02:03:01] [02miraheze/ManageWiki] 07AgentIsai 03ce4a39b - Do not list outside of global wiki or if you don't have the permissions [02:04:25] [02miraheze/ManageWiki] 07github-actions[bot] pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/ce4a39bc5c54...4c694cb521de [02:04:27] [02miraheze/ManageWiki] 07github-actions 034c694cb - CI: lint code to MediaWiki standards [02:04:29] [02ManageWiki] 07github-actions[bot] synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [02:05:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.88, 3.53, 3.78 [02:06:24] miraheze/ManageWiki - AgentIsai the build passed. [02:09:32] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.24, 1.94, 1.87 [02:09:55] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.010 second response time [02:10:02] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:10:19] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/bf2d6368fdbb...2ff389db70ad [02:10:22] [02miraheze/WikiDiscover] 07AgentIsai 032ff389d - Fix capitalization [02:11:32] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.89, 1.88, 1.86 [02:11:36] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.63, 2.74, 3.34 [02:13:40] miraheze/WikiDiscover - AgentIsai the build passed. [02:15:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.79, 3.57, 3.55 [02:15:39] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03closure-date [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/commit/4d5c43785ced [02:15:42] [02miraheze/WikiDiscover] 07AgentIsai 034d5c437 - Add to API [02:15:44] [02WikiDiscover] 07AgentIsai created branch 03closure-date - 13https://github.com/miraheze/WikiDiscover [02:15:45] [02WikiDiscover] 07AgentIsai opened pull request 03#97: T10666: Add closure date to API - 13https://github.com/miraheze/WikiDiscover/pull/97 [02:18:30] miraheze/WikiDiscover - AgentIsai the build has errored. [02:19:32] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.03, 1.76, 1.78 [02:19:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.05, 6.27 [02:21:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.53, 2.69, 3.21 [02:21:32] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.51, 1.64, 1.74 [02:22:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:22:00] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03closure-date [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/4d5c43785ced...64c007f5b702 [02:22:01] [02miraheze/WikiDiscover] 07AgentIsai 0364c007f - Add in WikiDiscover [02:22:02] [02WikiDiscover] 07AgentIsai synchronize pull request 03#97: T10666: Add closure date to API - 13https://github.com/miraheze/WikiDiscover/pull/97 [02:22:17] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.770 second response time [02:24:07] miraheze/WikiDiscover - AgentIsai the build has errored. [02:25:08] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.62, 5.23, 4.10 [02:25:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.92, 7.01, 6.56 [02:27:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.90, 6.29, 6.35 [02:28:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.60, 3.73, 3.80 [02:29:23] [02WikiDiscover] 07AgentIsai synchronize pull request 03#97: T10666: Add closure date to API - 13https://github.com/miraheze/WikiDiscover/pull/97 [02:29:26] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03closure-date [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/64c007f5b702...d59c3ba56130 [02:29:29] [02miraheze/WikiDiscover] 07AgentIsai 03d59c3ba - Fix [02:29:32] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.81, 1.39, 1.61 [02:30:13] [02WikiDiscover] 07AgentIsai synchronize pull request 03#97: T10666: Add closure date to API - 13https://github.com/miraheze/WikiDiscover/pull/97 [02:30:15] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03closure-date [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/d59c3ba56130...b534445c0d6e [02:30:17] [02miraheze/WikiDiscover] 07AgentIsai 03b534445 - Remove double arrow [02:32:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:32:27] miraheze/WikiDiscover - AgentIsai the build has errored. [02:34:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.75, 4.67, 3.99 [02:35:02] [02WikiDiscover] 07AgentIsai synchronize pull request 03#97: T10666: Add closure date to API - 13https://github.com/miraheze/WikiDiscover/pull/97 [02:35:04] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03closure-date [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/b534445c0d6e...c062e8de3123 [02:35:05] [02miraheze/WikiDiscover] 07AgentIsai 03c062e8d - Fix array [02:36:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.66, 3.81, 3.77 [02:37:47] miraheze/WikiDiscover - AgentIsai the build has errored. [02:38:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.51, 3.45, 3.22 [02:39:45] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [02:40:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.05, 3.14, 3.12 [02:42:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.01, 3.77, 3.70 [02:43:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [02:43:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.13, 3.89, 3.44 [02:44:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.57, 3.33, 3.55 [02:45:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.24, 3.27, 3.27 [02:46:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.69, 2.82, 3.34 [02:49:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.42, 3.84, 3.50 [02:51:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.37, 3.34, 3.36 [02:52:48] [02WikiDiscover] 07AgentIsai synchronize pull request 03#97: T10666: Add closure date to API - 13https://github.com/miraheze/WikiDiscover/pull/97 [02:52:50] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03closure-date [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/c062e8de3123...9de76fee1272 [02:52:52] [02miraheze/WikiDiscover] 07AgentIsai 039de76fe - Fix [02:52:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.79, 3.41, 3.35 [02:53:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:54:12] [02WikiDiscover] 07AgentIsai synchronize pull request 03#97: T10666: Add closure date to API - 13https://github.com/miraheze/WikiDiscover/pull/97 [02:54:13] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03closure-date [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/9de76fee1272...3b094d190d99 [02:54:15] [02miraheze/WikiDiscover] 07AgentIsai 033b094d1 - Fix function call [02:54:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.22, 2.87, 3.16 [02:55:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:10] miraheze/WikiDiscover - AgentIsai the build has errored. [02:57:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.698 second response time [03:01:25] [02WikiDiscover] 07AgentIsai synchronize pull request 03#97: T10666: Add closure date to API - 13https://github.com/miraheze/WikiDiscover/pull/97 [03:01:28] [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03closure-date [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/3b094d190d99...6c31c5c1e694 [03:01:30] [02miraheze/WikiDiscover] 07AgentIsai 036c31c5c - Test [03:01:51] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:04:57] miraheze/WikiDiscover - AgentIsai the build passed. [03:07:29] !log [www-data@test131] starting deploy of {'folders': 'w/extensions/WikiDiscover'} to all [03:07:30] !log [www-data@test131] finished deploy of {'folders': 'w/extensions/WikiDiscover'} to all - SUCCESS in 0s [03:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.60, 4.01, 3.36 [03:11:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.015 second response time [03:11:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.32, 2.93, 3.04 [03:12:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.88, 3.55, 3.44 [03:13:34] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:14:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.98, 3.33, 3.37 [03:17:27] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.857 second response time [03:30:42] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.46, 3.63, 3.47 [03:34:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.60, 2.89, 3.25 [03:34:49] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:10] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 40.05, 47.49, 22.25 [03:35:22] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [03:35:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.38, 3.03 [03:35:44] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw141 [03:36:20] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 2 backends are down. mw121 mw142 [03:36:38] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw142 [03:37:21] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 190 Threads: 24 Questions: 47379 Slow queries: 4590 Opens: 2846 Open tables: 2840 Queries per second avg: 249.363 [03:37:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.86, 3.20, 3.01 [03:37:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 1.39, 5.80, 11.79 [03:37:44] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw122 [03:37:54] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 13.45, 10.41, 7.50 [03:38:04] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.18, 9.76, 7.42 [03:38:20] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [03:38:38] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [03:38:54] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.240 second response time [03:39:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.90, 8.72, 12.15 [03:39:43] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [03:39:44] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [03:39:54] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.10, 8.20, 7.04 [03:40:04] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.76, 7.75, 6.97 [03:41:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.03, 8.27, 11.57 [03:43:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.69, 10.47, 11.98 [03:44:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.54, 3.80, 3.29 [03:45:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.79, 3.54, 3.28 [03:45:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.17, 10.42, 11.79 [03:47:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.25, 3.78, 3.39 [03:48:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:48:04] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.57, 3.70, 3.42 [03:49:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.95, 3.57, 3.36 [03:49:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.84, 11.24, 11.76 [03:50:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.76, 3.33, 3.31 [03:51:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.41, 3.16, 3.23 [03:52:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 1.36, 2.58, 7.91 [03:55:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 10.91, 11.52 [03:56:45] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.72, 1.56, 6.29 [03:56:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.25, 4.82, 3.99 [03:59:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.37, 11.64, 11.62 [04:02:45] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:02:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.07, 3.63, 3.79 [04:06:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.27, 4.14, 3.50 [04:08:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.47, 3.89, 3.48 [04:08:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.36, 2.92, 3.40 [04:12:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.39, 3.67, 3.46 [04:14:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.47, 3.38, 3.40 [04:16:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.10, 3.45, 3.35 [04:17:56] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:20:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.32, 2.96, 3.16 [04:29:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.71, 5.00, 3.84 [04:32:14] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:39:24] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.33, 3.78, 3.92 [04:43:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.65, 2.25, 3.27 [04:47:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.94, 3.27, 3.53 [04:49:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.62, 2.71, 3.30 [05:05:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.49, 3.97, 3.39 [05:06:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.07, 3.33, 2.54 [05:07:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.29, 3.32, 3.22 [05:08:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.63, 3.34, 2.64 [05:11:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.90, 4.52, 3.73 [05:12:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.08, 3.56, 2.84 [05:14:06] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.74, 3.08, 2.74 [05:14:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.19, 3.34, 2.90 [05:15:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.38, 3.61, 3.56 [05:17:09] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.46, 3.16, 3.39 [05:24:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.99, 3.77, 3.29 [05:28:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.47, 3.99, 3.47 [05:32:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.81, 3.99, 3.60 [05:45:44] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [05:52:46] !log [void@db121] set read_only=0 [05:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.78, 3.22, 3.40 [06:08:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.43, 3.30, 3.11 [06:15:02] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:18:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.84, 3.58 [06:18:44] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.08, 5.50, 3.13 [06:20:44] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.89, 5.06, 3.27 [06:30:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.37, 6.57, 5.69 [06:32:05] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:32:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.54, 6.67, 5.84 [06:38:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.14, 3.71, 3.56 [06:46:16] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [06:56:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.12, 3.81, 3.97 [07:01:34] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:01:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.94, 10.85, 11.75 [07:04:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.98, 10.29, 11.59 [07:05:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.21, 11.82, 11.91 [07:07:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.41, 10.42, 11.38 [07:10:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.57, 3.61 [07:11:00] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:11:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.46, 11.94, 11.81 [07:12:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.87, 3.72, 3.66 [07:14:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.27, 3.82, 3.70 [07:14:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.54, 11.96, 11.57 [07:16:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.33, 3.69, 3.67 [07:20:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.50, 2.72, 3.30 [07:21:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.00, 11.94, 11.96 [07:23:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.19, 11.95, 11.93 [07:31:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.41, 11.30, 11.91 [07:32:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 11.68, 11.99 [07:33:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.46, 11.54, 11.91 [07:37:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.11, 11.23, 11.74 [07:40:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.61, 12.37, 12.06 [07:40:43] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:43:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.40, 7.44, 6.43 [07:44:50] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:45:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.30, 6.52, 6.21 [07:49:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.90, 11.40, 11.36 [07:51:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.63, 11.48, 11.40 [07:53:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.46, 12.52, 11.79 [08:13:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.78, 11.50, 11.97 [08:14:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.91, 11.41, 11.97 [08:15:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.98, 12.82, 12.39 [08:16:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.23, 11.95, 12.09 [08:18:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.29, 11.35, 11.85 [08:20:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.15, 11.62, 11.88 [08:22:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.83, 11.35, 11.75 [08:29:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.16, 10.88, 11.84 [08:31:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.21, 11.08, 11.78 [08:33:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.97, 11.14, 11.71 [08:34:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.54, 10.70, 10.95 [08:36:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.53, 10.56, 10.87 [08:39:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.33, 11.32, 11.60 [08:40:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.58, 10.71, 10.79 [08:41:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.73, 10.73, 11.33 [08:43:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.03, 10.88, 11.29 [08:47:52] PROBLEM - db121 Backups SQL on db121 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864219 seconds old and 1751 bytes [09:09:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.62, 11.08, 11.90 [09:10:42] !log [@mwtask141] starting deploy of {'config': True} to all [09:10:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:51] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [09:10:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:13:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.85, 11.28, 11.73 [09:21:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.32, 10.70, 11.37 [09:25:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.25, 11.43, 11.51 [09:48:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 6.54, 6.21 [09:50:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.62, 7.37, 6.56 [09:52:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 7.32, 6.65 [09:57:46] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2508457005 secs [09:57:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.76, 6.63, 6.58 [10:03:46] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02512812614 secs [10:12:13] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:28:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.88, 6.82, 6.29 [10:30:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 7.43, 6.56 [10:32:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.90, 6.78, 6.42 [10:39:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.25, 6.75 [10:41:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.58, 7.28, 6.83 [10:41:56] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:45:01] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.50, 6.32, 6.59 [10:45:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:47:46] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1222916245 secs [10:57:46] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09528517723 secs [10:58:30] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.18, 3.03, 1.90 [10:58:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.25, 7.21, 6.76 [11:00:29] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.78, 2.61, 1.89 [11:00:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 7.14, 6.78 [11:01:46] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1422473192 secs [11:02:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.28, 6.59, 6.63 [11:05:46] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09824207425 secs [11:09:33] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/cef05e8c08b7...15a860a0de12 [11:09:34] [02miraheze/IncidentReporting] 07translatewiki 0315a860a - Localisation updates from https://translatewiki.net. [11:09:35] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/ImportDump/compare/22a619858e0a...9cf90726038b [11:09:37] [02miraheze/ImportDump] 07translatewiki 039cf9072 - Localisation updates from https://translatewiki.net. [11:09:39] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±81] 13https://github.com/miraheze/MirahezeMagic/compare/7f02b6015c36...14cc64dd8bca [11:09:41] [02miraheze/MirahezeMagic] 07translatewiki 0314cc64d - Localisation updates from https://translatewiki.net. [11:12:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 7.22, 6.85 [11:12:56] miraheze/IncidentReporting - translatewiki the build has errored. [11:19:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.01, 7.91, 7.24 [11:20:15] miraheze/ImportDump - translatewiki the build passed. [11:20:38] miraheze/MirahezeMagic - translatewiki the build passed. [11:27:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 7.99, 7.63 [11:35:46] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1212979853 secs [11:43:34] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:47:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.82, 11.36, 12.00 [11:49:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.15, 11.60, 12.00 [11:51:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.30, 5.98, 6.62 [11:51:46] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08435931802 secs [12:09:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.58, 6.75, 6.57 [12:11:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.87, 6.49, 6.50 [12:20:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.27, 3.73, 2.63 [12:21:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.21, 7.84, 6.95 [12:22:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.39, 5.26, 3.30 [12:23:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 7.44, 6.88 [12:25:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.80, 6.78, 6.71 [12:26:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.71, 3.93, 3.19 [12:28:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.46, 4.02, 3.30 [12:30:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.49, 3.44, 3.17 [12:32:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.29, 3.06, 3.06 [12:40:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.30, 2.94 [12:44:20] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.20, 3.19, 2.99 [12:46:08] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:46:44] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.96, 6.23, 3.28 [12:47:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 7.27, 6.83 [12:48:44] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.15, 4.29, 2.92 [12:49:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.85, 8.15, 7.17 [12:51:10] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:12] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.102 second response time [13:02:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:03:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.88, 7.40, 7.72 [13:07:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.08, 4.66, 3.12 [13:09:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.32, 3.80, 3.00 [13:13:35]