[00:03:19] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.21, 7.24, 3.34 [00:07:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.21, 6.64 [00:11:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.01, 6.31, 6.42 [00:12:37] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.52, 2.89, 2.03 [00:13:19] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.17, 7.78, 6.42 [00:14:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 6.93, 6.65 [00:15:19] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.56, 6.00, 5.93 [00:16:37] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.42, 2.77, 2.19 [00:18:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.59, 8.02, 7.10 [00:20:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.88, 7.16 [00:24:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.39, 6.35, 6.73 [00:33:15] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.55, 2.69 [00:35:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.06, 3.46, 2.77 [00:39:19] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.96, 3.32, 2.88 [00:41:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.79, 3.56, 3.15 [00:42:58] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.01, 3.32, 3.11 [01:00:36] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:05:36] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:08:22] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [01:30:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.83, 7.29, 6.68 [01:34:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.18, 6.46, 6.47 [01:37:14] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:58:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.93, 7.89, 7.05 [02:00:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.32, 7.31, 6.94 [02:06:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.23, 6.64, 6.80 [02:12:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 6.71, 6.76 [02:16:19] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [02:16:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.26, 6.58, 6.70 [02:22:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 7.16, 6.89 [02:24:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.68, 7.50, 7.01 [02:26:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.03, 6.95, 6.87 [02:28:11] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.99, 5.91, 6.50 [02:43:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.62, 6.92, 6.63 [02:49:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.82, 6.48, 6.54 [03:06:19] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:06:22] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [03:11:57] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [03:39:57] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:44:00] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.87, 6.23, 5.60 [03:45:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.83, 5.60, 5.44 [04:21:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 6.48, 6.06 [04:27:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.70, 6.35, 6.12 [04:31:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.86, 6.91, 6.00 [04:32:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.77, 6.95, 6.48 [04:33:55] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.75, 6.11, 5.82 [04:34:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.73, 6.46 [04:38:22] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [04:38:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.86, 6.84, 6.56 [04:40:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.56, 6.64, 6.51 [04:44:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 6.86, 6.66 [04:46:35] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.44, 6.56, 6.59 [05:21:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.95, 6.89, 6.77 [05:25:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.35, 6.95 [05:27:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.77, 6.78 [05:31:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.55, 6.93, 6.87 [05:36:24] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [05:39:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.19, 5.83, 6.51 [06:16:19] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:23:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 6.82, 6.45 [06:25:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.46, 6.36 [06:41:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.94, 9.56, 8.39 [06:44:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [06:47:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.53, 9.84, 8.93 [06:48:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [06:52:40] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [06:56:36] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [07:07:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.34, 7.38, 6.72 [07:08:19] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:09:23] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.94, 6.41, 5.82 [07:11:20] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.05, 6.09, 5.79 [07:11:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.78, 7.73, 7.03 [07:15:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.95, 6.17, 6.54 [07:38:22] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [07:41:42] PROBLEM - hcw.tomat.dev - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hcw.tomat.dev' expires in 15 day(s) (Fri 05 Jan 2024 07:20:56 GMT +0000). [07:41:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c1c741265f95...3d5af32b8506 [07:41:58] [02miraheze/ssl] 07MirahezeSSLBot 033d5af32 - Bot: Update SSL cert for hcw.tomat.dev [08:07:13] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [08:10:53] RECOVERY - hcw.tomat.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'hcw.tomat.dev' will expire on Tue 19 Mar 2024 06:41:50 GMT +0000. [10:16:20] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [10:22:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [10:23:03] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:26:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [10:41:57] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:51:02] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:03:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.73, 7.44, 6.48 [11:05:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.64, 7.25, 6.51 [11:07:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.64, 6.38 [11:08:19] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:09:57] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:40:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:50:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:08:24] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:16:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [12:20:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [12:28:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/059cfd56182b [12:28:26] [02miraheze/puppet] 07paladox 03059cfd5 - jobrunner-hi: increase smw.changePropagationClassUpdate memory limits [12:28:27] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [12:28:28] [02puppet] 07paladox opened pull request 03#3571: jobrunner-hi: increase smw.changePropagationClassUpdate memory limits - 13https://github.com/miraheze/puppet/pull/3571 [12:28:39] [02puppet] 07paladox closed pull request 03#3571: jobrunner-hi: increase smw.changePropagationClassUpdate memory limits - 13https://github.com/miraheze/puppet/pull/3571 [12:28:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/622aed83a304...89622b23daf1 [12:28:44] [02miraheze/puppet] 07paladox 0389622b2 - jobrunner-hi: increase smw.changePropagationClassUpdate memory limits (#3571) [12:28:46] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [12:28:48] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [12:31:24] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.48, 10.45, 9.82 [12:33:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.60, 11.51, 10.27 [12:37:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.69, 11.85, 10.71 [12:37:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.19, 3.00, 1.91 [12:39:11] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.26, 11.77, 10.80 [12:39:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.18, 2.30, 1.78 [12:41:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.84, 11.88, 10.96 [12:44:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 6.92, 6.45 [12:45:01] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.70, 12.12, 11.25 [12:46:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.94, 6.51 [12:48:16] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.86, 6.48, 6.39 [12:50:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.17, 11.90, 11.47 [12:52:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/14db92c40189...a49a8e125fa7 [12:52:06] [02miraheze/mw-config] 07paladox 03a49a8e1 - Enable captcha for contactpage form [12:53:06] miraheze/mw-config - paladox the build passed. [12:54:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f34fae5f8c2b [12:54:30] [02miraheze/mw-config] 07paladox 03f34fae5 - Fix SenderName for wgContactConfig (Doesn't affect metas config) [12:54:33] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [12:54:35] [02mw-config] 07paladox opened pull request 03#5406: Fix SenderName for wgContactConfig (Doesn't affect metas config) - 13https://github.com/miraheze/mw-config/pull/5406 [12:54:44] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.30, 12.21, 11.69 [12:55:14] [02mw-config] 07paladox closed pull request 03#5406: Fix SenderName for wgContactConfig (Doesn't affect metas config) - 13https://github.com/miraheze/mw-config/pull/5406 [12:55:15] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [12:55:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a49a8e125fa7...094ad399dafa [12:55:21] [02miraheze/mw-config] 07paladox 03094ad39 - Fix SenderName for wgContactConfig (Doesn't affect metas config) (#5406) [12:55:23] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [12:55:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:55:38] miraheze/mw-config - paladox the build passed. [12:56:13] miraheze/mw-config - paladox the build passed. [12:56:30] !log [@test131] starting deploy of {'config': True} to all [12:56:31] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [12:56:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:56:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:57:31] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [12:57:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:57:47] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 15s [12:57:53] !log [paladox@mwtask141] starting deploy of {'config': True} to all [12:58:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:58:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:58:31] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 37s [12:58:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:00:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 9.97, 11.55, 11.66 [13:20:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.23, 11.76, 11.11 [13:24:06] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.81, 3.96, 2.58 [13:25:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.56, 6.97, 6.68 [13:25:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.91, 11.91, 11.41 [13:27:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.05, 6.63, 6.59 [13:33:13] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 14.14, 8.96 [13:33:18] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [13:34:52] PROBLEM - mwtask141 Puppet on mwtask141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [13:35:13] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.20, 11.23, 8.52 [13:35:18] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [13:35:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 7.24, 6.79 [13:37:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.00, 9.24, 8.13 [13:37:18] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.08, 6.75, 6.66 [13:39:53] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.86, 3.05, 3.52 [13:41:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.02, 9.26, 10.07 [13:41:50] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.81, 2.32, 3.20 [14:01:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.97, 10.84, 10.01 [14:03:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.74, 11.44, 10.33 [14:05:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.89, 11.58, 10.52 [14:06:22] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [14:23:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.81, 6.89, 6.41 [14:25:10] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.29, 6.57, 6.35 [14:27:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.24, 11.35, 10.79 [14:29:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.99, 11.15, 10.79 [14:35:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.18, 11.45, 11.03 [14:37:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 9.93, 10.89, 10.88 [14:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 6.99, 6.60 [14:52:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.51, 6.47 [14:59:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.30, 11.50, 11.10 [15:01:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.25, 11.37, 11.10 [15:02:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.06, 8.99, 7.41 [15:03:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.27, 7.82, 6.35 [15:05:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.66, 7.50, 6.41 [15:07:16] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.83, 8.06, 6.73 [15:07:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.48, 11.67, 11.28 [15:09:13] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.56, 7.73, 6.79 [15:09:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.43, 11.27, 11.19 [15:13:08] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.16, 6.34, 6.43 [15:16:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.17, 7.55, 7.68 [15:31:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.81, 9.46, 10.14 [15:33:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.83, 6.33, 6.80 [15:51:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.87, 10.20, 9.83 [15:53:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.01, 9.69, 9.69 [15:59:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.29, 6.93, 6.64 [16:03:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.06, 6.41, 6.49 [16:06:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.36, 7.07, 6.08 [16:08:37] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.45, 2.80, 1.94 [16:08:58] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.84, 6.47, 5.98 [16:12:37] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.48, 2.95, 2.21 [16:17:33] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.07, 10.33, 9.72 [16:19:30] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.17, 9.98, 9.67 [16:47:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.33, 6.88, 6.48 [16:49:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.17, 7.30, 6.68 [16:51:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 6.91, 6.60 [16:53:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.01, 6.53, 6.49 [16:57:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.76, 6.82, 6.63 [16:58:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.26, 9.73, 9.25 [17:00:37] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:01:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.15, 7.50, 6.94 [17:03:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.42, 6.98 [17:03:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.03, 7.05, 6.33 [17:05:27] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.93, 6.32, 6.15 [17:06:14] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.29, 10.14, 9.73 [17:09:16] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.70, 2.58, 1.46 [17:13:15] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.48, 2.85, 1.87 [17:14:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.64, 7.49, 7.06 [17:18:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 7.31, 7.09 [17:19:53] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.27, 10.77, 10.15 [17:28:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.67, 7.47, 7.13 [17:30:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.33, 7.06, 7.03 [17:36:05] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.85, 3.50, 2.97 [17:36:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.76, 6.25, 6.71 [17:38:03] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.53, 4.36, 3.32 [17:38:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [17:42:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [17:43:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.51, 9.70, 10.09 [17:46:02] PROBLEM - wiki.denby.tech - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.denby.tech' expires in 15 day(s) (Fri 05 Jan 2024 17:35:07 GMT +0000). [17:46:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3d5af32b8506...c112d1fd888f [17:46:56] [02miraheze/ssl] 07MirahezeSSLBot 03c112d1f - Bot: Update SSL cert for wiki.denby.tech [17:47:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.79, 10.39, 10.28 [17:49:19] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.99, 10.12, 6.25 [17:51:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.54, 8.78, 6.20 [17:57:42] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.37, 2.68, 3.75 [17:59:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.48, 9.77, 10.18 [17:59:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.74, 7.93, 7.09 [18:00:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [18:00:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.30, 7.20, 6.20 [18:02:25] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.46, 7.30, 6.36 [18:03:36] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.45, 1.88, 3.12 [18:03:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.46, 10.19, 10.26 [18:04:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [18:05:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.61, 10.91, 10.50 [18:05:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.28, 7.94, 7.43 [18:06:20] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.70, 6.30, 6.15 [18:11:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.05, 11.42, 10.99 [18:15:45] RECOVERY - wiki.denby.tech - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.denby.tech' will expire on Tue 19 Mar 2024 16:46:41 GMT +0000. [18:16:19] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [18:19:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.40, 7.18, 7.03 [18:21:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.92, 6.93, 6.97 [18:23:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.11, 6.27, 6.72 [18:39:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.11, 11.13, 10.84 [18:39:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.32, 3.63, 2.28 [18:41:44] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.90, 2.51, 2.03 [18:43:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.30, 11.47, 11.07 [18:51:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.61, 7.69, 7.03 [18:53:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.21, 6.94 [18:57:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 7.31, 6.99 [18:59:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 6.99, 6.90 [19:03:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.47, 7.45, 7.08 [19:04:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [19:05:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 7.20, 7.03 [19:08:19] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:08:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [19:09:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.02, 11.18, 11.01 [19:10:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.94, 6.47, 6.78 [19:13:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.61, 11.52, 11.20 [19:14:27] PROBLEM - test131 Current Load on test131 is CRITICAL: LOAD CRITICAL - total load average: 4.39, 6.82, 3.35 [19:14:39] PROBLEM - test131 memcached on test131 is CRITICAL: connect to address 2a10:6740::6:406 and port 11211: Connection refused [19:14:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.53, 6.74, 6.83 [19:15:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.07, 11.60, 11.26 [19:16:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.47, 6.29, 6.66 [19:18:27] RECOVERY - test131 Current Load on test131 is OK: LOAD OK - total load average: 0.28, 3.17, 2.63 [19:20:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.74, 7.13, 6.92 [19:21:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.96, 11.83, 11.50 [19:24:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.52, 6.66, 6.76 [19:28:39] RECOVERY - test131 memcached on test131 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:406 port 11211 [19:30:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.50, 6.94, 6.85 [19:32:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.71, 6.78 [19:33:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 7.68, 8.97, 10.14 [19:42:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.89, 7.54, 7.01 [19:44:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.58, 7.21, 6.96 [19:44:41] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.27, 10.00, 10.03 [19:48:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.86, 7.83, 7.19 [19:48:34] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.31, 10.00, 10.06 [19:52:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.83, 7.03, 7.04 [19:55:55] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 6.05, 6.64 [20:03:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.36, 6.75, 6.83 [20:07:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.31, 6.59, 6.74 [20:09:02] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.64, 9.84, 9.75 [20:10:59] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.74, 9.82, 9.76 [20:18:05] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.78, 7.02, 6.28 [20:18:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.01, 8.09, 7.28 [20:20:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.74, 6.46, 6.16 [20:20:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.48, 7.99, 7.35 [20:24:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.51, 7.26 [20:26:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.63, 7.34 [20:27:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.64, 9.93, 9.80 [20:27:51] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.07, 6.83, 6.40 [20:29:31] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.78, 9.52, 9.67 [20:29:48] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.14, 5.85, 6.08 [20:31:16] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.93, 2.88, 1.46 [20:33:15] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.66, 2.76, 1.60 [20:37:58] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:45:07] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.94, 5.77, 5.00 [20:47:07] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.54, 5.82, 5.12 [20:48:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.35, 7.62, 7.20 [20:54:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.56, 7.89, 7.48 [20:56:48] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.79, 10.23, 9.87 [20:58:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.49, 9.93, 9.80 [21:03:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 6.97, 7.08 [21:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:05:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 6.94, 7.05 [21:13:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.39, 6.76 [21:15:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [21:17:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.89, 7.36, 7.09 [21:19:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.66, 7.39, 7.11 [21:20:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.91, 10.34, 9.87 [21:21:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.78, 7.07, 7.03 [21:27:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.89, 8.01, 7.45 [21:29:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.26, 7.72, 7.43 [21:35:46] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.08, 9.97, 10.06 [21:35:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.40, 7.39, 7.31 [21:37:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.29, 7.29, 7.28 [21:45:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:45:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.24, 9.78, 9.75 [21:47:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.09, 9.89, 9.81 [21:50:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:53:16] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:54:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.46, 10.26, 10.00 [21:55:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.06, 7.01 [21:58:34] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.52, 10.19, 10.05 [21:59:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.14, 7.48, 7.23 [22:06:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.41, 7.51, 6.36 [22:08:07] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 3.92, 6.15, 6.01 [22:09:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.59, 7.40 [22:11:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.68, 6.98, 7.21 [22:16:19] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [22:19:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.02, 7.93, 7.42 [22:21:15] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:21:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.95, 7.81, 7.44 [22:29:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.58, 7.52, 7.31 [22:31:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 7.36, 7.29 [22:47:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 7.36, 7.06 [22:51:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.04, 7.33, 7.16 [22:55:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.33, 7.17 [22:57:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.99, 6.83, 7.01 [23:03:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.01, 5.84, 6.60 [23:06:19] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:09:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 6.94, 6.82 [23:11:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.08, 6.78, 6.78 [23:31:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.83, 10.26, 9.01 [23:33:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.67, 9.77, 8.99 [23:35:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.41, 7.56, 7.07 [23:37:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.72, 6.75, 6.83 [23:39:18] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.84, 6.44, 6.70 [23:43:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.54, 10.54, 9.77 [23:49:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.27, 11.07, 10.24 [23:57:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.22, 11.86, 11.11 [23:58:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/37592d211f52 [23:58:12] [02miraheze/puppet] 07paladox 0337592d2 - jobrunner-hi: tune real/time for jobs [23:58:15] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [23:58:18] [02puppet] 07paladox opened pull request 03#3572: jobrunner-hi: tune real/time for jobs - 13https://github.com/miraheze/puppet/pull/3572 [23:59:34] [02puppet] 07paladox edited pull request 03#3572: jobrunner-hi: tune real/time for jobs - 13https://github.com/miraheze/puppet/pull/3572