[00:00:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.57, 9.75, 8.64 [00:01:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.12, 9.83, 8.85 [00:01:47] [02puppet] 07Universal-Omega edited pull request 03#2305: MediaWiki: Use Debian composer package - 13https://github.com/miraheze/puppet/pull/2305 [00:02:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.57, 9.08, 8.55 [00:03:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:05:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.15, 9.86, 8.89 [00:05:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.85, 1.46 [00:05:48] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 4 minutes ago with 365 failures. Failed resources (up to 3 shown): File[wiki.gesamtschule-nordkirchen.de],File[wiki.gesamtschule-nordkirchen.de_private],File[wiki.ciptamedia.org],File[wiki.ciptamedia.org_private] [00:06:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.31, 8.41, 8.46 [00:08:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.68, 1.46 [00:08:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.67, 9.38, 8.88 [00:08:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:09:56] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:14:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.27, 9.02, 8.81 [00:15:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.29, 10.35, 8.97 [00:17:23] [02puppet] 07Universal-Omega opened pull request 03#2353: Remove icinga-new and grafana-new - 13https://github.com/miraheze/puppet/pull/2353 [00:18:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:20:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.73, 9.24, 9.04 [00:20:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.32, 3.48, 3.39 [00:21:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.71, 9.48, 9.04 [00:23:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.69, 2.67, 3.09 [00:24:15] [02puppet] 07Universal-Omega opened pull request 03#2354: Remove `use_new_mon` - 13https://github.com/miraheze/puppet/pull/2354 [00:26:17] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use puppet111 by default - 13https://github.com/miraheze/puppet/pull/2330 [00:26:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.05, 8.47, 9.98 [00:27:03] [02puppet] 07Universal-Omega edited pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:28:15] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:29:03] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:29:29] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:29:49] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:30:17] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:30:35] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:31:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.23, 9.59, 9.86 [00:31:54] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:32:19] [02puppet] 07Universal-Omega synchronize pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [00:33:52] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:34:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.72, 7.75, 8.38 [00:35:21] [02puppet] 07Universal-Omega opened pull request 03#2355: Remove puppet3 from cp* - 13https://github.com/miraheze/puppet/pull/2355 [00:35:40] [02puppet] 07Universal-Omega synchronize pull request 03#2355: Remove puppet3 from cp* - 13https://github.com/miraheze/puppet/pull/2355 [00:35:55] [02puppet] 07Universal-Omega synchronize pull request 03#2355: Remove puppet3 from cp* - 13https://github.com/miraheze/puppet/pull/2355 [00:36:09] [02puppet] 07Universal-Omega synchronize pull request 03#2355: Remove puppet3 from cp* - 13https://github.com/miraheze/puppet/pull/2355 [00:37:28] paladox: ^ I'm not sure if the puppet111 beneath that was overriding it, but the cps still had puppet3 also. [00:38:02] [02puppet] 07paladox closed pull request 03#2355: Remove puppet3 from cp* - 13https://github.com/miraheze/puppet/pull/2355 [00:38:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/c4a3db3543e1...245b7f10f0f5 [00:38:05] [02miraheze/puppet] 07Universal-Omega 03245b7f1 - Remove puppet3 from cp* (#2355) [00:38:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:39:34] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 13 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [00:40:17] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:40:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.42, 3.05 [00:40:30] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:40:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.95, 7.65, 8.27 [00:40:38] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:40:49] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:40:57] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:41:09] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:41:21] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:41:31] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:41:39] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:41:50] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:42:04] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:42:20] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:42:30] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:43:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.49, 3.05, 2.97 [00:43:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.09, 8.76, 9.11 [00:44:26] [02puppet] 07Universal-Omega synchronize pull request 03#2329: hieradata: delete ovlon mediawiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2329 [00:44:39] [02puppet] 07Universal-Omega closed pull request 03#2327: site: remove ovlon MediaWiki servers + mon2 - 13https://github.com/miraheze/puppet/pull/2327 [00:45:33] [02puppet] 07Universal-Omega edited pull request 03#2329: Remove ovlon servers hieradata and site - 13https://github.com/miraheze/puppet/pull/2329 [00:46:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 8.76, 9.05 [00:47:49] [02puppet] 07Universal-Omega edited pull request 03#2329: Remove ovlon servers from hieradata and site - 13https://github.com/miraheze/puppet/pull/2329 [00:54:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.39, 7.46, 8.33 [00:57:46] [02dns] 07Universal-Omega opened pull request 03#248: Remove ovlon servers - 13https://github.com/miraheze/dns/pull/248 [00:58:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:01:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.28, 9.20, 8.93 [01:02:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.29, 8.59, 8.13 [01:03:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:04:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.48, 8.43, 8.14 [01:05:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.17, 10.34, 9.16 [01:08:03] [02puppet] 07Universal-Omega opened pull request 03#2356: gluster101 default - 13https://github.com/miraheze/puppet/pull/2356 [01:09:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.97, 9.36, 9.27 [01:11:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 10.00, 9.37 [01:12:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.40, 9.36, 9.25 [01:12:30] [02puppet] 07Universal-Omega opened pull request 03#2357: gluster111 default - 13https://github.com/miraheze/puppet/pull/2357 [01:13:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:15:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 9.86, 9.15 [01:16:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.49, 3.24 [01:19:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 3.35, 3.24 [01:19:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.18, 9.71, 9.31 [01:20:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.01, 9.73, 9.54 [01:20:44] [02puppet] 07Universal-Omega opened pull request 03#2358: puppet3 -> puppet111 - 13https://github.com/miraheze/puppet/pull/2358 [01:22:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.04, 8.61, 8.96 [01:23:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:23:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.86, 8.00, 8.50 [01:26:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.65, 8.57, 8.20 [01:26:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.73, 9.42, 8.50 [01:28:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.97, 7.65, 7.88 [01:29:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.84, 8.36, 8.24 [01:42:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.94, 7.81, 8.44 [01:48:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.07, 8.81 [01:51:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.42, 8.33, 8.57 [01:57:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.61, 9.51, 8.97 [01:59:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.64, 8.50, 8.69 [02:03:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:14:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.16, 7.74, 8.21 [02:16:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.68, 9.62, 9.02 [02:16:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.92, 8.88, 8.51 [02:18:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.96, 8.75, 8.78 [02:19:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.15, 8.34, 8.38 [02:21:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.41, 9.38, 8.97 [02:24:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.99, 8.73, 8.79 [02:28:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 8.72, 8.52 [02:32:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.97, 8.39, 8.47 [02:35:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.67, 2.92 [02:35:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 4.40, 3.22 [02:37:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.76, 2.97, 2.79 [02:38:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:41:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.16, 8.44, 8.56 [02:43:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.35, 3.26 [02:43:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.51, 8.65, 8.14 [02:43:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:44:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.31, 8.77, 8.67 [02:44:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.11, 1.79, 1.32 [02:46:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.27, 7.82, 7.89 [02:46:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.23, 7.99, 8.42 [02:47:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.69, 1.34, 1.22 [02:48:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:52:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.95, 8.58, 8.50 [02:53:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.05, 9.11, 8.77 [02:53:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:55:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.90, 8.06, 8.33 [03:04:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 9.07, 8.70 [03:04:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.91, 8.57, 8.23 [03:07:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.87, 9.79, 9.59 [03:07:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 8.31, 8.47 [03:07:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.60, 7.39, 7.82 [03:12:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.66, 9.63, 9.49 [03:13:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.82, 8.69, 8.40 [03:13:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.10, 8.65, 8.67 [03:14:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.33, 8.37, 7.94 [03:15:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 9.29, 9.37 [03:15:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 8.83, 8.36 [03:18:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.42, 10.05, 8.92 [03:20:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.65, 7.77, 7.83 [03:21:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.40, 9.59, 8.94 [03:21:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.65, 8.06, 8.30 [03:23:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:26:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.40, 9.45, 8.97 [03:27:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.85, 10.02, 9.45 [03:28:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:29:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.06, 8.70, 8.77 [03:30:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.98, 9.48, 9.37 [03:30:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.04, 8.72, 8.51 [03:32:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.22, 7.65, 8.45 [03:32:28] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.88, 7.86, 8.42 [03:32:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.52, 9.91, 9.52 [03:33:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:35:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.10, 9.40, 9.41 [03:36:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.62, 8.34, 8.47 [03:37:40] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:48:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:51:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-7 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/e5911bda3412 [03:51:39] [02miraheze/mw-config] 07Universal-Omega 03e5911bd - Set more cache variables [03:51:41] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-7 - 13https://github.com/miraheze/mw-config [03:51:42] [02mw-config] 07Universal-Omega opened pull request 03#4403: Set more cache variables - 13https://github.com/miraheze/mw-config/pull/4403 [03:51:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.07, 7.70, 8.47 [03:52:41] miraheze/mw-config - Universal-Omega the build passed. [03:53:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:55:54] [02mw-config] 07Universal-Omega closed pull request 03#4403: Set more cache variables - 13https://github.com/miraheze/mw-config/pull/4403 [03:55:55] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 [03:55:57] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 - 13https://github.com/miraheze/mw-config [03:55:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e4ad4edba455...9e864403852c [03:56:00] [02miraheze/mw-config] 07Universal-Omega 039e86440 - Set more cache variables (#4403) [03:56:54] miraheze/mw-config - Universal-Omega the build passed. [04:03:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:07:18] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 13 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [04:08:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:17:31] !log [@test101] starting deploy of {'config': True} to skip [04:17:32] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [04:17:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:21:45] !log [@mwtask111] starting deploy of {'config': True} to scsvg [04:21:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:21:53] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 7s [04:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:23:00] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [04:24:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-7 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/038b6ffd44af [04:24:45] [02miraheze/mw-config] 07Universal-Omega 03038b6ff - Set $wgMainStash for betaheze [04:24:46] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-7 - 13https://github.com/miraheze/mw-config [04:24:48] [02mw-config] 07Universal-Omega opened pull request 03#4404: Set $wgMainStash for betaheze - 13https://github.com/miraheze/mw-config/pull/4404 [04:25:55] miraheze/mw-config - Universal-Omega the build passed. [04:38:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:47:45] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.65, 8.90, 7.58 [04:50:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.91, 7.57, 7.29 [04:52:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.44, 9.84, 8.79 [04:52:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.12, 11.76, 9.30 [04:53:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:54:17] dmehus: what were you flooding? Lol [05:00:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.03, 9.36, 9.28 [05:06:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.57, 7.49, 8.43 [05:08:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 9.47, 9.83 [05:12:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 8.98, 7.67 [05:15:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.96, 8.14, 7.56 [05:23:23] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [05:24:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.14, 7.68, 8.49 [05:28:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:38:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:48:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:53:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:08:55] CosmicAlpha, trying to DM something to Agent :P [06:09:01] but forgot to paste as a snippet [06:09:03] my usual :) [06:09:32] dmehus: lol, I've done that before... [06:09:36] heh [06:09:47] I do it too often though [06:09:55] Reception123 teases me about it [06:09:59] I've done it like 4 times I think... [06:10:04] lol wow [06:10:08] that's a fair bit too [06:10:37] Lol Yeah... so I do know that excess flood message also... [06:12:44] dmehus: also don't know you saw but the performance task I closed this morning seeing as how it maintained overnight and still is now. I made one final performance change since then though, more caching, and a few more ways to better increase it even now. [06:22:44] CosmicAlpha, no didn't see that yet, but yeah, I've noticed the performance has improved even more :) [06:22:47] so that's great [06:27:16] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/us [06:27:16] thon3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.wikimicrofinanza.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:44:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 8.08, 7.54 [06:46:53] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.89, 7.65, 7.47 [06:54:24] dmehus: great! [06:57:26] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [07:08:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:13:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:38:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:43:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:53:25] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:05:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.47, 9.18, 8.14 [09:12:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/38f513aeec92 [09:12:17] [02miraheze/mw-config] 07Universal-Omega 0338f513a - Default $wgUseInstantCommons to false [09:12:18] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [09:12:20] [02mw-config] 07Universal-Omega opened pull request 03#4405: Default $wgUseInstantCommons to false - 13https://github.com/miraheze/mw-config/pull/4405 [09:13:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/38f513aeec92...4f26841f2944 [09:13:19] [02miraheze/mw-config] 07Universal-Omega 034f26841 - Update ManageWikiSettings.php [09:13:20] [02mw-config] 07Universal-Omega synchronize pull request 03#4405: Default $wgUseInstantCommons to false - 13https://github.com/miraheze/mw-config/pull/4405 [09:13:26] miraheze/mw-config - Universal-Omega the build passed. [09:13:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:13:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.95, 9.92, 8.84 [09:14:29] miraheze/mw-config - Universal-Omega the build passed. [09:14:31] [02mw-config] 07Universal-Omega edited pull request 03#4405: Default $wgUseInstantCommons to false - 13https://github.com/miraheze/mw-config/pull/4405 [09:16:21] [02mw-config] 07Universal-Omega closed pull request 03#4405: Default $wgUseInstantCommons to false - 13https://github.com/miraheze/mw-config/pull/4405 [09:16:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/9e864403852c...b272e5deb324 [09:16:24] [02miraheze/mw-config] 07Universal-Omega 03b272e5d - Default $wgUseInstantCommons to false (#4405) [09:16:26] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [09:16:27] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 [09:17:31] miraheze/mw-config - Universal-Omega the build passed. [09:18:16] !log [@test101] starting deploy of {'config': True} to skip [09:18:17] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:18:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:18:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:21:49] !log [@mwtask111] starting deploy of {'config': True} to scsvg [09:21:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:21:56] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 7s [09:22:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:22:51] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:23:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:27:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 9.82, 9.75 [09:28:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:34:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.12, 6.97, 8.40 [09:48:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:53:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:58:03] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [10:24:24] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [11:17:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.41, 2.72 [11:17:56] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [11:19:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.63, 2.53 [11:33:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:34:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.49, 9.29, 7.68 [11:36:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.74, 7.93, 7.40 [11:38:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:48:10] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:38:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.46, 2.70 [12:39:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.19, 4.15, 2.97 [12:40:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 9.15, 7.98 [12:41:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.58, 2.85 [12:42:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.00, 3.45, 2.92 [12:43:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.80, 2.89, 2.71 [12:44:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.61, 3.14 [12:45:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.39, 2.63, 2.68 [12:45:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.99, 8.40, 8.04 [12:48:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:50:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.87, 2.85, 2.96 [12:53:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:18:44] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.teessidehackspace.org.uk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:19:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.85, 8.83, 8.06 [13:19:21] Reception123: expired ^ [13:19:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 8.44, 7.89 [13:22:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.73, 8.14, 7.86 [13:22:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.10, 2.76 [13:25:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 2.43, 2.56 [13:30:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.23, 8.07, 8.12 [13:34:30] RhinosF1: I won't be available until ~tomorrow noon so if possible someone should work on getting the new setup for the SSL bot done [13:39:56] PROBLEM - wiki.teessidehackspace.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teessidehackspace.org.uk could not be found [13:48:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.71, 9.79, 8.52 [13:51:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.97, 8.91, 8.39 [13:54:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.29, 8.13, 8.17 [13:56:21] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [14:00:56] ^ that domain is pointed but no wiki is attached [14:02:51] Looks like the wiki was deleted [14:07:06] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:13:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:16:59] PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/ [14:16:59] on3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wiki.lefrenchmelee.fr. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:17:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.28, 2.75 [14:19:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.96, 2.54, 2.55 [14:25:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.47, 9.65, 8.78 [14:26:38] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:28:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 9.65, 8.94 [14:33:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.44, 8.15, 8.50 [14:33:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:37:14] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 13 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [14:39:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.21, 8.72, 8.81 [14:41:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.19, 8.42, 7.99 [14:44:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 8.42, 8.09 [14:47:50] RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: SSL OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:48:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:49:32] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Mon 28 Mar 2022 09:32:44 GMT +0000. [14:58:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:59:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.31, 8.54, 8.28 [14:59:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.39, 8.64, 7.88 [15:02:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.50, 8.44, 7.96 [15:05:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.99, 10.22, 9.10 [15:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.75, 10.14, 9.31 [15:07:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 9.68, 9.09 [15:08:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:10:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.62, 9.65, 9.36 [15:11:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 8.84, 8.27 [15:12:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.70, 9.09, 8.41 [15:13:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:14:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.54, 8.41, 8.22 [15:14:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.14, 8.67, 8.35 [15:16:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 8.72, 8.78 [15:16:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.11, 9.62, 9.34 [15:17:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.19, 8.14, 8.21 [15:18:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.10, 7.88, 8.44 [15:18:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.26, 9.42, 9.33 [15:23:12] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.teessidehackspace.org.uk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:23:41] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:28:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:43:41] ok : [RESOLVED] (DatasourceNoData 000000004 yes A !sre MediaWiki Exception Rate mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:43:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 8.16, 8.41 [15:46:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.82, 8.55, 8.51 [15:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:49:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.56, 8.31, 8.43 [15:55:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.94, 8.85, 8.62 [15:58:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.17, 9.80, 9.01 [15:59:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.54, 3.25 [16:01:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:02:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 2.96, 3.08 [16:04:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.89, 8.68, 7.94 [16:04:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.21, 8.76, 8.32 [16:05:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.02, 9.14, 8.45 [16:06:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.69, 8.26, 7.90 [16:07:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.27, 8.28, 8.22 [16:07:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.75, 9.19, 9.22 [16:07:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.98, 8.30, 8.26 [16:10:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 10.03, 9.50 [16:14:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:18:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 9.73, 9.72 [16:21:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.59, 9.64 [16:22:43] PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tulpa.info All nameservers failed to answer the query. [16:23:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.06, 9.77, 8.90 [16:23:42] [02ssl] 07MacFan4000 opened pull request 03#476: Remove www.wikimicrofinanza.it - 13https://github.com/miraheze/ssl/pull/476 [16:24:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.47, 9.20, 9.52 [16:25:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.72, 9.50, 8.93 [16:26:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.41, 8.62, 8.17 [16:27:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.31, 10.08, 9.79 [16:30:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.25, 9.08, 9.44 [16:31:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 9.46, 8.73 [16:34:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.53, 8.56, 8.50 [16:34:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.66, 7.86, 8.30 [16:34:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.89, 7.45, 7.87 [16:34:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:36:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.11, 7.89, 8.25 [16:38:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4405-Universal-Omega-patch-8 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/53c133d12ae2 [16:39:01] [02miraheze/mw-config] 07Universal-Omega 0353c133d - Revert "Default $wgUseInstantCommons to false (#4405)" [16:39:02] [02mw-config] 07Universal-Omega created branch 03revert-4405-Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [16:39:04] [02mw-config] 07Universal-Omega opened pull request 03#4406: Revert "Default $wgUseInstantCommons to false" - 13https://github.com/miraheze/mw-config/pull/4406 [16:39:08] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4405-Universal-Omega-patch-8 [16:39:10] [02mw-config] 07Universal-Omega closed pull request 03#4406: Revert "Default $wgUseInstantCommons to false" - 13https://github.com/miraheze/mw-config/pull/4406 [16:39:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/b272e5deb324...962e4ef36668 [16:39:13] [02miraheze/mw-config] 07Universal-Omega 03962e4ef - Revert "Default $wgUseInstantCommons to false (#4405)" (#4406) [16:39:14] [02mw-config] 07Universal-Omega deleted branch 03revert-4405-Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [16:40:11] miraheze/mw-config - Universal-Omega the build passed. [16:40:17] miraheze/mw-config - Universal-Omega the build passed. [16:40:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:41:10] .blocks add nick miraheze-github_ [16:41:11] RhinosF1: Successfully added block: miraheze-github_ [16:42:14] PROBLEM - wiki.controlistes.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.controlistes.fr could not be found [16:48:13] !log [@test101] starting deploy of {'config': True} to skip [16:48:14] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [16:48:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:50:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:52:39] !log [@mwtask111] starting deploy of {'config': True} to scsvg [16:52:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:46] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 7s [16:52:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:55:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.12, 9.89, 9.42 [16:58:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.30, 8.91, 8.43 [16:58:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.80, 9.58, 9.36 [17:03:51] .op [17:03:51] Attempting to OP... [17:04:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.05, 7.84, 8.17 [17:04:04] .deop [17:04:04] Attempting to OP... [17:04:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:06:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.54, 9.64, 9.45 [17:07:08] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u [17:07:09] ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikimicrofinanza.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:09:01] [02ssl] 07Universal-Omega opened pull request 03#477: Remove www.wikimicrofinanza.it - 13https://github.com/miraheze/ssl/pull/477 [17:09:32] [02ssl] 07Universal-Omega synchronize pull request 03#477: Remove www.wikimicrofinanza.it - 13https://github.com/miraheze/ssl/pull/477 [17:09:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.52, 10.24, 8.96 [17:09:59] [02ssl] 07RhinosF1 commented on pull request 03#477: Remove www.wikimicrofinanza.it - 13https://github.com/miraheze/ssl/pull/477#issuecomment-1015628897 [17:10:17] [02ssl] 07Universal-Omega commented on pull request 03#477: Remove www.wikimicrofinanza.it - 13https://github.com/miraheze/ssl/pull/477#issuecomment-1015629184 [17:10:19] [02ssl] 07Universal-Omega closed pull request 03#477: Remove www.wikimicrofinanza.it - 13https://github.com/miraheze/ssl/pull/477 [17:12:10] RECOVERY - wiki.controlistes.fr - reverse DNS on sslhost is OK: SSL OK - wiki.controlistes.fr reverse DNS resolves to cp20.miraheze.org - CNAME OK [17:12:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.30, 9.65, 9.64 [17:12:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 9.54, 8.92 [17:13:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.27, 2.91 [17:14:12] [02ssl] 07Universal-Omega opened pull request 03#478: Remove linkwiki.cf - 13https://github.com/miraheze/ssl/pull/478 [17:14:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:14:48] [02ssl] 07Universal-Omega synchronize pull request 03#478: Remove linkwiki.cf - 13https://github.com/miraheze/ssl/pull/478 [17:15:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.57, 11.65, 9.77 [17:16:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 16.75, 12.11, 9.65 [17:16:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 3.29, 3.00 [17:18:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 8.54, 10.38, 9.44 [17:18:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.95, 9.52, 9.50 [17:18:49] [02ssl] 07Universal-Omega opened pull request 03#479: Remove wiki.teessidehackspace.org.uk - 13https://github.com/miraheze/ssl/pull/479 [17:19:40] [02ssl] 07Universal-Omega synchronize pull request 03#479: Remove wiki.teessidehackspace.org.uk - 13https://github.com/miraheze/ssl/pull/479 [17:19:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [17:20:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [17:21:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.56, 8.97, 9.06 [17:22:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.69, 7.14, 8.08 [17:23:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:23:18] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:23:49] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:24:26] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.73, 11.09, 9.19 [17:24:35] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:24:41] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:24:57] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:18] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:26:53] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.695 second response time [17:27:43] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.572 second response time [17:27:50] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.960 second response time [17:28:11] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.748 second response time [17:28:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.60, 7.15, 8.16 [17:28:33] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.317 second response time [17:32:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.32, 12.50, 10.10 [17:37:43] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [17:39:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [17:40:47] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:41:33] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:41:39] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:42:17] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:34] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:44:37] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.031 second response time [17:44:39] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.025 second response time [17:44:57] !log restarting php-fpm on mw112 - locked up & unresponsive since 17:36 [17:45:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:45:20] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.306 second response time [17:45:30] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.336 second response time [17:45:34] CosmicAlpha: fyi ^ [17:47:04] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.304 second response time [17:47:10] RhinosF1: thanks! Any idea why? [17:47:41] CosmicAlpha: nope [17:48:51] !log same on mw102 - nothing for ~5 minutes [17:49:30] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 2.762 second response time [17:49:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:37] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.011 second response time [17:52:52] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 9.991 second response time [17:53:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [17:59:28] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.149 second response time [17:59:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.47, 8.60, 9.84 [18:00:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:02:02] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[python-is-python3] [18:08:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 9.56, 9.59 [18:11:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.88, 9.10, 9.48 [18:23:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.43, 9.93, 9.43 [18:26:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.69, 9.95, 9.54 [18:30:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.87, 8.02, 9.62 [18:31:23] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:34:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 10.16, 9.62 [18:36:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.49, 7.84, 9.77 [18:37:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.13, 9.54, 9.50 [18:39:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:44:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:47:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.06, 9.34, 9.05 [18:49:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.00, 10.15, 9.71 [18:50:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 9.00, 8.96 [18:50:54] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/245b7f10f0f5...cf058e57d023 [18:50:55] [02miraheze/puppet] 07JohnFLewis 03cf058e5 - firewall: close of 22 to the outside [18:51:00] paladox: ^ [18:51:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.19, 8.66, 8.03 [18:52:23] RECOVERY - wiki.tulpa.info - reverse DNS on sslhost is OK: SSL OK - wiki.tulpa.info reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:52:35] JohnLewis: think you want to exempt the cloud servers as well [18:53:04] Why? [18:53:15] What if one of the vms goes down [18:53:18] What if we reboot [18:53:25] Vms aren’t automatically started [18:53:40] That is why we have 2 for redundancy + an iLO with remote console [18:54:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.47, 8.39, 8.05 [18:54:05] I thought we cannot use the remote console because it requires a license [18:54:24] You can use it for a small period of time before it DCs [18:54:28] plus we have a trial license currently [18:54:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:54:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.80, 9.67, 9.67 [18:55:18] Hmm [18:57:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.91, 10.58, 10.02 [18:58:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:58:51] JohnLewis: how do we connect [18:59:15] * RhinosF1 has been going directly [19:00:24] RhinosF1: via bast [19:00:37] JohnLewis: is there an example ssh config [19:00:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.70, 9.79, 9.85 [19:00:56] paladox: ^ [19:01:36] * RhinosF1 has spent the day attempting to understand BagOStuff [19:02:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.55, 9.05, 8.88 [19:02:17] I’m mobile currently [19:02:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.26, 7.30, 8.32 [19:03:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.08, 9.63, 9.76 [19:04:44] PROBLEM - wiki.pacebi.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.pacebi.org All nameservers failed to answer the query. [19:04:46] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:04:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.08, 8.13, 8.56 [19:05:00] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cf058e57d023...0d15db51feb8 [19:05:02] [02miraheze/puppet] 07JohnFLewis 030d15db5 - syntaxx error missed by linter [19:05:38] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:06:22] RhinosF1: setting up a ProxyJump however your local client/software does it - https://wikitech.wikimedia.org/wiki/SRE/Production_access#Setting_up_your_SSH_config is an example [19:06:23] [url] SRE/Production access - Wikitech | wikitech.wikimedia.org [19:06:43] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:07:22] JohnLewis: does it matter which bast [19:07:31] bast.mh.o [19:07:40] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:07:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.38, 8.13, 8.50 [19:08:14] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:08:50] PROBLEM - bast101 Puppet on bast101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:09:20] PROBLEM - test101 SSH on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:20] PROBLEM - mw111 SSH on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:22] ah monitoring, icinga will need access [19:10:22] JohnLewis: Connection closed by UNKNOWN port 65535 [19:11:48] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0d15db51feb8...4d86ad679711 [19:11:50] [02miraheze/puppet] 07JohnFLewis 034d86ad6 - add icinga2 to firewall exemption [19:11:58] RhinosF1: which server are you connecting to? [19:12:21] JohnLewis: test101 via bas. [19:12:23] bast. [19:13:41] RhinosF1: that's the error I got with ssl request attempting to proxy also. [19:13:53] PROBLEM - db111 SSH on db111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:30] PROBLEM - phab121 SSH on phab121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:15:16] JohnLewis: same error ssh'ing to bast101 directly [19:15:28] PROBLEM - cloud12 SSH on cloud12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:15:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.94, 9.90, 9.26 [19:15:56] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u [19:15:56] ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikimicrofinanza.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:16:30] PROBLEM - cloud10 SSH on cloud10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:11] [02puppet] 07RhinosF1 opened pull request 03#2359: users: php 7.3 is no more - 13https://github.com/miraheze/puppet/pull/2359 [19:17:22] CosmicAlpha: ^ [19:17:39] PROBLEM - mail121 SSH on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:18:25] PROBLEM - mw121 SSH on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:18:55] PROBLEM - mw102 SSH on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:15] JohnLewis: need to fix salt I guess [19:19:32] PROBLEM - ldap111 SSH on ldap111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.23, 8.55, 9.50 [19:20:05] PROBLEM - gluster101 SSH on gluster101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:46] PROBLEM - es121 SSH on es121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:56] paladox: feel free to add the relevant exemption to it [19:21:33] Ok, can just add if it uses the class “base” then exempt it [19:22:00] hmm, looks like we still have the old infra lsbot running [19:22:15] paladox: hm? [19:22:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.02, 9.22, 8.69 [19:23:01] Exempting it so you can ssh to what ever server within the cluster [19:23:17] (as well as the old infra rc bot) [19:23:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.53, 9.01, 9.45 [19:23:41] paladox: hang on [19:23:44] Why do we need to allow that though? [19:23:52] JohnLewis: mw needs it [19:24:08] unless i mess with the config [19:24:12] I'll just change it to base then [19:24:19] If so many services require inter-SSH [19:24:42] !log [rhinos@mwtask111] starting deploy of {'config': True} to ovlon [19:24:57] PROBLEM - es101 SSH on es101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:58] * RhinosF1 quick test [19:25:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:07] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4d86ad679711...e71637a050d4 [19:25:08] [02miraheze/puppet] 07JohnFLewis 03e71637a - use class base for firewall restrictions [19:25:28] PROBLEM - jobchron121 SSH on jobchron121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:34] !log deploy killed [19:25:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:49] JohnLewis: yeah mw was blocked too [19:25:57] RhinosF1: you did ovlon [19:26:07] !log [rhinos@mwtask111] starting deploy of {'config': True} to scsvg [19:26:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:15] CosmicAlpha: why hasn't that been failed over [19:26:21] I have PRs. [19:26:36] oh it does claim to work [19:26:56] PROBLEM - mem101 SSH on mem101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.56, 7.79, 9.76 [19:27:03] PROBLEM - ns2 SSH on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:03] can someone merge https://github.com/miraheze/puppet/pull/2326/files [19:27:04] [url] deploy-mediawiki: remove ovlon by Universal-Omega · Pull Request #2326 · miraheze/puppet · GitHub | github.com [19:27:14] mw101 worked then it hung [19:27:17] [02puppet] 07JohnFLewis closed pull request 03#2250: irc: use irc.libera.chat rather than direct IPv6 address - 13https://github.com/miraheze/puppet/pull/2250 [19:27:27] PROBLEM - db121 SSH on db121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:42] [02puppet] 07JohnFLewis assigned pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [19:27:49] !log killed again [19:27:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:08] PROBLEM - mw112 SSH on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:45] [02puppet] 07JohnFLewis closed pull request 03#2326: deploy-mediawiki: remove ovlon - 13https://github.com/miraheze/puppet/pull/2326 [19:28:47] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e71637a050d4...31bad4019af6 [19:28:48] [02miraheze/puppet] 07Universal-Omega 0331bad40 - deploy-mediawiki: remove ovlon (#2326) [19:29:05] [02puppet] 07JohnFLewis closed pull request 03#2359: users: php 7.3 is no more - 13https://github.com/miraheze/puppet/pull/2359 [19:29:06] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/31bad4019af6...f804f39fff3c [19:29:08] [02miraheze/puppet] 07RhinosF1 03f804f39 - users: php 7.3 is no more (#2359) [19:29:12] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:24] [02puppet] 07JohnFLewis closed pull request 03#2358: puppet3 -> puppet111 - 13https://github.com/miraheze/puppet/pull/2358 [19:29:26] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/f804f39fff3c...37983c1430e5 [19:29:26] PROBLEM - db101 SSH on db101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:27] [02miraheze/puppet] 07Universal-Omega 0337983c1 - puppet3 -> puppet111 (#2358) [19:29:37] [02puppet] 07JohnFLewis closed pull request 03#2357: gluster111 default - 13https://github.com/miraheze/puppet/pull/2357 [19:29:38] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/37983c1430e5...b03f1b4d0057 [19:29:40] [02miraheze/puppet] 07Universal-Omega 03b03f1b4 - gluster111 default (#2357) [19:29:45] PROBLEM - cp31 SSH on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:54] [02puppet] 07JohnFLewis closed pull request 03#2356: gluster101 default - 13https://github.com/miraheze/puppet/pull/2356 [19:29:56] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b03f1b4d0057...24cf073a7292 [19:29:57] [02miraheze/puppet] 07Universal-Omega 0324cf073 - gluster101 default (#2356) [19:30:34] [02puppet] 07JohnFLewis closed pull request 03#2352: Revert "absent manageInactiveWikis cron" - 13https://github.com/miraheze/puppet/pull/2352 [19:30:34] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:35] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/24cf073a7292...d8988695d20e [19:30:37] [02miraheze/puppet] 07Universal-Omega 03d898869 - Revert "absent manageInactiveWikis cron (#2348)" (#2352) [19:31:13] [02puppet] 07JohnFLewis closed pull request 03#2330: hieradata common: use SCSVG servers - 13https://github.com/miraheze/puppet/pull/2330 [19:31:14] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±9] 13https://github.com/miraheze/puppet/compare/d8988695d20e...478330e2a237 [19:31:16] [02miraheze/puppet] 07Universal-Omega 03478330e - hieradata common: use SCSVG servers (#2330) [19:31:26] PROBLEM - es111 SSH on es111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:46] PROBLEM - cp30 SSH on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:04] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:32:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.07, 8.80, 9.25 [19:32:18] [02puppet] 07JohnFLewis closed pull request 03#2329: Remove ovlon servers from hieradata and site - 13https://github.com/miraheze/puppet/pull/2329 [19:32:19] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-25/±1] 13https://github.com/miraheze/puppet/compare/478330e2a237...278ce1a42014 [19:32:21] [02miraheze/puppet] 07Universal-Omega 03278ce1a - Remove ovlon servers from hieradata and site (#2329) [19:32:42] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:42] [02puppet] 07JohnFLewis closed pull request 03#2305: MediaWiki: Use Debian composer package - 13https://github.com/miraheze/puppet/pull/2305 [19:33:43] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/278ce1a42014...933821afb120 [19:33:45] [02miraheze/puppet] 07Universal-Omega 03933821a - MediaWiki: Use Debian composer package (#2305) [19:34:00] PROBLEM - puppet111 SSH on puppet111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:05] [02puppet] 07JohnFLewis closed pull request 03#2354: Remove `use_new_mon` - 13https://github.com/miraheze/puppet/pull/2354 [19:34:06] PROBLEM - gluster111 SSH on gluster111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:07] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±9] 13https://github.com/miraheze/puppet/compare/933821afb120...b0f8770b518d [19:34:08] [02miraheze/puppet] 07Universal-Omega 03b0f8770 - Remove `use_new_mon` (#2354) [19:34:22] [02puppet] 07JohnFLewis closed pull request 03#2353: Remove icinga-new and grafana-new - 13https://github.com/miraheze/puppet/pull/2353 [19:34:24] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/b0f8770b518d...30d7a2964c69 [19:34:24] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:34:25] [02miraheze/puppet] 07Universal-Omega 0330d7a29 - Remove icinga-new and grafana-new (#2353) [19:34:59] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [19:35:16] [02puppet] 07Universal-Omega synchronize pull request 03#2350: Move prometheus::es_exporter to graylog121 - 13https://github.com/miraheze/puppet/pull/2350 [19:35:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.07, 7.53, 8.23 [19:35:21] [02puppet] 07Universal-Omega edited pull request 03#2350: Add prometheus::es_exporter to graylog121 - 13https://github.com/miraheze/puppet/pull/2350 [19:35:25] [02dns] 07JohnFLewis closed pull request 03#248: Remove ovlon servers - 13https://github.com/miraheze/dns/pull/248 [19:35:26] [02miraheze/dns] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/8ad1982d96a8...3185add73e7f [19:35:28] [02miraheze/dns] 07Universal-Omega 03c741a20 - Remove ovlon servers [19:35:29] [02miraheze/dns] 07JohnFLewis 033185add - Merge pull request #248 from Universal-Omega/patch-10 [19:35:34] [02dns] 07JohnFLewis closed pull request 03#245: Remove sportsanalytics.wiki zone - 13https://github.com/miraheze/dns/pull/245 [19:35:34] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:35] [02miraheze/dns] 07JohnFLewis pushed 032 commits to 03master [+0/-2/±0] 13https://github.com/miraheze/dns/compare/3185add73e7f...376d4f2430f8 [19:35:37] [02miraheze/dns] 07Universal-Omega 0348d7f02 - Remove sportsanalytics.wiki zone [19:35:37] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:35:38] [02miraheze/dns] 07JohnFLewis 03376d4f2 - Merge pull request #245 from Universal-Omega/patch-9 [19:35:44] [02dns] 07JohnFLewis closed pull request 03#244: Remove rangpurpedia.xyz zone - 13https://github.com/miraheze/dns/pull/244 [19:35:46] [02miraheze/dns] 07JohnFLewis pushed 032 commits to 03master [+0/-2/±0] 13https://github.com/miraheze/dns/compare/376d4f2430f8...8892ac8c4622 [19:35:47] [02miraheze/dns] 07Universal-Omega 03192e935 - Remove rangpurpedia.xyz zone [19:35:49] [02miraheze/dns] 07JohnFLewis 038892ac8 - Merge pull request #244 from Universal-Omega/patch-8 [19:35:50] PROBLEM - graylog121 SSH on graylog121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.31, 7.93, 8.43 [19:36:05] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:46] PROBLEM - wiki.pacebi.org - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/p [19:36:46] ist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query pacebi.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:36:52] It's annoying i have to enter my passphrase twice for ssh :( [19:37:38] PROBLEM - gluster121 SSH on gluster121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:39:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/425c88e4aca3 [19:39:32] [02miraheze/puppet] 07paladox 03425c88e - base: Allow salt hosts to connect to port 22 [19:39:33] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [19:39:34] [02puppet] 07paladox opened pull request 03#2360: base: Allow salt hosts to connect to port 22 - 13https://github.com/miraheze/puppet/pull/2360 [19:39:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.37, 7.20, 8.28 [19:40:02] [02puppet] 07paladox closed pull request 03#2360: base: Allow salt hosts to connect to port 22 - 13https://github.com/miraheze/puppet/pull/2360 [19:40:02] PROBLEM - mem121 SSH on mem121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/30d7a2964c69...6888be707572 [19:40:05] [02miraheze/puppet] 07paladox 036888be7 - base: Allow salt hosts to connect to port 22 (#2360) [19:40:06] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [19:40:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [19:40:13] PROBLEM - mw101 SSH on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:20] paladox: uh? [19:40:29] When do we deploy non base costs? [19:40:52] JohnLewis: i don't understand [19:40:57] we just discussed this above? [19:41:05] paladox: it's applied to base now [19:41:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.32, 10.69, 9.81 [19:41:07] And then I said I'll change it to base [19:41:11] oh [19:41:12] John did that after I said mw needs too [19:41:12] i see [19:41:16] sorry i misread that [19:41:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6888be707572...b2bb749b5d80 [19:41:35] [02miraheze/puppet] 07paladox 03b2bb749 - Revert "base: Allow salt hosts to connect to port 22 (#2360)" [19:44:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:44:41] PROBLEM - mw122 SSH on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.55, 8.80, 9.20 [19:46:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.61, 9.41, 8.82 [19:46:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.47, 9.64, 9.15 [19:47:02] [02puppet] 07Universal-Omega opened pull request 03#2361: Default jobchron121 - 13https://github.com/miraheze/puppet/pull/2361 [19:47:51] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:48:17] [02puppet] 07Universal-Omega opened pull request 03#2362: Make bacula2 use puppet111 - 13https://github.com/miraheze/puppet/pull/2362 [19:49:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.43, 10.05, 9.61 [19:50:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.61, 8.14, 9.50 [19:50:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.72, 9.78, 9.05 [19:53:37] RECOVERY - es111 SSH on es111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [19:53:49] [02puppet] 07JohnFLewis commented on pull request 03#2362: Make bacula2 use puppet111 - 13https://github.com/miraheze/puppet/pull/2362#issuecomment-1015782288 [19:53:50] [02puppet] 07JohnFLewis closed pull request 03#2362: Make bacula2 use puppet111 - 13https://github.com/miraheze/puppet/pull/2362 [19:54:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.05, 9.70, 9.60 [19:54:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:56:08] RECOVERY - puppet111 SSH on puppet111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [19:57:31] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [19:58:11] RECOVERY - graylog121 SSH on graylog121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [19:58:14] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [19:58:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.36, 8.73, 9.11 [19:58:53] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [19:59:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.34, 8.07, 8.38 [19:59:51] RECOVERY - gluster121 SSH on gluster121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:00:05] RECOVERY - gluster111 SSH on gluster111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:01:40] RECOVERY - mem121 SSH on mem121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:02:08] RECOVERY - mw101 SSH on mw101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:02:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 9.32, 9.28 [20:02:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.97, 7.99, 8.45 [20:05:54] RECOVERY - mw122 SSH on mw122 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:05:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.28, 10.15, 9.62 [20:06:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 7.62, 8.49 [20:08:03] RECOVERY - mw111 SSH on mw111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:08:28] RECOVERY - db111 SSH on db111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:08:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:09:27] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:09:43] RECOVERY - phab121 SSH on phab121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:10:22] RECOVERY - cloud12 SSH on cloud12 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:10:40] RECOVERY - ldap111 SSH on ldap111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:11:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.80, 9.81, 9.25 [20:11:30] RECOVERY - cloud10 SSH on cloud10 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:12:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.12, 8.88, 8.65 [20:12:48] RECOVERY - mail121 SSH on mail121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:13:06] RECOVERY - mw121 SSH on mw121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:13:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.97, 9.30, 9.56 [20:13:23] RECOVERY - mw102 SSH on mw102 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:14:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.63, 8.62, 8.33 [20:15:00] RECOVERY - es121 SSH on es121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:15:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.38, 8.90, 8.81 [20:17:25] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.47, 9.60, 8.78 [20:17:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.75, 9.70, 9.43 [20:18:00] RECOVERY - gluster101 SSH on gluster101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:18:13] RECOVERY - test101 SSH on test101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:18:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:18:52] RECOVERY - es101 SSH on es101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:19:04] RECOVERY - jobchron121 SSH on jobchron121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:20:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.68, 10.66, 9.98 [20:20:27] RECOVERY - mem101 SSH on mem101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:20:42] RECOVERY - ns2 SSH on ns2 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:20:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.26, 9.14, 8.76 [20:21:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.52, 10.18, 9.68 [20:21:14] RECOVERY - db121 SSH on db121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:21:21] RECOVERY - mw112 SSH on mw112 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:22:17] RECOVERY - db101 SSH on db101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:22:23] RECOVERY - cp31 SSH on cp31 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:22:50] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:23:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.71, 11.34, 9.82 [20:23:31] RECOVERY - cp30 SSH on cp30 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:23:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.05, 8.29, 8.50 [20:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:24:58] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [20:25:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.36, 9.89, 9.29 [20:25:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.71, 9.61, 9.41 [20:29:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.38, 9.68, 9.74 [20:30:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.02, 9.95, 9.45 [20:30:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.31, 8.97, 9.19 [20:33:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.40, 9.45, 9.32 [20:36:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.64, 9.02, 9.68 [20:37:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.26, 9.93, 9.76 [20:38:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:39:10] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 13 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [20:39:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.37, 8.88, 8.97 [20:42:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.94, 8.75, 8.92 [20:44:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.20, 9.31, 9.16 [20:45:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.66, 10.10, 9.82 [20:45:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:47:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.26, 8.51, 8.86 [20:48:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.02, 8.95, 8.90 [20:50:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.06, 9.01, 8.98 [20:51:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.87, 8.64, 8.40 [20:52:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.43, 8.77, 8.82 [20:53:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.43, 9.26, 9.16 [20:54:13] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.60, 8.04, 8.20 [20:55:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.35, 8.36, 8.65 [20:58:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.80, 7.93, 8.41 [21:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:05:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.26, 9.82, 10.00 [21:06:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:07:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.23, 9.74, 9.78 [21:07:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.79, 10.16, 10.10 [21:08:54] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:10:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.71, 9.03, 8.64 [21:10:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 10.34, 9.95 [21:13:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.55, 8.10, 8.36 [21:16:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.76, 9.85, 9.98 [21:18:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.57, 10.24, 9.42 [21:19:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.28, 9.13, 8.76 [21:23:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.21, 9.07, 9.85 [21:25:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.79, 9.88, 10.02 [21:27:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 9.46, 9.57 [21:30:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.84, 3.13 [21:30:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.51, 9.61, 9.66 [21:30:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.25, 9.46, 9.56 [21:31:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 9.32, 8.90 [21:32:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.52, 8.64, 8.48 [21:32:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 3.18, 2.99 [21:34:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.09, 9.10, 8.89 [21:34:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.05, 8.96, 8.61 [21:36:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 9.88, 9.70 [21:36:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.58, 9.66, 9.14 [21:37:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.63, 8.17, 8.37 [21:38:36] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 13 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [21:39:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.89, 8.44, 9.17 [21:39:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.26, 9.43, 9.13 [21:40:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.79, 9.62, 9.88 [21:41:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:41:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.46, 9.23, 9.29 [21:42:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.59, 11.09, 10.37 [21:44:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.74, 8.63, 9.06 [21:44:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.66, 10.40, 9.85 [21:45:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:47:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.07, 9.27, 9.50 [21:50:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 9.73, 9.65 [21:51:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.81, 8.66, 9.65 [21:53:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.04, 9.50, 9.26 [21:53:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.11, 9.56, 9.60 [21:55:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.14, 9.04, 9.14 [21:56:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.33, 9.49, 9.69 [22:01:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.72, 9.58, 9.51 [22:02:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 9.93, 9.85 [22:04:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.82, 8.63, 9.18 [22:04:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.01, 7.87, 8.50 [22:05:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.20, 10.32, 10.04 [22:07:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.71, 9.40, 9.12 [22:08:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.41, 9.50, 9.77 [22:09:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.73, 8.10, 8.69 [22:10:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.46, 9.97, 9.54 [22:11:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.87, 8.46, 8.65 [22:13:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 9.78, 9.58 [22:13:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.68, 9.94, 9.85 [22:14:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.26, 9.11, 8.63 [22:16:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.61, 10.61, 9.93 [22:16:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6f67c292e231 [22:16:33] [02miraheze/mw-config] 07Universal-Omega 036f67c29 - Enable more caching [22:16:35] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [22:16:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.72, 10.04, 9.28 [22:17:29] [02mw-config] 07Universal-Omega opened pull request 03#4407: Enable more caching - 13https://github.com/miraheze/mw-config/pull/4407 [22:18:04] [02mw-config] 07Universal-Omega edited pull request 03#4407: Enable more caching - 13https://github.com/miraheze/mw-config/pull/4407 [22:18:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.60, 10.14, 9.31 [22:18:39] miraheze/mw-config - Universal-Omega the build passed. [22:18:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.12, 9.98, 9.84 [22:19:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.09, 4.37, 3.63 [22:19:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.16, 9.51, 9.23 [22:19:42] [02mw-config] 07RhinosF1 commented on pull request 03#4407: Enable more caching - 13https://github.com/miraheze/mw-config/pull/4407#issuecomment-1015885876 [22:20:04] [02mw-config] 07Universal-Omega commented on pull request 03#4407: Enable more caching - 13https://github.com/miraheze/mw-config/pull/4407#issuecomment-1015886128 [22:20:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.51, 8.26, 8.50 [22:20:50] RhinosF1: how is it not compatible with ManageWiki? I don't understand how ManageWiki would interfere with sidebar cache... [22:21:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.97, 9.68, 9.26 [22:21:14] [02mw-config] 07RhinosF1 commented on pull request 03#4407: Enable more caching - 13https://github.com/miraheze/mw-config/pull/4407#issuecomment-1015886950 [22:22:08] CosmicAlpha: the presence of the links on the sidebar and the fact they change text [22:22:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.72, 3.48 [22:22:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.61, 9.56, 9.26 [22:22:34] Sidebar cache expects same for everyone [22:22:41] It only varies on language [22:24:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.31, 9.59, 9.28 [22:24:31] RhinosF1: well that can be fixed I think... [22:25:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.39, 9.09, 9.16 [22:26:18] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/a489e2d0bc17 [22:26:20] [02miraheze/ManageWiki] 07Universal-Omega 03a489e2d - Use SidebarBeforeOutput hook [22:26:21] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [22:26:23] [02ManageWiki] 07Universal-Omega opened pull request 03#324: Use SidebarBeforeOutput hook - 13https://github.com/miraheze/ManageWiki/pull/324 [22:26:30] RhinosF1: ^ [22:27:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.77, 9.34, 9.22 [22:27:12] That should add to the sidebar lastly, after the cache functions so it wouldn't ever be cached. [22:28:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.75, 3.18, 3.34 [22:28:15] Ok [22:28:21] Feel free to deploy [22:30:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.26, 8.54, 9.83 [22:30:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.03, 6.66, 8.10 [22:30:46] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:31:22] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.65, 7.42, 8.41 [22:33:01] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:20] miraheze/ManageWiki - Universal-Omega the build passed. [22:33:29] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:33:39] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:33:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6f67c292e231...f7eca22b1ee8 [22:33:50] [02miraheze/mw-config] 07Universal-Omega 03f7eca22 - Disable sidebar cache for now [22:33:51] [02mw-config] 07Universal-Omega synchronize pull request 03#4407: Enable more caching - 13https://github.com/miraheze/mw-config/pull/4407 [22:34:00] [02mw-config] 07Universal-Omega edited pull request 03#4407: Enable more caching - 13https://github.com/miraheze/mw-config/pull/4407 [22:34:57] miraheze/mw-config - Universal-Omega the build passed. [22:35:55] [02mw-config] 07Universal-Omega closed pull request 03#4407: Enable more caching - 13https://github.com/miraheze/mw-config/pull/4407 [22:35:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.42, 10.11, 9.41 [22:35:57] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config [22:35:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/962e4ef36668...92decb334527 [22:36:00] [02miraheze/mw-config] 07Universal-Omega 0392decb3 - Enable more caching (#4407) [22:36:01] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 [22:36:58] miraheze/mw-config - Universal-Omega the build passed. [22:39:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [22:39:48] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:42:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.77, 6.57, 8.33 [22:42:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:42:45] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.643 second response time [22:42:46] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [22:42:46] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.043 second response time [22:45:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.01, 8.82, 8.54 [22:47:20] !log [@test101] starting deploy of {'config': True} to skip [22:47:21] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:47:45] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:47:51] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:48:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [22:48:25] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:48:32] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:49:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.79, 7.13, 7.90 [22:49:22] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:01] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:10] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:13] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:51:52] !log [@mwtask111] starting deploy of {'config': True} to scsvg [22:52:26] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 34s [22:52:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [22:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:07] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.054 second response time [22:54:12] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.097 second response time [22:54:20] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.488 second response time [22:54:56] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.583 second response time [22:55:01] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.281 second response time [22:55:19] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.882 second response time [22:55:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 5.532 second response time [22:55:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:55:43] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.050 second response time [22:56:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:57:17] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.688 second response time [22:57:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:57:20] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.659 second response time [22:59:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:35] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:01:49] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:04] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:47] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:12] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [23:04:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:07:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:11:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [23:14:11] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 849s [23:15:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:42] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 940s [23:16:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [23:18:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.85, 6.84, 9.45 [23:20:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:20:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:21:35] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22194 bytes in 2.395 second response time [23:22:25] !log restarted nginx/php7.4 on mw122 [23:22:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:38] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.779 second response time [23:23:06] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.634 second response time [23:23:10] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.724 second response time [23:23:21] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.102 second response time [23:26:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.95, 6.89, 8.37 [23:29:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:43:13] [02puppet] 07Universal-Omega opened pull request 03#2363: Add webVideoTranscodePrioritized to jobQueueCollector - 13https://github.com/miraheze/puppet/pull/2363 [23:50:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.65, 7.99, 7.20 [23:53:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.39, 8.40, 7.51