[00:06:34] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.74, 2.99, 2.23 [00:06:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.76, 4.96, 4.67 [00:07:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.14, 7.98, 7.47 [00:07:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:08:32] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.43, 2.79, 2.25 [00:08:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.58, 6.05, 5.08 [00:09:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.17, 9.73, 8.13 [00:10:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.50, 3.18 [00:10:33] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.55, 5.56, 5.03 [00:12:31] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.61, 6.22, 5.30 [00:14:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:14:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.17, 3.14 [00:16:28] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.20, 5.52, 5.23 [00:18:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.74, 4.73, 4.96 [00:22:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 4.64, 3.70 [00:23:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [00:24:36] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 5.39, 6.02, 5.44 [00:25:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:26:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.75, 3.54 [00:26:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.34, 5.73, 5.40 [00:28:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.91, 3.61 [00:28:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.07, 6.55, 5.73 [00:30:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.68, 3.56 [00:32:29] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.07, 5.56, 5.53 [00:36:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.13, 6.48, 5.90 [00:42:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.74, 2.87, 3.32 [00:45:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.95, 6.10, 7.78 [00:47:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.71, 6.83, 7.83 [00:49:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.24, 7.41, 7.93 [00:51:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.81, 7.60, 7.93 [00:54:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.23, 7.18, 7.93 [00:55:20] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.12, 7.75, 7.67 [00:57:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.49, 7.33, 7.51 [00:58:16] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.67, 7.67, 7.94 [00:59:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:59:20] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 14.00, 9.34, 8.19 [01:00:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.07, 6.97, 7.64 [01:03:04] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.24, 5.30, 6.60 [01:03:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.83, 6.90, 7.45 [01:06:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.56, 5.05, 6.61 [01:07:20] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.76, 5.51, 6.73 [01:08:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.35, 4.70, 5.78 [01:09:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.53, 3.42 [01:10:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.28, 18.40, 23.29 [01:17:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.15, 4.53, 3.87 [01:17:56] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.29, 5.09, 5.30 [01:18:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:19:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.77, 5.07, 5.25 [01:20:36] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.96, 22.13, 22.59 [01:21:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.64, 7.84, 6.57 [01:21:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.60, 5.73, 5.47 [01:22:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.93, 21.61, 22.36 [01:23:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:23:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.97, 7.78, 6.70 [01:24:36] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.82, 22.89, 22.75 [01:25:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:26:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.89, 22.34, 22.60 [01:28:36] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.81, 24.06, 23.19 [01:29:04] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.66, 6.23, 6.38 [01:29:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.96, 3.75, 3.94 [01:30:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:30:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.80, 22.42, 22.70 [01:33:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:38:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:39:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.91, 5.51, 5.93 [01:41:38] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.02, 5.31, 5.78 [01:42:36] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.51, 22.20, 22.08 [01:44:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.54, 22.35, 22.13 [01:45:34] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.37, 5.85, 5.94 [01:48:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.84, 3.74 [01:49:31] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.85, 6.08, 5.99 [01:50:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:50:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.24, 24.64, 22.92 [01:50:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.90, 3.78 [01:51:20] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.33, 6.73, 5.58 [01:52:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.35, 1.01 [01:53:20] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.79, 6.32, 5.58 [01:54:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.79, 23.10, 22.78 [01:54:40] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.71, 3.73 [01:54:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.19, 0.99 [01:55:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:00:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.68, 3.75 [02:01:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.50, 5.38, 5.92 [02:02:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.91, 3.82 [02:04:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.64, 3.73 [02:04:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.61, 18.04, 20.32 [02:06:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.78, 3.76 [02:07:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.63, 1.94, 1.41 [02:09:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.81, 1.51, 1.32 [02:11:14] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.68, 3.97, 4.86 [02:16:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.96, 3.95 [02:18:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 4.01, 3.97 [02:22:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.76, 3.89 [02:23:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.18, 6.26, 5.24 [02:25:20] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.65, 5.85, 5.23 [02:26:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 4.00, 3.95 [02:34:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.58, 3.83 [02:44:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.20, 5.59, 4.89 [02:46:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.65, 5.44, 4.92 [02:48:53] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.25, 5.07, 4.85 [02:50:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 3.56, 3.51 [02:54:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.85, 3.68 [02:55:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:56:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 4.24, 3.85 [02:56:36] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.69, 25.72, 20.65 [02:56:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.78, 6.86, 5.72 [02:58:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.64, 3.67 [02:58:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.31, 5.49, 5.36 [03:00:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.89, 22.64, 20.61 [03:00:53] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.12, 4.74, 5.10 [03:02:37] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.43, 20.25, 20.01 [03:06:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 4.06, 3.87 [03:08:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.81, 3.56, 3.71 [03:16:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.65, 3.61 [03:18:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.51, 3.56 [03:20:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.16, 3.99, 3.72 [03:22:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.70, 3.65 [03:24:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.33, 4.47, 3.94 [03:28:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.95, 3.87 [03:40:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.02, 3.39 [03:44:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.52, 3.49 [03:46:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.22, 3.38 [03:50:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.34, 4.04, 3.66 [03:52:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.70, 3.59 [04:08:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.02, 4.12, 3.81 [04:10:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.59, 3.66 [04:22:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 3.08, 3.39 [04:43:46] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.32, 5.41, 4.46 [04:45:45] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.12, 4.94, 4.40 [04:58:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 4.00, 3.52 [05:00:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.78, 3.50 [05:03:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.79, 3.05, 3.29 [05:20:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.45, 4.03, 3.52 [05:22:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.80, 3.50 [05:28:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.88, 3.35, 3.39 [05:39:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.51, 3.40 [05:41:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.89, 3.56 [05:41:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.61, 4.89, 4.36 [05:43:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.79, 5.59, 4.68 [05:44:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.82, 3.62 [05:45:52] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.40, 5.15, 4.62 [05:46:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.56, 3.38, 1.94 [05:46:17] PROBLEM - wiki.creaturathegame.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.creaturathegame.com' expires in 15 day(s) (Tue 04 Jan 2022 05:42:13 GMT +0000). [05:46:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.89, 3.66 [05:47:50] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.93, 4.39, 4.41 [05:48:23] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fortis.wiki' expires in 15 day(s) (Tue 04 Jan 2022 05:44:00 GMT +0000). [05:48:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.91, 3.70 [05:49:05] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fortis.wiki' expires in 15 day(s) (Tue 04 Jan 2022 05:44:00 GMT +0000). [05:50:00] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.79, 1.96, 1.68 [05:50:12] PROBLEM - factorio-mods.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'factorio-mods.tk' expires in 15 day(s) (Tue 04 Jan 2022 05:47:30 GMT +0000). [05:50:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.77, 4.61, 3.98 [05:51:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.33, 1.46, 1.53 [05:52:19] PROBLEM - fortis.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fortis.wiki' expires in 15 day(s) (Tue 04 Jan 2022 05:44:00 GMT +0000). [05:52:25] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bluearchive.wiki' expires in 15 day(s) (Tue 04 Jan 2022 05:46:50 GMT +0000). [05:52:26] PROBLEM - wikidiffs.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikidiffs.ga' expires in 15 day(s) (Tue 04 Jan 2022 05:45:52 GMT +0000). [05:53:15] PROBLEM - ircwiki.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ircwiki.cf' expires in 15 day(s) (Tue 04 Jan 2022 05:48:25 GMT +0000). [05:56:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.51, 3.59, 3.75 [05:57:41] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.80, 5.54, 4.98 [06:00:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.45, 3.62 [06:01:38] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.91, 4.53, 4.73 [06:02:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.76, 3.73 [06:04:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 3.74, 3.71 [06:06:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.07, 3.11, 3.48 [06:07:02] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.18, 6.22, 3.86 [06:08:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 2.99, 3.39 [06:09:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.43, 6.75, 4.35 [06:12:53] !log rename axechainsawwiki to woodlandwiki [06:12:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:17:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.92, 6.58, 5.33 [06:19:28] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JDD08 [06:19:29] [02miraheze/dns] 07Reception123 03936ceb5 - add clockupwiki.org zone [06:20:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.30, 3.23 [06:23:36] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/JDDEj [06:23:38] [02miraheze/ssl] 07Reception123 030a02c9b - add wiki.meower.org cert [06:23:39] [02miraheze/ssl] 07Reception123 03573d827 - add clockupwiki.org cert [06:24:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 3.80, 3.43 [06:26:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:26:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.83, 3.50 [06:28:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.14, 3.64 [06:30:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 3.69, 3.54 [06:30:36] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ysmwikiwiki --username-prefix zhwikipedia --no-updates --report 1 /home/reception/Wikipedia-20211212072200.xml && sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ysmwikiwiki --username-prefix zhwikipedia --no-updates --report 1 /home/reception/Wikipedia-20211212072638.xml && sudo -u www-data php [06:30:37] /srv/mediawiki/w/maintenance/importDump.php --wiki=ysmwikiwiki --username-prefix zhwikipedia --no-updates --report 1 /home/reception/Wikipedia-20211214112200.xml [06:30:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:36:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.55, 3.51 [06:38:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.56, 3.52 [06:40:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.81, 3.61 [06:42:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.35, 3.24, 3.42 [06:43:02] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.85, 7.21, 6.16 [06:45:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.60, 6.93, 6.18 [06:46:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 3.17, 3.37 [06:47:02] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.16, 7.22, 6.37 [06:49:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.26, 6.99, 6.37 [06:50:56] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=fanonhubwiki --username-prefix ficreation --report 1 --no-updates home/reception/ficreationwiki_xml_d3f3e37ea661d156c84e.xml (START) [06:50:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 3.82, 3.58 [06:50:59] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=fanonhubwiki --username-prefix ficreation --report 1 --no-updates home/reception/ficreationwiki_xml_d3f3e37ea661d156c84e.xml (END - exit=256) [06:51:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:51:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:51:09] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=fanonhubwiki --username-prefix ficreation --report 1 --no-updates /home/reception/ficreationwiki_xml_d3f3e37ea661d156c84e.xml (START) [06:51:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.94, 3.14, 3.36 [06:56:49] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.15, 6.63, 6.49 [07:12:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:16:49] PROBLEM - db12 APT on db12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:17:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:17:18] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.64, 7.24, 6.50 [07:17:57] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=fanonhubwiki --username-prefix ficreation --report 1 --no-updates /home/reception/ficreationwiki_xml_d3f3e37ea661d156c84e.xml (END - exit=256) [07:18:07] PROBLEM - db12 Current Load on db12 is CRITICAL: CRITICAL - load average: 44.77, 18.21, 9.30 [07:18:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:46] RECOVERY - db12 APT on db12 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [07:19:15] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.97, 7.69, 6.77 [07:23:10] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 3.61, 5.86, 6.25 [07:24:01] PROBLEM - db12 Current Load on db12 is WARNING: WARNING - load average: 1.41, 6.59, 6.83 [07:26:01] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 1.50, 4.93, 6.19 [07:52:41] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.18, 4.84, 4.10 [07:54:36] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.58, 4.01, 3.89 [08:02:55] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:03:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [08:03:09] PROBLEM - cp30 ferm_active on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:03:16] PROBLEM - ping6 on cp30 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:03:16] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:03:22] PROBLEM - cp30 APT on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:03:35] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:03:40] PROBLEM - cp30 PowerDNS Recursor on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:03:42] PROBLEM - cp30 SSH on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:03:54] PROBLEM - ping4 on cp30 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:03:57] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:03] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:07] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [08:04:19] PROBLEM - cp30 HTTPS on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:04:19] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:24] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:28] PROBLEM - cp30 NTP time on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:36] PROBLEM - cp30 Stunnel Http for mon2 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:39] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:44] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:04:53] PROBLEM - cp30 conntrack_table_size on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:05:49] PROBLEM - Host cp30 is DOWN: PING CRITICAL - Packet loss = 100% [08:07:23] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:30] PROBLEM - dreamsit.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:14:12] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Jan 2022 03:39:45 GMT +0000. [08:20:59] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.97, 1.91, 1.48 [08:22:54] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.75, 1.56, 1.40 [08:39:35] RECOVERY - Host cp30 is UP: PING OK - Packet loss = 0%, RTA = 81.79 ms [08:39:37] RECOVERY - ping6 on cp30 is OK: PING OK - Packet loss = 0%, RTA = 76.94 ms [08:39:40] RECOVERY - ping4 on cp30 is OK: PING OK - Packet loss = 0%, RTA = 77.93 ms [08:39:41] RECOVERY - cp30 PowerDNS Recursor on cp30 is OK: DNS OK: 0.151 seconds response time. miraheze.org returns 149.56.141.75,2607:5300:201:3100::5ebc [08:39:46] RECOVERY - cp30 APT on cp30 is OK: APT OK: 17 packages available for upgrade (0 critical updates). [08:40:07] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 35% [08:40:19] RECOVERY - cp30 NTP time on cp30 is OK: NTP OK: Offset -0.006643176079 secs [08:40:30] RECOVERY - cp30 ferm_active on cp30 is OK: OK ferm input default policy is set [08:40:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.50, 0.17, 0.06 [08:40:42] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 15102 MB (38% inode=97%); [08:40:43] RECOVERY - cp30 SSH on cp30 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [08:41:05] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15860 bytes in 0.314 second response time [08:41:08] RECOVERY - cp30 conntrack_table_size on cp30 is OK: OK: nf_conntrack is 0 % full [08:41:09] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15871 bytes in 0.482 second response time [08:41:13] RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3123 bytes in 0.590 second response time [08:41:32] RECOVERY - cp30 Stunnel Http for mon2 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 35408 bytes in 0.386 second response time [08:41:43] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15871 bytes in 0.324 second response time [08:41:43] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15871 bytes in 0.325 second response time [08:41:52] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15870 bytes in 0.323 second response time [08:41:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:42:21] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [08:42:23] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15871 bytes in 0.334 second response time [08:42:24] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [08:42:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:43:52] RECOVERY - dreamsit.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'dreamsit.com.br' will expire on Tue 01 Feb 2022 05:18:33 GMT +0000. [09:05:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [09:06:34] PROBLEM - ping6 on cp30 is CRITICAL: PING CRITICAL - Packet loss = 100% [09:06:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [09:07:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:08:37] RECOVERY - ping6 on cp30 is OK: PING OK - Packet loss = 0%, RTA = 76.42 ms [09:08:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:12:10] PROBLEM - db13 APT on db13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:14:09] RECOVERY - db13 APT on db13 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [09:19:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:24:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:09:52] [02dns] 07RhinosF1 opened pull request 03#233: mediawiki: failover deployment & mwtask - 13https://git.io/JDyu6 [10:43:13] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JDyKv [10:43:15] [02miraheze/mw-config] 07RhinosF1 0318e570e - LS: setup betawiki [10:43:16] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [10:43:18] [02mw-config] 07RhinosF1 opened pull request 03#4291: LS: setup betawiki - 13https://git.io/JDyKJ [10:44:19] miraheze/mw-config - RhinosF1 the build passed. [10:51:04] [02dns] 07RhinosF1 opened pull request 03#234: Create betaheze.org - 13https://git.io/JDyiT [10:52:07] Reception123: ^ [10:52:19] https://phabricator.miraheze.org/T8399#171404 needs assigning to me when done [10:52:19] [url] ⚓ T8399 Move Sysadmin testwiki (test3wiki) to a custom domain | phabricator.miraheze.org [10:55:28] [02dns] 07RhinosF1 synchronize pull request 03#234: Create betaheze.org - 13https://git.io/JDyiT [10:58:40] RhinosF1: oh heh, since when did we purchase that? [10:58:53] Reception123: 5 minutes ago [10:58:56] thank Agent [10:59:02] cool, thanks Agent ! [10:59:07] [02dns] 07Reception123 closed pull request 03#234: Create betaheze.org - 13https://git.io/JDyiT [10:59:09] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JDyXO [10:59:10] [02miraheze/dns] 07RhinosF1 03ef960e2 - Create betaheze.org (#234) [10:59:16] np [11:00:48] Reception123: also merge https://github.com/miraheze/puppet/pull/2162/files [11:00:48] [url] deploy: support 2 DCs by RhinosF1 · Pull Request #2162 · miraheze/puppet · GitHub | github.com [11:01:37] Reception123: please create the SSL but leave MW side to me [11:01:45] as wildcard on *. [11:02:50] [02puppet] 07Reception123 closed pull request 03#2162: deploy: support 2 DCs - 13https://git.io/JD11R [11:02:51] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JDyXb [11:02:53] [02miraheze/puppet] 07RhinosF1 03d86c279 - deploy: support 2 DCs (#2162) [11:03:04] RhinosF1: so do we just want a regular betaheze.org SSL? [11:03:30] Reception123: wildcard across betaheze.org,*.betaheze.org pls [11:03:39] but don't add it to a wiki [11:03:42] i'll do that [11:05:22] time for wildcard fun [11:06:20] hmm, I never remember if you have to do both or not [11:06:22] i.e. [11:06:31] /root/ssl-certificate -d *.betaheze.org -g -w [11:06:32] OR [11:06:44] /root/ssl-certificate -d *.betaheze.org -s betaheze.org -g -w [11:06:51] /root/ssl-certificate -d *.betaheze.org -s betaheze.org -g -w [11:07:04] I guess the latter one can't hurt [11:09:49] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDyM2 [11:09:50] [02miraheze/dns] 07Reception123 03aacde01 - add acme challenge for betaheze.org [11:11:42] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDyMh [11:11:44] [02miraheze/dns] 07Reception123 038a0e41c - Update betaheze.org [11:12:22] RhinosF1: "- [ ] Copy Private Keys for renewals!" heh, don't even remind me! [11:12:36] I think I'm finally done with those now after 3 months [11:12:38] Reception123: yep! [11:12:51] RhinosF1: DNS syntax error though :( [11:13:04] ?[mNotice: /Stage[main]/Dns/Exec[gdnsd-syntax]/returns: # error: CNAME not allowed alongside other data at domainname 'betaheze.org.'?[0m [11:13:04] ?[mNotice: /Stage[main]/Dns/Exec[gdnsd-syntax]/returns: # fatal: rfc1035: Cannot load zonefile 'betaheze.org', failing?[0m [11:13:04] Reception123: where? [11:13:16] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [11:14:27] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [11:15:20] Reception123: does /usr/sbin/gdnsd checkconf say anything more usefuk [11:15:23] useful [11:16:40] RhinosF1: nope it's the same output [11:17:03] RhinosF1: I guess it's unusual to have CNAMEs under Wildcard services [11:17:07] since all our other domains are [11:17:08] @ DYNA geoip!cp [11:17:26] so my guess is that's what it's mad about [11:17:44] what about if we tried moving the CNAME to ;Services? [11:17:50] and use DYNA for WC services [11:19:15] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:19:43] uh what? [11:19:46] [02dns] 07RhinosF1 opened pull request 03#235: beta: use A/AAAA for wildcard - 13https://git.io/JDySk [11:19:55] Reception123: ^ [11:20:02] it doesn't run twice [11:20:10] so it'll be undeployed [11:20:13] ah [11:20:20] [02dns] 07Reception123 closed pull request 03#235: beta: use A/AAAA for wildcard - 13https://git.io/JDySk [11:20:22] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDySn [11:20:23] let's see if it prefers that [11:20:23] [02miraheze/dns] 07RhinosF1 033a52beb - beta: use A/AAAA for wildcard (#235) [11:20:27] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [11:21:53] seems happy now [11:22:23] yep [11:22:31] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDySH [11:22:32] [02miraheze/dns] 07Reception123 0354efcec - add ACME challenge to betaheze.org [11:23:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.79, 21.33, 17.76 [11:25:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.51, 19.89, 17.66 [11:27:10] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JDy9S [11:27:11] [02miraheze/ssl] 07Reception123 03bcc7848 - add betaheze.org wildcard [11:34:33] [02ssl] 07RhinosF1 opened pull request 03#443: Update certs.yaml - 13https://git.io/JDyQi [11:34:40] Reception123: ^ [11:35:43] miraheze/ssl - RhinosF1 the build has errored. [11:36:20] RhinosF1: ^ [11:36:35] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDy7L [11:36:36] [02miraheze/puppet] 07Reception123 035968dc3 - update matomo to 4.6.2 [11:36:44] ye [11:39:12] [02ssl] 07RhinosF1 synchronize pull request 03#443: Update certs.yaml - 13https://git.io/JDyQi [11:39:49] Reception123: ^ [11:40:22] miraheze/ssl - RhinosF1 the build has errored. [11:40:22] [02ssl] 07RhinosF1 synchronize pull request 03#443: Update certs.yaml - 13https://git.io/JDyQi [11:40:38] now [11:42:31] !log upgrade matomo to 4.6.2 (git pull; php) [11:42:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:45] [02ssl] 07Reception123 closed pull request 03#443: Update certs.yaml - 13https://git.io/JDyQi [11:42:47] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDy5I [11:42:48] [02miraheze/ssl] 07RhinosF1 03d762281 - Update certs.yaml (#443) [11:43:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.57, 2.94 [11:45:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.35, 2.93 [11:45:35] [02dns] 07RhinosF1 opened pull request 03#236: Update betaheze.org - 13https://git.io/JDy5y [11:45:46] Reception123: pls deploy ^ [11:48:15] [02dns] 07Reception123 closed pull request 03#236: Update betaheze.org - 13https://git.io/JDy5y [11:48:16] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDydB [11:48:18] [02miraheze/dns] 07RhinosF1 03aca7c7f - Update betaheze.org (#236) [11:48:29] Reception123: lmk when deployed [11:50:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:51:52] PROBLEM - beta.betaheze.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - beta.betaheze.org reverse DNS resolves to test3.miraheze.org [11:51:57] PROBLEM - betaheze.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - betaheze.org reverse DNS resolves to test3.miraheze.org [11:52:04] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.35, 5.51, 4.64 [11:52:26] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_matomo] [11:52:34] RhinosF1: done [11:52:36] but rDNS mad [11:52:47] Reception123: ye I can fix that later [11:53:04] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [11:53:05] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [11:53:10] Reception123: so is puppet over the certificate [11:53:59] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.92, 4.64, 4.43 [11:55:11] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [11:55:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:55:34] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [11:56:06] [02ssl] 07RhinosF1 opened pull request 03#444: betaheze: fix - 13https://git.io/JDybn [11:56:16] Reception123: ^ [11:56:58] [02ssl] 07Reception123 closed pull request 03#444: betaheze: fix - 13https://git.io/JDybn [11:56:59] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDybg [11:57:01] [02miraheze/ssl] 07RhinosF1 030eeac64 - betaheze: fix (#444) [11:57:05] in that case we'll just wait for 58 for it to do its own thing [11:58:02] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [11:58:37] Reception123: ok, that should recover now [11:58:37] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [11:58:41] I claimed the task back [11:58:50] Will merge config + add to wiki later [12:00:44] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [12:01:16] ack [12:01:40] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [12:02:06] Reception123: it's up to you when we drop test3wiki [12:02:13] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[beta.betaheze.org],File[beta.betaheze.org_private] [12:02:21] But we won't carry it over as test whatever wiki [12:04:21] probably whenever we move to the new infra is a good time [12:08:23] Ye [12:12:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 6.79, 5.62 [12:12:18] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [12:14:04] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.58, 6.68, 5.71 [12:16:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.40, 6.99, 6.02 [12:16:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:17:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.11, 6.47, 5.96 [12:21:02] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:21:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:21:45] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:22:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.45, 3.05 [12:23:22] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:23:29] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:24:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.08, 2.96 [12:25:32] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [12:25:43] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:26:49] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [12:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:29:02] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:45:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.55, 4.83, 4.17 [12:49:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.36, 4.65, 4.23 [13:09:04] PROBLEM - wikilukas.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikilukas.tk could not be found [13:09:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.47, 18.69, 17.70 [13:11:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.82, 18.36, 17.70 [13:12:11] PROBLEM - www.wikilukas.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wikilukas.tk could not be found [13:16:30] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.60, 5.43, 4.95 [13:18:25] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.71, 4.91, 4.83 [13:31:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 3.43, 3.17 [13:33:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.40, 3.18 [13:35:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 3.03, 3.07 [13:50:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.35, 3.14 [13:52:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.14, 3.09 [14:19:49] PROBLEM - wiki.meower.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['amanda.ns.cloudflare.com.', 'augustus.ns.cloudflare.com.'], 'CNAME': 'miraheze.wgytcraft.cf.'} [14:36:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.19, 3.74, 3.10 [14:38:59] PROBLEM - www.mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mwcosmos.com' expires in 15 day(s) (Tue 04 Jan 2022 14:36:25 GMT +0000). [14:39:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:40:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.35, 3.43, 3.15 [14:41:21] PROBLEM - wiki.mirahezebots.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mirahezebots.org' expires in 15 day(s) (Tue 04 Jan 2022 14:38:28 GMT +0000). [14:41:46] PROBLEM - www.wiki.mirahezebots.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mirahezebots.org' expires in 15 day(s) (Tue 04 Jan 2022 14:38:28 GMT +0000). [14:42:02] PROBLEM - mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mwcosmos.com' expires in 15 day(s) (Tue 04 Jan 2022 14:36:25 GMT +0000). [14:42:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.20, 3.10 [14:44:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:46:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.69, 3.31 [14:47:17] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'datacrondatabase.com' expires in 15 day(s) (Tue 04 Jan 2022 14:39:19 GMT +0000). [14:47:37] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/d [14:47:37] ackages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: lcn.zfc.id.lv., lcn.zfc.id.lv. [14:48:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.05, 3.04, 3.12 [14:51:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.71, 3.61, 3.29 [14:53:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.49, 3.29 [14:55:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.19, 3.20 [15:01:04] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:11:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.10, 7.01, 6.18 [15:13:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.47, 6.44, 6.07 [15:25:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.41, 5.48, 4.61 [15:25:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.43, 3.25 [15:27:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.74, 5.01, 4.55 [15:27:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.62, 3.33 [15:29:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.52, 3.33 [15:31:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.24, 3.26 [15:37:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.08, 5.50, 4.93 [15:39:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.89, 5.22, 4.90 [15:41:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.50, 5.86, 5.16 [15:41:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 3.82, 3.48 [15:41:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.45, 7.27, 5.96 [15:42:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:42:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.18, 20.34, 18.91 [15:43:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.68, 3.48 [15:43:36] [02puppet] 07RhinosF1 opened pull request 03#2163: monitoring::ssl: allow domains pointing to test3 - 13https://git.io/JDS9a [15:43:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.77, 6.50, 5.86 [15:43:41] [02puppet] 07RhinosF1 edited pull request 03#2163: monitoring::ssl: allow domains pointing to test - 13https://git.io/JDS9a [15:43:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.61, 20.15, 19.05 [15:45:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.09, 5.83, 5.34 [15:45:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.85, 3.55 [15:45:27] Reception123: something like https://github.com/miraheze/puppet/pull/2163 should work [15:45:28] [url] monitoring::ssl: allow domains pointing to test by RhinosF1 · Pull Request #2163 · miraheze/puppet · GitHub | github.com [15:45:32] Not 100% on the regex [15:46:05] [02puppet] 07RhinosF1 commented on pull request 03#2163: monitoring::ssl: allow domains pointing to test - 13https://git.io/JDSHW [15:47:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:47:08] .mh meta SAL [15:47:08] https://meta.miraheze.org/wiki/SAL [15:47:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.74, 3.55 [15:47:33] CosmicAlpha: fyi if you deploy I've added support for new DC [15:47:52] Please shout if it's not working as expected (not showing servers or showing wrong ones) [15:48:34] New should be behind --servers=scsvg, =all will auto point to =ovlon (current servers) [15:49:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.82, 4.59, 4.94 [15:49:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.09, 3.70 [15:53:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.95, 3.78 [15:54:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.87, 3.75 [15:56:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.52, 3.64 [16:00:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.72, 3.70 [16:02:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:07:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:10:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.72, 3.74 [16:13:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:17:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 2.84, 3.29 [16:18:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:21:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.52, 3.49 [16:25:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.11, 3.34 [16:32:48] RhinosF1: OK [16:34:13] CosmicAlpha: betawiki now works [16:37:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.82, 5.79, 5.09 [16:37:52] RhinosF1: OK, and what are we doing with test3wiki? [16:37:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.98, 3.71, 3.42 [16:38:20] CosmicAlpha: being deleted once test3 goes [16:41:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.96, 5.70, 5.25 [16:41:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.27, 3.71, 3.50 [16:43:28] CosmicAlpha: I gave you admin+crat [16:44:15] CosmicAlpha: a review on https://github.com/miraheze/mw-config/pull/4291/files would be ncie [16:44:16] [url] LS: setup betawiki by RhinosF1 · Pull Request #4291 · miraheze/mw-config · GitHub | github.com [16:44:24] s/nci/nic [16:44:24] RhinosF1 meant to say: CosmicAlpha: a review on https://github.com/miraheze/mw-config/pull/4291/files would be nice [16:45:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.66, 4.60, 4.95 [16:45:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.91, 3.60 [16:49:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.61, 5.24, 5.14 [16:49:19] RhinosF1: left a comment. [16:51:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.67, 5.08, 5.10 [16:51:14] [02mw-config] 07RhinosF1 commented on pull request 03#4291: LS: setup betawiki - 13https://git.io/JD9O7 [16:51:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JD9Ob [16:51:19] [02miraheze/mw-config] 07Universal-Omega 035923ad6 - Update comment to betawiki [16:51:27] CosmicAlpha: replied [16:52:19] miraheze/mw-config - Universal-Omega the build passed. [16:52:55] RhinosF1: approved [16:52:57] CosmicAlpha: I need to create any databases that need so but I'll merge tonight [16:53:22] RhinosF1: ty for sysop+crat on betawiki [16:53:33] Np [16:54:23] !log [@test3] starting deploy of {'config': True} to skip [16:54:24] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [16:54:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:55:36] That's good [16:55:42] I didn't break skip [16:55:54] RhinosF1: can betawiki be moved to c3 (db12) rather than c4 (db13) [16:55:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 4.00, 3.85 [16:56:02] CosmicAlpha: if you access, please run puppet on 11 so you can catch if stupid [16:56:17] CosmicAlpha: Reception would have too but no objections [16:56:54] RhinosF1: no access, that's why the only thing I pushed was a change to comment. But if absolutely necessary I can get access. Otherwise I will in a couple hours. [16:57:16] CosmicAlpha: shouldn't be a worry no [16:57:31] It won't cause harm if it doesn't pick right servers [16:57:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.86, 4.42, 4.03 [16:58:10] Reception123: can you move betawiki to c3 (db12)? [16:58:24] Oh Reception isn't online. [17:01:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.72, 3.84 [17:03:27] it pinged him on Discord though [17:03:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.14, 4.36, 4.07 [17:05:10] Oh OK. [17:05:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.96, 3.96 [17:11:16] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JD9Cl [17:11:17] [02miraheze/landing] 07Universal-Omega 039de1c27 - i18n: Move "!" outside of link [17:11:17] !log [@mw11] starting deploy of {'config': True} to ovlon [17:11:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:11:28] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 10s [17:11:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:12:12] miraheze/landing - Universal-Omega the build passed. [17:12:19] Yey deploy worked [17:14:38] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9Ws [17:14:39] [02miraheze/landing] 07Universal-Omega 03167fdf8 - Update index.php [17:14:41] [02landing] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fxEQA [17:22:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:22:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.27, 6.94, 5.65 [17:23:07] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD98R [17:23:08] [02miraheze/landing] 07Universal-Omega 03b99eb66 - Update getTranslations.php [17:23:56] Reception123: 16:58:12 Reception123: can you move betawiki to c3 (db12)? [17:24:02] Domain is active [17:24:07] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.60, 7.12, 6.03 [17:24:10] beta.betaheze.org [17:24:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.57, 6.15, 5.49 [17:24:53] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD98j [17:24:54] [02miraheze/landing] 07Universal-Omega 03161fb3d - Update en.json [17:25:28] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD943 [17:25:30] [02miraheze/landing] 07Universal-Omega 0328c7784 - Update getTranslations.php [17:26:04] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.10, 6.11, 5.79 [17:26:29] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD94V [17:26:30] [02miraheze/landing] 07Universal-Omega 0341df046 - Update donate.php [17:26:50] [02landing] 07Universal-Omega opened pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [17:27:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:27:15] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD94M [17:27:17] [02miraheze/landing] 07Universal-Omega 03509b7a1 - Update getTranslations.php [17:27:18] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [17:27:44] miraheze/landing - Universal-Omega the build passed. [17:28:05] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD94d [17:28:06] [02miraheze/landing] 07Universal-Omega 03643cd8e - Update en.json [17:28:08] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [17:28:12] miraheze/landing - Universal-Omega the build passed. [17:29:01] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9Be [17:29:03] [02miraheze/landing] 07Universal-Omega 038ebbc63 - Update getTranslations.php [17:29:03] miraheze/landing - Universal-Omega the build passed. [17:29:04] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [17:29:44] RhinosF1: ^ FYI, thoughts on that? [17:29:58] miraheze/landing - Universal-Omega the build passed. [17:30:01] CosmicAlpha: if it works [17:30:28] Make sure things are appropriately escaped etc [17:31:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.14, 3.39 [17:32:07] RhinosF1: Yeah it uses the same functions that escape normal translations, with the exception of using a new function, which does replaces on $1 and $2 for things that won't be the same each year so i18n stays constant. [17:34:48] Ok [17:35:04] Can do, likely tomorrow [17:35:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.82, 6.52, 5.44 [17:36:39] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9Rx [17:36:40] [02miraheze/landing] 07Universal-Omega 034c16beb - Update qqq.json [17:36:42] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [17:36:53] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9Rj [17:36:55] [02miraheze/landing] 07Universal-Omega 03aea0d1b - Update en.json [17:36:56] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [17:37:13] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD90U [17:37:14] [02miraheze/landing] 07Universal-Omega 03493e13a - Update getTranslations.php [17:37:16] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [17:37:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.21, 6.77, 5.64 [17:37:36] miraheze/landing - Universal-Omega the build passed. [17:37:49] miraheze/landing - Universal-Omega the build passed. [17:38:09] miraheze/landing - Universal-Omega the build passed. [17:38:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.00, 7.36, 6.02 [17:38:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:39:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.39, 20.56, 18.74 [17:39:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.54, 6.68, 5.75 [17:39:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.75, 7.25, 6.19 [17:39:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.72, 4.35, 3.77 [17:40:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.19, 7.03, 6.07 [17:41:41] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.97, 6.23, 5.19 [17:41:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.64, 7.30, 6.34 [17:43:36] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.57, 5.67, 5.09 [17:43:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.86, 8.57, 6.94 [17:45:21] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.32, 8.22, 6.68 [17:46:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 6.27, 5.68 [17:47:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.22, 7.91, 6.75 [17:47:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.36, 20.04, 19.42 [17:48:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.15, 5.99, 5.66 [17:49:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.75, 4.74, 4.88 [17:49:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.65, 7.97, 7.28 [17:50:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.97, 8.23, 6.99 [17:52:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.82, 7.68, 6.94 [17:53:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.30, 21.08, 20.06 [17:55:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.41, 6.36, 5.69 [17:55:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.09, 19.99, 19.78 [17:55:22] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.79, 1.77, 1.20 [17:57:01] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.03, 6.78, 6.72 [17:57:09] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.99, 6.97, 6.45 [17:57:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.16, 6.14, 5.68 [17:57:21] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.93, 1.43, 1.14 [17:58:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:59:06] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.24, 6.62, 6.37 [18:00:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.36, 6.79, 6.76 [18:01:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.65, 3.95 [18:03:50] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.20, 7.20, 7.04 [18:05:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.75, 5.15, 4.79 [18:05:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:05:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.38, 20.98, 19.95 [18:05:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.01, 6.94, 6.98 [18:07:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.77, 5.67, 5.05 [18:07:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.88, 20.13, 19.80 [18:07:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.26, 6.43, 6.79 [18:09:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.89, 7.08, 6.28 [18:11:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.58, 4.84, 4.87 [18:13:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.18, 21.25, 20.20 [18:13:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 7.51, 6.62 [18:13:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.92, 7.45, 6.70 [18:13:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.94, 8.48, 7.43 [18:13:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.95, 4.06, 3.89 [18:13:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.87, 7.61, 6.87 [18:15:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.44, 19.84, 19.83 [18:15:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.94, 6.57, 6.38 [18:15:41] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.36, 6.63, 6.49 [18:15:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.83, 7.00, 7.02 [18:15:55] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.19, 6.90, 6.69 [18:17:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.14, 6.51, 6.56 [18:17:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.64, 3.77 [18:19:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.42, 7.33, 6.29 [18:19:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 4.04, 3.91 [18:21:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.93, 20.97, 20.21 [18:21:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.98, 7.12, 6.33 [18:21:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.85, 7.41, 7.13 [18:21:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.98, 3.93 [18:23:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.43, 5.45, 5.03 [18:23:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.24, 19.87, 19.92 [18:23:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.97, 6.66, 6.25 [18:23:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.10, 6.42, 6.79 [18:23:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 4.09, 3.96 [18:25:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:25:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.91, 3.91 [18:26:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:27:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.50, 5.83, 5.30 [18:27:23] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.91, 6.66, 6.35 [18:29:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.11, 5.72, 5.31 [18:29:20] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.57, 6.33, 6.27 [18:29:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.17, 7.62, 7.13 [18:29:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 4.30, 4.06 [18:30:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.90, 21.05, 20.42 [18:31:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.86, 5.13, 5.15 [18:31:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:31:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.06, 7.10, 7.00 [18:31:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.79, 3.90 [18:32:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.70, 19.52, 19.94 [18:32:40] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9Kd [18:32:42] [02miraheze/landing] 07Universal-Omega 0386498bc - Update getTranslations.php [18:32:43] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [18:33:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.11, 5.49, 5.25 [18:33:38] miraheze/landing - Universal-Omega the build passed. [18:33:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.85, 6.29, 6.71 [18:35:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:35:48] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.03, 21.43, 20.78 [18:37:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.71, 5.74, 5.42 [18:37:09] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD96z [18:37:11] [02miraheze/landing] 07Universal-Omega 03700b0c3 - Update getTranslations.php [18:37:12] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [18:38:01] [02landing] 07Universal-Omega edited pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [18:38:10] miraheze/landing - Universal-Omega the build passed. [18:39:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.97, 18.29, 19.76 [18:40:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:45:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.52, 4.45, 4.94 [18:45:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 2.97, 3.38 [18:48:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:51:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 6.17, 5.75 [18:51:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.61, 6.85, 6.07 [18:53:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.65, 5.19, 4.93 [18:53:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:53:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.62, 6.04, 5.75 [18:53:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.33, 6.54, 6.04 [18:55:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.36, 5.00, 4.89 [18:58:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:59:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.45, 5.90, 5.23 [18:59:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.54, 21.16, 19.89 [19:03:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.39, 19.41, 19.58 [19:03:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:04:52] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9Me [19:04:53] [02miraheze/landing] 07Universal-Omega 039091b99 - Update donate.php [19:04:55] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [19:05:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.60, 5.65, 5.48 [19:05:48] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9Mm [19:05:49] miraheze/landing - Universal-Omega the build passed. [19:05:50] [02miraheze/landing] 07Universal-Omega 03de6e104 - Update index.php [19:05:51] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [19:06:11] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9MO [19:06:12] [02miraheze/landing] 07Universal-Omega 035652732 - Update getTranslations.php [19:06:14] [02landing] 07Universal-Omega synchronize pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [19:06:48] miraheze/landing - Universal-Omega the build passed. [19:07:14] miraheze/landing - Universal-Omega the build passed. [19:09:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:11:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.95, 4.58, 5.07 [19:11:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.32, 3.26 [19:14:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:14:34] [02landing] 07Universal-Omega closed pull request 03#73: Add localisation to sitenotice - 13https://git.io/JD94K [19:14:36] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JD9DK [19:14:38] [02miraheze/landing] 07Universal-Omega 03a66b9aa - Add localisation to sitenotice (#73) [19:14:39] [02landing] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fxEQA [19:14:41] [02miraheze/landing] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:15:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.13, 6.12, 5.59 [19:15:11] [02puppet] 07RhinosF1 synchronize pull request 03#2163: monitoring::ssl: allow domains pointing to test - 13https://git.io/JDS9a [19:15:25] [02puppet] 07RhinosF1 commented on pull request 03#2163: monitoring::ssl: allow domains pointing to test - 13https://git.io/JD9Dy [19:15:36] miraheze/landing - Universal-Omega the build passed. [19:15:58] CosmicAlpha: check my puppet pr [19:17:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.29, 6.02, 5.60 [19:19:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.15, 5.39, 5.20 [19:21:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.98, 6.63, 5.95 [19:21:50] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.19, 4.86, 5.03 [19:23:00] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.63, 7.44, 6.31 [19:23:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:23:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.48, 7.39, 6.57 [19:23:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.82, 3.48 [19:24:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 7.25, 6.37 [19:24:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.36, 6.85, 6.05 [19:25:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.63, 3.45 [19:26:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.66, 6.25, 5.92 [19:27:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.99, 9.24, 7.40 [19:27:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.85, 2.91, 3.20 [19:28:50] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.06, 6.34, 6.21 [19:29:48] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.51, 20.57, 19.91 [19:30:07] PROBLEM - oecumene.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 7.04, 6.36 [19:31:42] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.16, 19.43, 19.58 [19:31:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.29, 7.62, 7.16 [19:32:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.65, 6.78, 6.36 [19:33:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:34:30] [02puppet] 07Universal-Omega reviewed pull request 03#2163 commit - 13https://git.io/JD99a [19:34:55] RhinosF1: ^ [19:35:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.45, 7.40, 7.13 [19:37:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.54, 6.94, 6.28 [19:37:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.25, 6.64, 6.89 [19:38:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 3.74, 3.43 [19:39:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.40, 6.30, 6.74 [19:41:35] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.29, 7.98, 6.83 [19:43:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.59, 8.00, 6.99 [19:43:47] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.229 second response time [19:43:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.03, 7.10, 6.96 [19:43:58] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 3.825 second response time [19:43:59] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 2.157 second response time [19:43:59] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 3.685 second response time [19:44:07] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.007 second response time [19:44:07] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.248 second response time [19:44:08] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.010 second response time [19:44:09] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.017 second response time [19:44:13] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 2.844 second response time [19:44:14] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.131 second response time [19:44:21] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.042 second response time [19:44:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.86, 3.58 [19:44:23] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.411 second response time [19:44:28] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.273 second response time [19:44:34] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.287 second response time [19:44:38] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.365 second response time [19:45:02] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.267 second response time [19:45:03] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 2.673 second response time [19:45:03] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.117 second response time [19:45:05] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 3.109 second response time [19:45:12] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.010 second response time [19:45:22] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 0.395 second response time [19:45:25] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 1.882 second response time [19:45:25] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.62, 8.24, 7.19 [19:45:33] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.032 second response time [19:45:52] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 251 bytes in 0.259 second response time [19:46:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.14, 3.71 [19:46:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.57, 6.98, 6.17 [19:46:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:46:33] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.17, 7.32, 6.16 [19:48:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.83, 3.66 [19:48:30] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.19, 6.71, 6.09 [19:49:16] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.38, 7.88, 7.33 [19:50:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.04, 7.39, 6.68 [19:51:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.30, 5.03, 4.58 [19:51:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.27, 22.85, 20.59 [19:51:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.88, 3.70 [19:52:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.27, 7.15, 6.67 [19:52:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 7.01, 6.50 [19:53:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.64, 3.64 [19:54:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.52, 7.08, 6.58 [19:55:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.28, 4.86, 4.61 [19:55:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.53, 22.20, 20.91 [19:56:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:56:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.19, 6.44, 6.41 [19:57:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.87, 3.70 [19:58:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.85, 6.29, 6.35 [19:58:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.18, 6.77, 6.67 [19:59:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.79, 3.69 [20:01:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.56, 5.84, 6.69 [20:01:26] PROBLEM - oecumene.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['fortaleza.porkbun.com.', 'maceio.porkbun.com.', 'salvador.porkbun.com.', 'curitiba.porkbun.com.'], 'CNAME': None} [20:03:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.90, 2.92, 3.37 [20:05:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.63, 19.74, 20.40 [20:06:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:08:15] dmehus: will you please see your very old PM [20:08:22] PROBLEM - oecumene.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - oecumene.org reverse DNS resolves to ec2-44-227-76-166.us-west-2.compute.amazonaws.com [20:08:34] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9dV [20:08:36] [02miraheze/landing] 07Universal-Omega 0342d1352 - fix spaces [20:08:37] [02landing] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fxEQA [20:08:39] [02landing] 07Universal-Omega opened pull request 03#74: fix spaces - 13https://git.io/JD9dw [20:09:42] miraheze/landing - Universal-Omega the build passed. [20:11:05] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9FL [20:11:07] [02miraheze/landing] 07Universal-Omega 03afc2db5 - Update index.php [20:11:08] [02landing] 07Universal-Omega synchronize pull request 03#74: fix spaces - 13https://git.io/JD9dw [20:11:30] [02landing] 07Universal-Omega closed pull request 03#74: fix spaces - 13https://git.io/JD9dw [20:11:32] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JD9FY [20:11:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:11:33] [02miraheze/landing] 07Universal-Omega 03e324308 - fix spaces (#74) [20:11:35] [02landing] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fxEQA [20:11:36] [02miraheze/landing] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [20:11:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.68, 7.39, 7.93 [20:11:55] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.334 second response time [20:11:56] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.009 second response time [20:11:56] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.707 second response time [20:12:07] miraheze/landing - Universal-Omega the build passed. [20:12:07] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.066 second response time [20:12:07] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.341 second response time [20:12:08] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.024 second response time [20:12:09] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.019 second response time [20:12:10] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.262 second response time [20:12:14] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.018 second response time [20:12:21] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.006 second response time [20:12:23] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.515 second response time [20:12:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.342 second response time [20:12:31] miraheze/landing - Universal-Omega the build passed. [20:12:34] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.319 second response time [20:12:38] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.367 second response time [20:13:02] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.380 second response time [20:13:03] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.349 second response time [20:13:03] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.008 second response time [20:13:05] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.335 second response time [20:13:12] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.027 second response time [20:13:22] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.320 second response time [20:13:22] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.031 second response time [20:13:33] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.032 second response time [20:13:38] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.009 second response time [20:13:56] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.326 second response time [20:14:13] how did spaces cause fatals on landing before? (did not happen on test3) all I did to fix is edit on mw11 directly, backspace the spaces, re-add spaces in editor, save, and worked, copied and pasted the changes to github and fixed... this makes no sense to me. [20:19:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.56, 3.40 [20:20:15] CosmicAlpha: probably tech reminding you it's nearly Christmas so we're gonna have fun [20:21:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 4.05, 3.60 [20:21:50] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.98, 5.50, 6.63 [20:22:14] RECOVERY - oecumene.org - reverse DNS on sslhost is OK: SSL OK - oecumene.org reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [20:28:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.92, 3.73 [20:31:08] PROBLEM - oecumene.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - oecumene.org reverse DNS resolves to ec2-44-227-76-166.us-west-2.compute.amazonaws.com [20:32:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.62, 3.61 [20:34:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.42, 3.54 [20:38:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.38, 3.07, 3.39 [20:42:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.40, 3.46 [20:42:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.76, 6.59, 5.65 [20:43:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.17, 21.63, 18.98 [20:43:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 6.69, 5.81 [20:43:50] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.36, 4.72, 4.23 [20:44:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.58, 3.51 [20:44:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.44, 6.20, 5.61 [20:45:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:45:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.98, 21.61, 19.28 [20:45:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.30, 6.08, 5.69 [20:45:45] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.65, 4.42, 4.18 [20:47:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.28, 20.23, 19.05 [20:47:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.90, 3.67 [20:48:04] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.25, 7.02, 5.89 [20:49:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.08, 3.76 [20:50:29] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.89, 7.91, 6.61 [20:50:36] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.05, 7.69, 6.53 [20:51:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.49, 20.59, 19.52 [20:52:04] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.75, 6.84, 6.09 [20:52:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.29, 7.04, 6.43 [20:52:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.88, 6.50, 5.94 [20:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.97, 3.80 [20:54:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.68, 7.67, 6.77 [20:55:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 4.06, 3.86 [20:56:03] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.41, 6.07, 5.98 [20:56:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.01, 6.71, 6.53 [20:56:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.47, 6.64, 6.16 [20:57:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.60, 19.53, 19.51 [20:57:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.80, 3.77 [20:58:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.58, 7.96, 7.02 [20:58:53] RECOVERY - oecumene.org - reverse DNS on sslhost is OK: SSL OK - oecumene.org reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [21:00:12] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.67, 7.38, 6.91 [21:00:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.41, 6.94, 6.43 [21:02:08] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.42, 6.75, 6.74 [21:02:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.70, 6.40, 6.28 [21:03:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JD9jf [21:03:24] [02miraheze/mediawiki] 07Universal-Omega 030ee3913 - labeler: use `--rebase` [21:03:25] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:03:27] [02mediawiki] 07Universal-Omega opened pull request 03#4678: labeler: use `--rebase` - 13https://git.io/JD9jJ [21:05:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:06:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.12, 8.08, 7.26 [21:06:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 7.41, 6.77 [21:06:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.16, 7.68, 6.82 [21:07:48] PROBLEM - oecumene.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - oecumene.org reverse DNS resolves to ec2-44-227-76-166.us-west-2.compute.amazonaws.com [21:07:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.98, 3.79 [21:08:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:08:04] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.12, 6.80, 5.97 [21:08:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.93, 7.16, 6.78 [21:09:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.66, 8.29, 6.73 [21:09:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.75, 3.72 [21:09:57] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.69, 7.33, 7.18 [21:10:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.25, 6.37, 6.53 [21:10:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.44, 7.61, 7.05 [21:11:02] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.84, 7.66, 6.06 [21:11:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.11, 7.67, 6.69 [21:12:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.43, 6.55, 6.73 [21:13:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:15:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.98, 7.66, 6.47 [21:15:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:15:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.88, 7.04, 7.00 [21:15:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.82, 3.73 [21:17:01] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.20, 7.74, 6.64 [21:17:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.76, 7.11, 7.02 [21:19:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.87, 7.42, 6.65 [21:19:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.76, 6.52, 6.59 [21:19:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.05, 7.59, 7.21 [21:20:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.69, 7.22, 6.52 [21:21:40] PROBLEM - oecumene.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['fortaleza.porkbun.com.', 'curitiba.porkbun.com.', 'salvador.porkbun.com.', 'maceio.porkbun.com.'], 'CNAME': None} [21:21:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.57, 7.24, 7.13 [21:22:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.63, 6.57, 6.38 [21:25:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:26:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.12, 7.60, 7.33 [21:27:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.74, 6.23, 6.46 [21:28:37] PROBLEM - oecumene.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - oecumene.org reverse DNS resolves to ec2-44-227-76-166.us-west-2.compute.amazonaws.com [21:31:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.79, 6.36, 6.75 [21:31:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.84, 3.96 [21:34:03] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.70, 5.51, 6.57 [21:43:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.34, 3.51 [21:45:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.01, 3.36 [21:51:18] [02dns] 07Universal-Omega opened pull request 03#237: Remove expired oecumene.org - 13https://git.io/JDHTe [21:51:48] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.54, 21.22, 18.75 [21:51:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.32, 7.96, 6.89 [21:51:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.60, 3.48 [21:52:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:52:06] [02ssl] 07Universal-Omega opened pull request 03#445: Remove expired oecumene.org - 13https://git.io/JDHTL [21:52:27] [02ssl] 07Universal-Omega synchronize pull request 03#445: Remove expired oecumene.org - 13https://git.io/JDHTL [21:53:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.70, 6.98, 6.66 [21:53:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.08, 6.57, 4.78 [21:53:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.32, 3.39 [21:55:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.46, 22.47, 19.88 [21:55:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.10, 6.10, 6.38 [21:57:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:57:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.28, 5.54, 4.77 [21:59:24] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.89, 19.68, 19.27 [22:01:36] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.94, 5.73, 4.98 [22:01:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.39, 3.36 [22:03:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.78, 5.40, 4.95 [22:03:49] [02mediawiki] 07Universal-Omega closed pull request 03#4678: labeler: use `--rebase` - 13https://git.io/JD9jJ [22:03:51] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [22:03:53] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [22:05:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.58, 4.73, 4.75 [22:05:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.82, 3.53 [22:07:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.54, 3.46 [22:11:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.84, 3.59 [22:13:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.22, 3.28, 3.42 [22:15:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 2.95, 3.27 [22:19:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:21:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.69, 5.22, 4.90 [22:22:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.98, 21.20, 19.42 [22:23:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.00, 4.84, 4.80 [22:24:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.27, 20.16, 19.24 [22:29:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.46, 6.60, 5.41 [22:31:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.63, 21.32, 19.97 [22:33:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.11, 5.78, 5.33 [22:33:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.17, 21.73, 20.22 [22:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:35:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.52, 6.91, 5.78 [22:37:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:38:50] !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=betawiki /srv/mediawiki/w/extensions/CentralNotice/CentralNotice.sql [22:38:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:50] [02mw-config] 07RhinosF1 closed pull request 03#4291: LS: setup betawiki - 13https://git.io/JDyKJ [22:41:51] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [22:41:53] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [22:41:54] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDHmK [22:41:56] [02miraheze/mw-config] 07RhinosF1 037def265 - LS: setup betawiki (#4291) [22:42:54] miraheze/mw-config - RhinosF1 the build passed. [22:43:11] !log [@test3] starting deploy of {'config': True} to skip [22:43:12] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:43:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.93, 6.55, 5.46 [22:43:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.17, 7.07, 5.79 [22:45:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.56, 7.58, 6.07 [22:47:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.86, 6.95, 5.90 [22:47:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.75, 7.13, 6.08 [22:49:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.53, 3.29 [22:49:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.35, 6.46, 5.85 [22:51:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.11, 3.10, 3.17 [22:51:50] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.28, 6.53, 6.11 [22:57:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:00:01] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:04:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.52, 8.95, 6.97 [23:05:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.14, 7.29, 6.61 [23:06:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:06:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.79, 7.86, 6.81 [23:07:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.52, 3.30 [23:08:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:08:15] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.24, 19.31, 16.15 [23:09:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.30, 5.92, 6.23 [23:10:16] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.40, 19.74, 16.70 [23:10:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.69, 6.66, 6.58 [23:11:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.63, 2.77, 3.07 [23:11:57] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb [23:12:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:12:21] !log [@mw11] starting deploy of {'config': True} to ovlon [23:12:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:06] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 44s [23:13:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:53] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [23:14:07] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:14:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:14:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.32, 8.08, 7.17 [23:14:32] [02puppet] 07RhinosF1 synchronize pull request 03#2163: monitoring::ssl: allow domains pointing to test - 13https://git.io/JDS9a [23:16:30] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.60, 6.66, 6.76 [23:18:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.24, 6.95, 6.01 [23:19:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.42, 7.42, 6.77 [23:20:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.93, 6.45, 5.93 [23:20:21] [02ssl] 07Universal-Omega opened pull request 03#446: Update test1 and test2 redirects to point at beta.betaheze.org - 13https://git.io/JDHnB [23:20:27] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.69, 9.15, 7.76 [23:22:35] [02ssl] 07Universal-Omega synchronize pull request 03#446: Update test1 and test2 redirects to point at beta.betaheze.org - 13https://git.io/JDHnB [23:22:56] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.21, 6.90, 5.74 [23:23:07] [02ssl] 07Universal-Omega edited pull request 03#446: Update test1 and test2 redirects to point at beta.betaheze.org - 13https://git.io/JDHnB [23:23:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.71, 6.97, 6.82 [23:23:54] RhinosF1: ^ [23:24:01] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.70, 9.71, 7.40 [23:24:05] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:24:53] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 7.37, 6.08 [23:25:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.77, 6.56, 6.68 [23:26:01] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [23:26:16] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.90, 21.03, 19.11 [23:26:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.53, 6.29 [23:27:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.25, 7.77, 7.12 [23:28:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.79, 7.35, 7.61 [23:28:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.20, 6.51, 6.04 [23:28:55] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1733s [23:29:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 17.05, 11.37, 8.47 [23:30:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.26, 7.77, 7.70 [23:30:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.20, 7.40, 6.55 [23:31:06] RhinosF1, uh, `(Cannot access the database: Cannot access the database: Unknown database 'betawiki' (db12) (db12))` [23:31:28] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 1886s [23:31:49] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.68, 6.58, 6.77 [23:31:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:16] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.16, 20.19, 19.48 [23:32:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.78, 7.07, 7.46 [23:32:33] dmehus: how strange, I can access Betaheze Beta well [23:33:17] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.34, 7.06, 6.15 [23:33:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:33:53] Agent, you can or can't? [23:34:12] RhinosF1: did you merge the changes to betawiki? That's why it needed to be on db12/c3 first which Reception123 was going to do tomorrow. [23:34:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.14, 9.50, 8.28 [23:34:25] dmehus: that's your issue. [23:34:28] CosmicAlpha, ah, that makes sense :) [23:35:07] does Reception123 need to move the actual database to db12/c3, or do we just need to update the db cluster in ManageWiki? [23:35:18] dmehus: I worded it funny, I can access it [23:35:32] dmehus: Both, but needs to be moved first. [23:35:54] Agent, hrm, strange, probably cache if you can access it still, per CosmicAlpha's explanation ^ [23:35:57] CosmicAlpha, ack [23:36:10] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.51, 6.33, 6.36 [23:36:12] Or I can change the echo shared tracking cluster to db13 but it will break test3wiki again which I guess is fine. [23:36:18] dmehus: ^ [23:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:36:46] CosmicAlpha, that's probably fine. We should honestly just drop `test3wiki` after we finalize the set up of `betawiki` [23:36:57] We are going to [23:37:00] oh, good [23:37:05] then that should be fine then [23:37:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.73, 7.12, 6.39 [23:37:15] just make sure you make a detailed log entry [23:37:56] dmehus: it's a config change. [23:38:43] CosmicAlpha, I'd still log it either way, though [23:38:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDHl8 [23:38:58] [02miraheze/mw-config] 07Universal-Omega 0377a2cf5 - Set the beta cluster to db13 (for betawiki) [23:39:02] yeah I still can't access `betawiki`. Very weird some can, but others can't. Not sure what the issue is [23:39:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.90, 6.41, 6.22 [23:39:20] Has to do with global echo notifications dmehus [23:39:28] ah [23:39:48] that makes sense then [23:39:59] miraheze/mw-config - Universal-Omega the build passed. [23:41:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:41:53] !log [@mw11] starting deploy of {'config': True} to ovlon [23:41:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.17, 6.50, 6.38 [23:42:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.21, 7.31, 7.99 [23:42:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:10] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 16s [23:42:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.39, 6.69, 7.81 [23:43:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.32, 6.22, 6.30 [23:44:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:49:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:50:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.10, 6.90, 7.34 [23:51:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.90, 3.92, 3.31 [23:52:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.28, 6.63, 7.19 [23:53:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.82, 5.34, 6.58 [23:53:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.81, 3.35 [23:54:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:55:00] !log [@test3] starting deploy of {'config': True} to skip [23:55:01] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.32, 20.31, 23.43 [23:55:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.39, 5.91, 6.77 [23:57:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.79, 5.07, 5.97 [23:57:51] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Echo/db_patches/echo_unread_wikis.sql --wiki=betawiki [23:57:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:59] dmehus: should work now [23:59:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.92, 6.01, 6.22 [23:59:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:59:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.10, 22.27, 23.38 [23:59:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.37, 3.30