[00:00:55] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.05, 5.45, 2.83 [00:03:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 3.29, 3.26 [00:03:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/607df0bf7c7d...978bef3dbaaa [00:03:02] [02miraheze/mw-config] 07Universal-Omega 03978bef3 - raise cpuLimit for luasandbox [00:03:12] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:03:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.86, 3.75, 3.70 [00:03:38] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [00:03:40] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 780 failures. Last run 2 minutes ago with 780 failures. Failed resources (up to 3 shown): File[/etc/vim/vimrc.local],File[/usr/local/bin/cgroup-mediawiki-clean],File[/etc/ImageMagick-6/policy.xml],File[/usr/local/bin/prometheus-phpfpm-exporter] [00:04:00] miraheze/mw-config - Universal-Omega the build passed. [00:04:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.82, 3.68 [00:05:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/978bef3dbaaa...517cccd81caf [00:05:41] [02miraheze/mw-config] 07Universal-Omega 03517cccd - raise cpuLimit for luasandbox [00:05:46] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:05:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:58] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [00:06:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.61, 3.62 [00:06:45] miraheze/mw-config - Universal-Omega the build passed. [00:07:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/517cccd81caf...f93fd89de518 [00:07:22] [02miraheze/mw-config] 07Universal-Omega 03f93fd89 - lower cpuLimit for luasandbox [00:07:27] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:07:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:41] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [00:07:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:45] miraheze/mw-config - Universal-Omega the build passed. [00:08:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.57, 3.38 [00:09:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.73, 3.72 [00:10:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.66, 3.62 [00:10:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.36, 10.83, 9.46 [00:10:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.39, 1.26, 1.79 [00:11:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f93fd89de518...d5a6c0523a4b [00:11:12] [02miraheze/mw-config] 07Universal-Omega 03d5a6c05 - Raise luasandbox cpuLimit [00:11:17] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:33] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 16s [00:11:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.54, 3.09, 3.40 [00:12:29] miraheze/mw-config - Universal-Omega the build passed. [00:12:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.05, 10.00, 9.31 [00:12:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.71, 3.45 [00:12:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.76, 1.12, 1.67 [00:14:45] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:14:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.54, 3.43 [00:14:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:57] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [00:15:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.95, 3.83 [00:15:31] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:15:41] [02mw-config] 07Universal-Omega commented on pull request 03#4578: Update Wikibase.php with propertyOrderUrl - 13https://github.com/miraheze/mw-config/pull/4578#issuecomment-1098591469 [00:15:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:42] [02mw-config] 07Universal-Omega closed pull request 03#4578: Update Wikibase.php with propertyOrderUrl - 13https://github.com/miraheze/mw-config/pull/4578 [00:15:45] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [00:15:46] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:15:56] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [00:15:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:57] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:16:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:04] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:16:05] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:16:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:17] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [00:16:18] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:16:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:31] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [00:16:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:32] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:16:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:40] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:16:41] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:16:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 10.52, 9.67 [00:16:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:46] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [00:16:47] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:16:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:54] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:16:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.30, 3.09, 3.27 [00:16:55] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:16:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:00] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [00:17:01] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:17:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:07] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [00:17:08] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:17:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:15] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:17:16] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:17:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:21] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [00:17:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:22] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:17:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.40, 3.64 [00:17:31] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:17:32] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:17:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:43] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [00:17:44] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:17:50] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:17:51] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:17:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:59] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:17:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:00] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:18:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:18:08] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:18:09] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:18:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:17] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:18:18] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:18:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:28] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [00:18:29] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:18:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:38] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:18:39] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:18:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.59, 9.90, 9.53 [00:18:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:49] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:18:50] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:18:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:57] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:18:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:58] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:19:08] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [00:19:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:44] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:21:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:06] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [00:22:07] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:22:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:22] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 15s [00:22:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.05, 11.54, 10.31 [00:22:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.66, 11.10, 10.13 [00:23:04] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:23:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:23:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:14] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [00:23:16] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:23:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:22] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [00:23:23] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:23:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:29] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [00:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.49, 11.12, 10.30 [00:27:00] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:27:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:12] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [00:27:13] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:27:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:24] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [00:27:25] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:27:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 2.98, 3.38 [00:27:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:45] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [00:27:46] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:27:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:57] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [00:27:58] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:28:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:09] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [00:28:10] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:28:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:21] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [00:28:22] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:28:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:28] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:28:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:29] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:28:39] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:28:40] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:28:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:50] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [00:28:52] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:28:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:00] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:29:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:01] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:29:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:09] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:29:10] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:29:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:17] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:29:18] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:29:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:25] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:29:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:26] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:29:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:32] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [00:29:33] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:29:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:40] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:29:41] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:29:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:48] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:29:49] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:29:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:56] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:29:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:57] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:30:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:03] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:30:04] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:12] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:30:13] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:30:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:18] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [00:30:19] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:30:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.56, 10.06, 10.09 [00:30:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:27] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:30:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:28] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:30:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:35] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:30:36] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:30:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:46] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [00:30:46] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:30:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:53] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:30:54] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:30:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:01] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:31:02] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:31:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:12] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [00:31:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:40] what the... [00:31:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:31:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:04] I was not even here for a few minutes why was I deploying so much.. [00:32:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:20] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [00:32:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.92, 10.20, 10.18 [00:32:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:41:33] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [00:41:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.20, 2.90 [00:42:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.26, 2.95 [00:42:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.21, 4.20, 3.54 [00:43:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.35, 3.32, 2.99 [00:43:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.25, 2.99 [00:44:11] !log sudo apt upgrade, then sudo reboot on test101 (upgrades kernel to 5.10.0-13-amd64) [00:44:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:39] PROBLEM - test101 Check Gluster Clients on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [00:46:39] RECOVERY - test101 Check Gluster Clients on test101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:46:47] !log mount /mnt/mediawiki-static on test101 [00:46:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.53, 10.97, 10.15 [00:50:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.88, 3.74 [00:51:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.68, 11.28, 10.37 [00:51:33] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.17, 3.26, 3.10 [00:53:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:55:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.62, 9.87, 10.00 [00:56:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.01, 3.40 [00:59:05] !log sudo apt upgrade, then sudo reboot on mwtask111 (upgrades kernel to 5.10.0-13-amd64) [00:59:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.52, 3.27 [00:59:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:27] !log root@mwtask111:~# mount /mnt/mediawiki-static [01:00:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.23, 10.63, 10.28 [01:03:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 3.70, 3.39 [01:04:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 3.16, 3.23 [01:04:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.06, 9.88, 10.05 [01:10:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.86, 1.97, 1.43 [01:10:58] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [01:10:58] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:12:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.78, 1.56, 1.35 [01:23:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/8501f0598d92 [01:23:06] [02miraheze/mw-config] 07Universal-Omega 038501f05 - remove egLoopsCountLimit overrides [01:23:08] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [01:23:09] [02mw-config] 07Universal-Omega opened pull request 03#4581: remove egLoopsCountLimit overrides - 13https://github.com/miraheze/mw-config/pull/4581 [01:24:16] miraheze/mw-config - Universal-Omega the build passed. [01:25:24] [02dns] 07MacFan4000 opened pull request 03#267: add atnarsia.com zone - 13https://github.com/miraheze/dns/pull/267 [01:27:43] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.13, 2.04, 1.42 [01:29:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.55, 1.50, 1.29 [01:36:24] [02mw-config] 07Universal-Omega closed pull request 03#4581: remove egLoopsCountLimit overrides - 13https://github.com/miraheze/mw-config/pull/4581 [01:36:26] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [01:36:27] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:36:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d5a6c0523a4b...78fceff2404c [01:36:30] [02miraheze/mw-config] 07Universal-Omega 0378fceff - remove egLoopsCountLimit overrides (#4581) [01:36:35] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [01:36:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:50] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [01:36:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:56] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [01:37:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:04] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 8s [01:37:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:30] miraheze/mw-config - Universal-Omega the build passed. [01:37:40] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [01:37:41] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 1s [01:37:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:22] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:41:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/78fceff2404c...08b53f762cd2 [01:41:43] [02miraheze/mw-config] 07Universal-Omega 0308b53f7 - re-enable DataTransfer [01:42:48] miraheze/mw-config - Universal-Omega the build passed. [01:43:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:43:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/08b53f762cd2...6122cf4cfd2d [01:43:51] [02miraheze/mw-config] 07Universal-Omega 036122cf4 - Disable RegexFunctions [01:44:00] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [01:44:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:12] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 11s [01:44:18] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [01:44:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:27] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [01:44:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:28] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [01:44:29] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 2s [01:44:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:58] miraheze/mw-config - Universal-Omega the build passed. [01:45:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.57, 3.18 [01:48:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:49:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.95, 3.13, 3.09 [01:57:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.23, 3.10 [01:59:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.34, 2.90 [02:00:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.31, 2.93 [02:01:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.31, 9.21, 8.19 [02:01:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.01, 3.11, 3.09 [02:03:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:03:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.83, 9.09, 8.27 [02:04:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 2.96, 2.88 [02:09:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 4.05, 3.47 [02:11:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.41, 3.45, 3.33 [02:13:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:13:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.91, 3.04, 3.20 [02:17:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.29, 3.01 [02:17:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.64, 3.44 [02:17:44] PROBLEM - test101 MediaWiki Rendering on test101 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.107 second response time [02:18:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:19:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.80, 3.24 [02:19:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 4.04, 3.61 [02:19:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.65, 3.07 [02:20:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.74, 1.41, 0.99 [02:21:45] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 882 bytes in 0.810 second response time [02:22:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.20, 0.97 [02:23:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:23:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.85, 3.28 [02:27:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.52, 4.24, 3.55 [02:29:20] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [02:29:21] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [02:29:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:29:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:29:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.90, 3.50 [02:34:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.35, 3.54, 3.70 [02:37:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.91, 4.05, 3.68 [02:38:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:38:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.95, 3.83 [02:43:06]