[00:03:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.22, 19.22, 17.18 [00:03:48] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 28 failures. Last run 3 minutes ago with 28 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:04:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.23, 4.41, 3.69 [00:05:02] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 5.52, 3.66, 2.33 [00:05:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.03, 17.84, 16.91 [00:05:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.51, 7.40, 5.73 [00:06:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.95, 3.61 [00:06:59] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.48, 3.57, 2.45 [00:07:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMSrF [00:07:23] [02miraheze/mw-config] 07Universal-Omega 03f4bffb0 - fix [00:07:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.26, 6.22, 5.50 [00:08:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 4.43, 3.83 [00:08:29] miraheze/mw-config - Universal-Omega the build passed. [00:08:55] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.80, 3.28, 2.48 [00:09:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.76, 22.12, 18.91 [00:11:12] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [00:11:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.83, 6.23, 5.58 [00:11:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:29] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 17s [00:11:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:20] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/Cargo/maintenance/cargoRecreateData.php --wiki=dmlwikiwiki (END - exit=0) [00:12:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.29, 5.92, 5.55 [00:13:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.59, 27.56, 21.67 [00:14:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.92, 7.28, 4.60 [00:22:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:22:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.52, 6.79, 5.30 [00:24:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.70, 3.97 [00:24:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.89, 6.76, 5.47 [00:26:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.41, 6.26, 5.47 [00:27:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.05, 6.85, 6.00 [00:30:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.98, 5.78, 5.71 [00:31:48] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.42, 4.91, 5.66 [00:36:45] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.86, 3.83, 5.05 [00:37:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.33, 18.07, 23.22 [00:44:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 2.97, 3.28 [00:47:13] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.32, 16.71, 20.29 [00:47:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.62, 4.28, 3.72 [00:49:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.40, 6.18, 5.25 [00:49:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.89, 3.66 [00:50:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.99, 6.19, 5.37 [00:51:40] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.53, 5.12, 4.97 [00:51:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.50, 4.54, 3.92 [00:52:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.06, 6.88, 5.70 [00:54:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.69, 6.81, 5.83 [00:55:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.27, 6.35, 5.22 [00:57:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.48, 5.57, 5.06 [00:58:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.82, 6.21, 5.86 [01:06:25] PROBLEM - mw8 JobRunner Service on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [01:10:35] known ^, me [01:10:41] !log install gdb on mw8 [01:10:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:11:28] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 29 minutes ago with 0 failures [01:17:14] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.26, 6.85, 5.91 [01:19:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.25, 6.15, 5.77 [01:21:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 6.23, 5.30 [01:22:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.69, 7.23, 5.61 [01:23:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.26, 5.59, 5.19 [01:26:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.70, 6.34, 5.65 [01:28:25] RECOVERY - mw8 JobRunner Service on mw8 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [01:37:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.37, 3.86 [01:39:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.33, 4.40, 4.17 [01:40:17] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.55, 7.76, 6.31 [01:41:17] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 15.72, 21.39, 19.06 [01:42:10] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.64, 6.52, 6.02 [01:43:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 12.60, 18.36, 18.22 [01:51:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.61, 3.93 [01:53:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.97, 4.01, 4.05 [01:59:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.77, 3.94 [02:13:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.97, 2.71, 3.29 [02:20:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.42, 3.48 [02:32:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.10, 3.92, 3.62 [02:36:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.80, 3.68 [02:44:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.07, 3.36 [02:52:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.68, 3.49, 3.34 [02:54:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.95, 3.40, 3.33 [02:58:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.49, 3.42, 3.40 [03:02:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.34, 3.38 [05:33:10] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:06:56] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.19, 5.67, 3.76 [06:08:56] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.10, 5.74, 4.02 [06:13:51] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 51798 MB (11% inode=98%); [06:21:52] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48070 MB (10% inode=98%); [06:41:47] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.76, 1.86, 1.23 [06:43:47] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.27, 1.30, 1.10 [07:43:51] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 51608 MB (11% inode=98%); [08:11:51] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48402 MB (10% inode=98%); [08:13:51] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49927 MB (11% inode=98%); [08:23:51] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48361 MB (10% inode=98%); [08:29:21] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:57:17] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:34:10] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.31, 2.20, 1.44 [10:36:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.89, 1.72, 1.35 [10:37:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.58, 1.35, 1.26 [10:38:38] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.55, 2.02, 1.21 [10:40:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.70, 1.54, 1.13 [11:17:53] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.96, 1.49, 0.83 [11:19:53] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.01, 1.25, 0.82 [12:45:06] Please take https://developereconomics.net/?member_id=ZStKkRQC&lang=en&locale=en - it's a very interesting survey with some great data. (I'm part of the referral program). [12:45:07] [url] Developer Nation Survey | Developer tools, apps, design, games | developereconomics.net [13:23:16] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/d [13:23:16] ackages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: lcn.zfc.id.lv., lcn.zfc.id.lv. [13:37:13] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:58:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.79, 6.74, 4.96 [13:59:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.55, 7.12, 5.21 [13:59:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.42, 3.01 [14:00:18] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.94, 7.96, 5.96 [14:00:45] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 6.10, 4.94 [14:01:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.69, 6.44, 5.21 [14:01:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.22, 2.98 [14:06:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.56, 6.69, 6.08 [14:10:22] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.77, 6.24, 5.07 [14:10:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.57, 3.26 [14:12:16] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.63, 5.65, 5.00 [14:16:42] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.05, 3.14 [14:41:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.84, 4.12, 3.58 [14:49:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.64, 4.00, 3.84 [14:51:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.81, 4.65, 4.09 [15:07:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.78, 3.99 [15:19:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 2.97, 3.37 [15:30:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 21.04, 10.18, 6.19 [15:32:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.50, 7.02, 5.14 [15:32:27] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.33, 19.14, 15.60 [15:33:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:33:16] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 5.81, 4.80, 2.06 [15:33:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.17, 10.43, 7.02 [15:34:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.96, 6.28, 5.12 [15:34:26] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.55, 17.99, 15.61 [15:35:15] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.83, 3.22, 1.81 [15:35:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.45, 3.36 [15:37:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.55, 3.38 [15:37:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.70, 7.27, 6.49 [15:38:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:38:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.20, 7.28, 5.47 [15:38:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.33, 7.30, 6.80 [15:39:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.56, 3.40 [15:39:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.98, 6.23, 6.20 [15:40:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.81, 7.64, 6.15 [15:40:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.26, 6.09, 5.25 [15:40:24] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.16, 6.61, 6.60 [15:42:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.74, 6.67, 5.97 [15:45:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.17, 3.29 [15:53:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.85, 3.52 [15:53:20] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 10.28, 4.94, 2.44 [15:54:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.38, 7.34, 6.40 [15:55:09] [02mw-config] 07magogre closed pull request 03#4244: Add a new canned response - 13https://git.io/JMXBG [15:55:13] [02mw-config] 07magogre reopened pull request 03#4244: Add a new canned response - 13https://git.io/JMXBG [15:55:16] PROBLEM - test3 Current Load on test3 is WARNING: WARNING - load average: 1.94, 3.71, 2.30 [15:55:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.45, 3.42 [15:57:15] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.30, 2.52, 2.03 [16:00:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.26, 7.40, 6.87 [16:02:04] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.37, 7.49, 6.94 [16:03:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.89, 3.64 [16:04:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.42, 7.08, 6.86 [16:05:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.69, 3.61 [16:06:00] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.75, 6.33, 6.62 [16:09:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.66, 3.58 [16:11:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.76, 3.63 [16:17:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.64, 3.58 [16:47:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.16, 3.27, 3.85 [16:50:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.18, 7.15, 5.45 [16:51:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.03, 4.18, 4.08 [16:52:01] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.01, 6.06, 5.14 [16:52:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.83, 6.87, 5.56 [16:53:28] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:53:57] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.83, 5.94, 5.20 [16:56:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.78, 8.07, 6.31 [16:58:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.01, 7.60, 5.92 [16:59:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.76, 3.96 [16:59:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.38, 7.37, 6.04 [17:00:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:00:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.60, 7.46, 6.07 [17:01:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.69, 7.29, 6.13 [17:03:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.34, 7.94, 6.50 [17:03:42] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.93, 6.55, 6.05 [17:05:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:05:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.40, 7.39, 6.48 [17:06:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.40, 6.39, 6.09 [17:06:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.04, 7.93, 7.91 [17:07:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.90, 6.62, 6.31 [17:07:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.80, 3.82 [17:09:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.74, 3.81 [17:12:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.54, 7.22, 7.48 [17:13:03] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.33, 8.59, 7.13 [17:13:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.17, 8.48, 6.84 [17:15:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.62, 7.53, 6.69 [17:16:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.69, 7.73, 7.66 [17:16:59] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.72, 7.78, 7.15 [17:19:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.64, 7.50, 6.59 [17:19:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.63, 4.18, 3.96 [17:20:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.70, 8.07, 7.76 [17:21:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.72, 7.57, 6.75 [17:21:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.73, 6.41, 6.47 [17:22:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.62, 6.81, 7.34 [17:22:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.72, 6.58, 6.75 [17:23:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.33, 6.61, 6.49 [17:25:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.78, 3.89 [17:25:19] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.45, 6.82, 6.34 [17:26:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.69, 5.63, 6.73 [17:27:13] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.46, 5.86, 6.03 [17:35:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.51, 3.59 [17:39:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.82, 3.72 [17:41:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.13, 4.49, 3.96 [17:43:00] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 8.05, 6.76 [17:43:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.57, 7.87, 6.57 [17:44:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.48, 7.32, 6.43 [17:44:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.49, 7.93, 6.87 [17:45:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.00, 6.70, 6.29 [17:46:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.77, 6.46, 6.21 [17:50:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.94, 7.88, 6.56 [17:50:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.13, 8.39, 7.10 [17:50:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.39, 6.77, 6.66 [17:51:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.63, 9.32, 7.27 [17:51:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:52:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.70, 7.18, 6.48 [17:52:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.72, 7.05, 6.75 [17:54:28] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.45, 5.99, 6.14 [17:54:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.98, 6.27, 6.50 [17:54:48] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.12, 7.04, 6.83 [17:56:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:56:41] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.71, 6.57, 6.67 [17:57:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.72, 3.46, 3.91 [17:59:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 4.04, 4.07 [18:05:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.79, 3.98 [18:06:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.14, 8.59, 7.16 [18:07:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 4.20, 4.10 [18:08:03] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.12, 8.19, 6.80 [18:08:04] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.88, 8.00, 7.07 [18:10:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.96, 7.52, 6.71 [18:11:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.74, 6.72, 5.95 [18:13:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.71, 6.35, 5.91 [18:15:55] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.91, 6.66, 6.63 [18:16:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.78, 7.71, 7.64 [18:21:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.35, 7.47, 6.93 [18:22:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 17.09, 10.77, 8.75 [18:24:27] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.35, 20.56, 18.77 [18:25:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:26:26] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.48, 20.06, 18.84 [18:29:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 3.44, 6.85, 7.32 [18:30:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:32:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 3.99, 6.00, 7.57 [18:32:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.53, 6.52, 7.81 [18:33:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.57, 5.74, 6.77 [18:38:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.63, 4.84, 6.51 [18:38:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.87, 4.85, 6.59 [18:39:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.65, 3.95 [18:45:28] [02puppet] 07Universal-Omega opened pull request 03#2142: MediaWiki: use deploy repos for mathoid and 3d2png - 13https://git.io/JMQhM [18:49:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.92, 3.86 [18:50:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.45, 7.79, 6.19 [18:51:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.43, 7.59, 5.89 [18:52:00] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.13, 6.45, 6.14 [18:52:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.73, 7.17, 6.15 [18:52:53] !log create mediawiki-mathoid-deploy and mediawiki-3d2png-deploy repositories on GitHub [18:53:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.46, 6.91, 5.86 [18:53:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.59, 7.58, 6.38 [18:54:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.10, 7.50, 6.68 [18:54:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.04, 7.66, 6.43 [18:55:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.50, 6.07, 5.67 [18:56:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.39, 7.94, 6.97 [18:56:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.36, 7.38, 6.50 [18:57:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.53, 7.38, 6.69 [18:57:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JM7vl [18:57:52] [02miraheze/puppet] 07paladox 031eeaf57 - Disable unprivileged user namespaces on Bullseye and Buster hosts with 5.10 [18:57:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.11, 7.29, 6.86 [18:58:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.57, 6.60, 6.33 [18:59:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.74, 3.98 [18:59:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.98, 6.40, 6.43 [18:59:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.72, 6.36, 6.57 [19:00:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.58, 6.07, 6.42 [19:03:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JM7fX [19:04:00] [02miraheze/puppet] 07paladox 038f41b48 - base::sysctl: set vm.swappiness to 0 for all hosts [19:04:01] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:04:03] [02puppet] 07paladox opened pull request 03#2143: base::sysctl: set vm.swappiness to 0 for all hosts - 13https://git.io/JM7f1 [19:04:41] [02puppet] 07paladox closed pull request 03#2143: base::sysctl: set vm.swappiness to 0 for all hosts - 13https://git.io/JM7f1 [19:04:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:04:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JM7f7 [19:04:46] [02miraheze/puppet] 07paladox 032c2cb9d - base::sysctl: set vm.swappiness to 0 for all hosts (#2143) [19:04:47] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:07:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 2.74, 3.39 [19:15:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JM7Tm [19:15:52] [02miraheze/puppet] 07paladox 03a47c4c9 - base::sysctl: Move redis specific stuff to redis class [19:15:53] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:15:55] [02puppet] 07paladox opened pull request 03#2144: base::sysctl: Move redis specific stuff to redis class - 13https://git.io/JM7TO [19:16:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JM7T4 [19:16:21] [02miraheze/puppet] 07paladox 03452f6c1 - Update sysctl.pp [19:16:23] [02puppet] 07paladox synchronize pull request 03#2144: base::sysctl: Move redis specific stuff to redis class - 13https://git.io/JM7TO [19:19:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JM7Tp [19:19:41] [02miraheze/puppet] 07paladox 03c897e44 - sysfs::conffile: Convert to data types [19:21:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.53, 4.15, 3.63 [19:21:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JM7ku [19:21:50] [02miraheze/puppet] 07paladox 03c02129e - base::sysctl: Move redis specific stuff to redis class (#2144) [19:21:52] [02puppet] 07paladox closed pull request 03#2144: base::sysctl: Move redis specific stuff to redis class - 13https://git.io/JM7TO [19:21:53] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:21:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:23:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.29, 6.30, 5.64 [19:25:09] !log reboot mw8 [19:25:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:24] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.25, 6.60, 5.59 [19:25:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.12, 5.70, 5.52 [19:27:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.77, 3.67 [19:27:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.58, 6.36, 5.64 [19:29:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.85, 3.70 [19:29:23] !log reboot bacula2 [19:29:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:33] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:31:31] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:35:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.53, 3.63 [19:37:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.65, 3.64 [19:41:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.92, 3.12, 3.45 [19:43:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.03, 3.09, 3.40 [19:51:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.68, 3.55 [19:52:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.22, 7.31, 4.64 [19:52:48] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 16.64, 10.75, 7.34 [19:53:04] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:54:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.36, 7.43, 5.02 [19:54:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.22, 7.09, 5.95 [19:56:01] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.92, 6.16, 4.85 [19:56:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.03, 5.85, 5.63 [19:56:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.20, 7.82, 6.97 [19:58:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.34, 6.59, 6.61 [20:01:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.79, 3.89 [20:03:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.97, 4.07, 3.99 [20:05:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.72, 3.86 [20:15:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.41, 3.99, 3.78 [20:17:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.73, 3.71 [20:17:22] [02miraheze/mediawiki-mathoid-deploy] 07Universal-Omega pushed 031 commit to 03master [+3000/-0/±0] 13https://git.io/JM7G4 [20:17:24] [02miraheze/mediawiki-mathoid-deploy] 07Universal-Omega 039d74d0a - Initial mathoid import [20:17:25] [02mediawiki-mathoid-deploy] 07Universal-Omega created branch 03master - 13https://git.io/JM7GB [20:19:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.06, 6.80, 5.83 [20:21:42] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.78, 6.16, 5.72 [20:21:51] paladox: ^ FYI - and I tried to do 3d2png also, but for some reason packages keep failing to install on my PC, so I can't push it. [20:22:15] you can build it on test3 if you want [20:23:31] [02mw-config] 07RhinosF1 commented on pull request 03#4249: LS: blacklist .*betawiki in prep for future usage - 13https://git.io/JM7ZG [20:27:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 2.97, 3.40 [20:29:42] paladox: same issue on test3 https://www.irccloud.com/pastebin/hmCxPaA7/ [20:31:40] try now CosmicAlpha [20:31:47] ^ something trying to use python 2 syntax but running on python 3 [20:32:01] yehhttps://gerrit.wikimedia.org/r/c/3d2png/+/720699/2 [20:32:03] https://gerrit.wikimedia.org/r/c/3d2png/+/720699/2 [20:33:01] majavah: you wuldn't happen to know how to debug php-fpm OOMs? [20:36:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.40, 4.52, 3.80 [20:36:30] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.20, 6.56, 5.97 [20:36:40] thanks paladox! [20:36:56] paladox: you installed Python is python3 didn't you [20:37:04] yes [20:37:28] paladox: Error: EACCES: permission denied, open '/srv/3d2png/package-lock.json' [20:37:34] paladox: that's why [20:37:46] It should be failing with simply Python doesn't exist [20:38:08] RhinosF1: python3 wasn't supported in node-canvas, it required a newer version of that which supports python3 [20:38:11] * RhinosF1 did say Python != python3 and it's bad practice because the syntax is so different [20:38:16] python2 is not supported under debian 11 [20:38:23] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.15, 6.08, 5.86 [20:38:35] paladox: yes but my point is we shouldn't be trying python3 code on Python anyway [20:38:58] CosmicAlpha: try now [20:39:06] It shouldn't be telling you python doesn't exist rather than attempting to execute it [20:39:11] paladox: my general advice for OOMs is to attach a debugger at a point with high memory usage, but I'm not familiar of c and tooling around it [20:39:30] ok [20:42:11] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.92, 7.37, 6.42 [20:43:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.01, 6.69, 6.06 [20:46:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.81, 6.26, 6.19 [20:47:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.80, 7.54, 6.71 [20:47:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.08, 5.88, 5.95 [20:48:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.55, 3.83, 3.93 [20:49:06] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.15, 5.99, 6.24 [20:50:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.64, 7.24, 5.72 [20:50:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.19, 6.72, 5.65 [20:52:36] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.76, 6.35, 5.59 [20:52:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.77, 6.22, 5.59 [20:54:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.62, 3.78 [20:56:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.59, 3.76 [21:00:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.84, 3.82 [21:02:02] [02miraheze/mediawiki-3d2png-deploy] 07Universal-Omega pushed 031 commit to 03master [+3000/-0/±0] 13https://git.io/JM7WQ [21:02:04] [02miraheze/mediawiki-3d2png-deploy] 07Universal-Omega 03cee05f8 - Initial 3d2png import [21:02:05] [02mediawiki-3d2png-deploy] 07Universal-Omega created branch 03master - 13https://git.io/JM7W5 [21:02:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.75, 3.79 [21:02:36] paladox: ^ yep, that worked now! Thanks! Also, should a deploy repo be done for femiwiki npm also? Though that being an extension submodule in the mediawiki repo makes it a bit more difficult I think. Probably still possible though can't say for sure. [21:03:19] Yes it should. [21:05:48] [02puppet] 07Universal-Omega synchronize pull request 03#2142: MediaWiki: use deploy repos for mathoid and 3d2png - 13https://git.io/JMQhM [21:06:06] paladox: all right, will try then. [21:10:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.69, 2.70, 3.33 [21:37:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.28, 3.52, 3.26 [21:39:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.56, 3.30 [21:40:57] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.72, 1.21 [21:41:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.74, 3.39 [21:42:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.39, 1.14 [21:43:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.47, 3.33 [21:48:44] [02puppet] 07Universal-Omega synchronize pull request 03#2142: MediaWiki: use deploy repos for mathoid and 3d2png - 13https://git.io/JMQhM [21:49:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 3.90, 3.53 [21:50:35] [02puppet] 07Universal-Omega edited pull request 03#2142: MediaWiki: use deploy repos for mathoid and 3d2png - 13https://git.io/JMQhM [21:50:57] paladox, I couldn't really find a good way to do femiwiki here, so I just did that, adding `--no-optional --only=production` so only the two ones needed for it are installed. Neither one of them requires any more either. Hopefully that would be good enough, otherwise I don't really know how. [21:51:17] ok [21:51:30] [02puppet] 07Universal-Omega synchronize pull request 03#2142: MediaWiki: use deploy repos for mathoid and 3d2png - 13https://git.io/JMQhM [21:51:32] [02puppet] 07Universal-Omega synchronize pull request 03#2142: MediaWiki: use deploy repos for mathoid and 3d2png - 13https://git.io/JMQhM [21:51:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.50, 3.43 [21:53:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 3.82, 3.55 [21:55:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.97, 3.16, 3.34 [21:59:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.17, 4.06, 3.65 [22:03:28]