[00:03:14] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 25 failures. Last run 3 minutes ago with 25 failures. Failed resources (up to 3 shown): File[/etc/ferm/functions.conf],File[/etc/default/ferm],File[/usr/lib/nagios/plugins/check_conntrack],File[/usr/lib/nagios/plugins/check_ferm] [00:06:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.40, 3.10 [00:12:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.34, 3.23 [00:12:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.95, 23.78, 19.31 [00:17:22] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.13, 7.97, 5.34 [00:29:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.84, 3.49 [00:31:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.50, 3.40 [00:33:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.28, 3.33 [00:33:13] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:41:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.79, 3.53 [00:43:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.43, 3.43 [00:45:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.30, 3.38 [00:55:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.81, 4.22, 3.67 [00:59:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.75, 3.61 [01:02:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 13.66, 17.07, 23.87 [01:03:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.95, 3.70 [01:05:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.73, 3.65 [01:07:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 4.09, 3.79 [01:08:05] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.95, 5.13, 6.00 [01:11:59] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.43, 5.69, 5.98 [01:13:56] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.24, 5.44, 5.86 [01:14:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.87, 21.07, 22.01 [01:15:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 5.74, 6.21, 6.13 [01:16:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.49, 19.97, 21.50 [01:17:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.87, 3.93 [01:17:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.43, 5.45, 5.85 [01:19:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 4.21, 4.05 [01:23:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.84, 3.96 [01:24:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.42, 17.86, 20.19 [01:25:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.53, 4.24, 4.07 [01:25:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.55, 3.53, 4.79 [01:26:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.78, 6.90, 5.80 [01:28:10] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.44, 6.27, 5.69 [01:35:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.73, 3.93 [01:39:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.88, 3.96 [01:41:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.64, 3.86 [01:51:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.66, 3.71 [01:52:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.82, 3.77 [01:54:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 4.06, 3.87 [01:56:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.92, 3.83 [02:04:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.85, 2.92, 3.38 [02:21:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.11, 4.07, 3.55 [02:22:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.80, 18.41, 15.74 [02:23:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.72, 3.47 [02:24:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.13, 16.91, 15.53 [02:25:06] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JPhCt [02:25:07] [02miraheze/mediawiki] 07Universal-Omega 0323fe7c3 - Update CreateWiki [02:25:22] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all [02:25:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:27:10] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JPhC0 [02:27:11] [02miraheze/mediawiki] 07Universal-Omega 03c1243b2 - Update CreateWiki [02:27:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.12, 3.67 [02:29:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.39, 3.62, 3.55 [02:29:38] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip [02:29:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 4.17, 3.77 [02:37:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.95, 3.55, 3.70 [02:39:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.84, 3.79 [02:41:14] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 696s [02:41:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.75, 3.81, 3.78 [02:41:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.58, 3.58 [02:51:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.56, 20.92, 17.37 [02:53:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.51, 3.56 [02:53:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.02, 20.42, 17.69 [02:55:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.25, 18.71, 17.38 [02:56:01] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1839s [02:56:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.91, 3.70 [03:01:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.78, 3.71 [03:03:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.80, 3.71 [03:05:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.51, 3.61 [03:10:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.63, 3.68, 3.59 [03:12:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.45, 3.52 [03:14:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.60, 2.92, 3.33 [03:31:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.20, 3.10 [03:33:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.39, 3.15, 3.09 [04:21:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 3.68, 3.12 [04:22:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.49, 3.12 [04:28:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.25, 3.16 [04:32:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.63, 3.73, 3.36 [04:34:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.17, 3.20 [06:04:27] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.56, 7.30, 5.53 [06:16:41] !log reception@mwtask1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/DiscussionTools/sql/mysql/discussiontools_subscription.sql --wiki zhwpwikiwiki [06:16:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:48] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPhFW [06:20:50] [02miraheze/puppet] 07Universal-Omega 038c87181 - T8230: don't set noindex on Special:WikiForum (#2109) [06:20:51] [02puppet] 07Reception123 closed pull request 03#2109: T8230: don't set noindex on Special:WikiForum - 13https://git.io/JPpHU [06:23:42] [02puppet] 07Reception123 closed pull request 03#2102: mediawiki.conf.erb: remove comment - 13https://git.io/JPoeD [06:23:44] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPhFy [06:23:45] [02miraheze/puppet] 07Universal-Omega 039c7e1d1 - mediawiki.conf.erb: remove comment (#2102) [06:27:44] [02puppet] 07Universal-Omega reviewed pull request 03#2105 commit - 13https://git.io/JPhbC [06:56:27] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.52, 7.31, 7.87 [07:00:27] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.65, 8.07, 8.05 [07:02:27] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.39, 7.86, 7.98 [07:04:27] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.22, 8.14, 8.07 [07:17:47] PROBLEM - db12 APT on db12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:19:31] PROBLEM - db12 Current Load on db12 is CRITICAL: CRITICAL - load average: 11.74, 10.36, 6.89 [07:19:44] RECOVERY - db12 APT on db12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:21:31] PROBLEM - db12 Current Load on db12 is WARNING: WARNING - load average: 2.39, 7.25, 6.17 [07:23:31] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 1.07, 5.22, 5.56 [07:26:27] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.41, 7.05, 7.89 [07:29:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPjfI [07:29:38] [02miraheze/mw-config] 07Universal-Omega 038847dec - Fix sv6 and sv7 values in wgFileExtensions [07:30:35] miraheze/mw-config - Universal-Omega the build passed. [07:38:27] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.38, 7.65, 7.62 [07:41:09] !log [@mw11] starting deploy of {'config': True} to all [07:41:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:19] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 9s [07:41:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:27] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.40, 7.45, 7.56 [07:48:22] PROBLEM - db13 APT on db13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:48:27] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.79, 7.72, 7.57 [07:50:27] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.72, 7.41, 7.48 [07:50:38] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 14.10, 7.48, 5.16 [07:52:27] RECOVERY - db13 APT on db13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:53:42] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:54:35] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 2.52, 7.52, 6.09 [07:54:41] !log [@test3] starting deploy of {'config': True} to skip [07:54:42] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:54:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:54:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:35] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 1.68, 5.51, 5.52 [07:58:42] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:06:27] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.14, 5.92, 6.68 [08:18:16] PROBLEM - sportsanalytics.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sportsanalytics.wiki' expires in 15 day(s) (Mon 22 Nov 2021 08:11:56 GMT +0000). [11:54:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.11, 2.70 [11:56:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.82, 2.54, 2.54 [11:57:32] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JPjD5 [11:57:34] [02miraheze/ssl] 07Reception123 03a275009 - add grc.repository.archiopedia.org cert [11:58:30] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JPjDj [11:58:32] [02miraheze/dns] 07Reception123 03756b3f3 - add grayravens.com zone [12:39:31] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.70, 1.81, 1.08 [12:41:29] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.70, 1.39, 1.00 [12:49:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 18.07, 10.98, 6.45 [12:53:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.47, 7.26, 5.97 [12:55:28] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 2.54, 5.80, 5.60 [13:49:00] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.90, 1.75, 1.23 [13:51:00] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.57, 1.34, 1.14 [14:31:46] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 116.29 ms [14:36:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.45, 3.09 [14:38:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.30, 3.08 [15:00:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXeCe [15:00:12] [02miraheze/puppet] 07paladox 03dd7c429 - base: Mask the puppet service [15:01:45] [02puppet] 07paladox synchronize pull request 03#2105: l10nupdate: move to mwdeploy and avoid I/O starvation - 13https://git.io/JPxxK [15:01:57] [02puppet] 07paladox reviewed pull request 03#2105 commit - 13https://git.io/JXeCR [15:03:08] [02puppet] 07paladox synchronize pull request 03#2105: l10nupdate: move to mwdeploy and avoid I/O starvation - 13https://git.io/JPxxK [15:03:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±2] 13https://git.io/JXeC1 [15:03:29] [02miraheze/puppet] 07RhinosF1 036f0db80 - l10nupdate: move to mwdeploy and avoid I/O starvation (#2105) [15:03:31] [02puppet] 07paladox closed pull request 03#2105: l10nupdate: move to mwdeploy and avoid I/O starvation - 13https://git.io/JPxxK [15:04:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.25, 3.12 [15:16:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.36, 3.31 [15:20:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.62, 4.67, 3.83 [15:22:27] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.47, 9.68, 6.59 [15:24:22] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:24:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.54, 3.91, 3.74 [15:26:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 4.02, 3.78 [15:26:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.53, 6.81, 6.11 [15:26:49] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXeB4 [15:26:50] [02miraheze/puppet] 07JohnFLewis 037f34717 - graylog: Set ES Codec to best_compression [15:28:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.76, 3.72 [15:28:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.53, 5.96, 5.87 [15:29:58] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppet] [15:31:01] paladox: ^ [15:31:09] looking [15:31:55] Error: /Stage[main]/Base::Puppet/Service[puppet]/enable: change from 'false' to 'mask' failed: Could not disable puppet: [15:32:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXeRx [15:32:53] [02miraheze/puppet] 07paladox 034d0a309 - Revert "base: Mask the puppet service" [15:33:57] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:34:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.91, 2.84, 3.34 [15:51:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.56, 6.26, 5.10 [15:53:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.15, 5.03, 4.79 [16:07:12] PROBLEM - cp13 Current Load on cp13 is CRITICAL: CRITICAL - load average: 2.36, 2.60, 1.48 [16:09:08] PROBLEM - cp13 Current Load on cp13 is WARNING: WARNING - load average: 0.50, 1.80, 1.31 [16:11:04] RECOVERY - cp13 Current Load on cp13 is OK: OK - load average: 0.63, 1.44, 1.24 [16:21:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.90, 7.50, 5.33 [16:23:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.47, 6.42, 5.21 [16:32:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.42, 3.10 [16:34:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.31, 3.06, 3.01 [16:49:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.35, 3.04 [16:53:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.29, 3.39, 3.15 [17:05:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.31, 3.06 [17:07:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.95, 3.20, 3.05 [17:08:29] [02puppet] 07Universal-Omega opened pull request 03#2110: Remove absented LocalisationUpdate cron and cleanup spacing - 13https://git.io/JXe9F [17:10:14] [02puppet] 07Universal-Omega synchronize pull request 03#2110: Remove absented LocalisationUpdate cron and cleanup spacing - 13https://git.io/JXe9F [17:10:15] [02puppet] 07Universal-Omega synchronize pull request 03#2110: Remove absented LocalisationUpdate cron and cleanup spacing - 13https://git.io/JXe9F [17:11:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.72, 3.29 [17:13:01] [02puppet] 07Universal-Omega opened pull request 03#2111: deploy-mediawiki: remove commented l10nupdate from l10n argument - 13https://git.io/JXeHS [17:13:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.34, 3.20 [17:14:13] Ty CosmicAlpha [17:14:21] And ty paladox for merging [17:14:30] Hopefully it'll work at 11pm [17:15:07] RhinosF1: No problem. [17:21:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.56, 3.30 [17:23:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.25, 3.22 [17:44:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.14, 6.69, 5.90 [17:45:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.66, 3.28 [17:46:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.03, 6.28, 5.83 [17:48:39] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.50, 7.79, 6.37 [17:48:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.01, 7.46, 5.96 [17:50:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.48, 8.52, 6.84 [17:50:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.93, 7.90, 6.33 [17:52:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.47, 7.59, 6.71 [17:52:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.76, 3.44 [17:54:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.99, 7.45, 6.49 [17:54:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.22, 6.50, 6.10 [17:54:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.48, 3.38 [17:56:35] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.57, 6.43, 6.23 [17:56:36] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.76, 6.24, 6.38 [17:58:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.20, 3.29 [18:08:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.68, 3.43 [18:12:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.45, 3.36, 3.38 [18:21:26] PROBLEM - cp15 Current Load on cp15 is CRITICAL: CRITICAL - load average: 1.53, 2.02, 1.28 [18:23:25] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.53, 1.51, 1.18 [18:27:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.72, 3.50 [18:33:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.65, 3.48 [18:35:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.08, 3.29 [18:48:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.11, 3.90, 3.46 [18:50:55] SRE, 350+ jobs on `metawiki`. Probably should run jobs and investigate the cause afterward [18:52:45] dmehus: seems to have self-cleared. [18:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.86, 3.62 [18:54:49] CosmicAlpha, yeah, still very slow though [18:59:24] dmehus: yeah has been for awhile though. But still clears. [18:59:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.50, 7.10, 5.96 [19:01:19] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.35, 6.14, 5.74 [19:01:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.12, 3.38 [19:01:52] CosmicAlpha, yeah [19:10:38] It might be interesting to try and debug what generates so many jobs, it seems only dmehus reports it as well [19:11:04] yeah [19:12:27] CosmicAlpha, mind running `fixStuckGlobalRename.php` on Techboyg5? Their first attached wiki is a deleted wiki, which is a known bug. It's not `techwikiwiki`, either, which was just deleted, as that's the bottom wiki on the list [19:12:55] https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1&viewPanel=6 only shows a peak for the global user page update though [19:15:45] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 113.62 ms [19:16:45] JohnLewis, ah, that makes sense. The "usual suspect," in a sense. It could've been the global rename that triggered it, but on the other hand, back in like May/June when you made those changes to job handling, global renames all were handled super speedily. Since August or so, they've reverted to being even slower than before, so not sure what's causing that [19:19:43] 2k jobs in a minute is very fast [19:20:24] That’s about 35 jobs a second, considering we only have count(mw*) + 1 job runners [19:41:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.38, 3.14 [19:45:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.92, 4.25, 3.52 [19:47:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.95, 3.50 [19:51:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.36, 3.35 [19:54:55] [02CreateWiki] 07Universal-Omega closed pull request 03#255: Use settings log if there is more than one change - 13https://git.io/JPPVM [19:54:58] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [19:55:00] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:59:33] dmehus: currently mobile. I can in about 15 minutes if it's still necessary. [20:02:04] dmehus: unless jobs are climbing [20:02:08] There's no need to report [20:02:16] If it's a clear spike, give time to clear [20:02:21] You know how to use grafana [20:06:17] CosmicAlpha, SGTM. RhinosF1, ack [20:06:49] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 127.11 ms [20:10:43] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 125.39 ms [20:15:37] dmehus: what is their old and new usernames (not sure what one you gave me) and what wiki is it stuck on? [20:19:04] CosmicAlpha, not sure the deleted wiki [20:19:11] let me look up their old/new username [20:19:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.95, 7.06, 5.36 [20:19:20] there's a command to check what wiki it's stuck on [20:19:26] I don't know what it is [20:20:06] Hmm. Let me look. [20:20:34] CosmicAlpha, [20:20:35] Old username: Techboyg5 [20:20:35] New/requested username: Deleted Account 4116142 [20:21:13] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.07, 7.64, 5.78 [20:21:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.81, 6.84, 5.17 [20:22:36] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.95, 6.37, 5.12 [20:22:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.72, 6.85, 5.62 [20:22:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.57, 3.31 [20:23:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 7.56, 5.98 [20:23:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.14, 6.64, 5.31 [20:24:36] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 6.32, 5.24 [20:26:36] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.37, 6.30, 5.37 [20:26:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.14, 6.99, 5.93 [20:26:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.31, 3.28 [20:28:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.87, 6.75, 5.96 [20:29:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.75, 6.31, 5.96 [20:29:20] [02ManageWiki] 07Universal-Omega closed pull request 03#314: T8234: allow (un)locking and (un)deleting actions in combination with others - 13https://git.io/JP6BY [20:29:22] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [20:29:23] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [20:30:36] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.87, 5.66, 5.66 [20:34:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.48, 3.33 [20:36:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.80, 3.46 [20:40:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.44, 3.39 [20:42:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.88, 3.55 [20:44:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.49, 3.45 [20:48:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 3.85, 3.57 [20:49:05] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commutativealgebrawiki --logwiki=metawiki 'Techboyg5' 'Deleted Account 4116142' (END - exit=0) [20:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.33, 7.60, 6.16 [20:50:15] dmehus: ^ [20:50:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.81, 3.59 [20:51:11] JohnLewis: can I perform a CU to check https://phabricator.miraheze.org/T8258 [20:51:12] [url] Login | phabricator.miraheze.org [20:51:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 7.81, 6.43 [20:52:05] Rhinos [20:52:18] didn’t type my message :P [20:52:24] heh [20:52:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 4.06, 3.70 [20:52:35] I’ll reply on the task, I don’t think a CU is needed [20:53:24] JohnLewis: I mainly want to check something really obvious hasn't been missed [20:53:33] But if you know cause that's cool [20:53:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.74, 6.61, 6.15 [20:53:57] CU is weird because it's hard to get instinct as we are less familiar with it for a reason [20:54:21] JohnLewis: that's a fa [20:54:25] Fair point [20:54:35] Either that or a failed login or something [20:55:38] Yeah, I don’t think it’s an oddity, rather weird intentional design [20:56:07] You can probably confirm either by checking AF logs or account creation logs in graylog I would assume [20:56:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.65, 3.52, 3.58 [20:57:14] JohnLewis: it's a fair enough shout [20:57:28] dmehus seems happy and to have found an abuse log [20:57:41] I wish I knew the CU interface better but I'm kinda glad I don't [20:57:51] dmehus: public fine? [20:58:03] I can't see anything private now [21:00:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.51, 3.53 [21:00:35] RhinosF1, replied via DM. Feel free to discuss with John and see his thoughts [21:00:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.89, 7.22, 6.27 [21:02:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.38, 3.47 [21:02:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.30, 6.25, 6.03 [21:04:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.01, 3.32 [21:05:20] CosmicAlpha, did you run the CA script? I see the rename seems to be done [21:05:40] dmehus: yes. [21:05:50] CosmicAlpha, ah, ty [21:05:58] no problem. [21:06:01] ah it was `commutativealgebrawiki` it was stuck on [21:06:05] yep [21:07:02] hrm [21:07:08] it doesn't seem to be finished yet, https://meta.miraheze.org/wiki/Special:CentralAuth/Techboyg5 [21:07:09] [url] Global account information for Techboyg5 - Miraheze Meta | meta.miraheze.org [21:07:26] It still hasn't completed `techwikiwiki` [21:07:28] Looks fine to me [21:07:32] 2 wikis left [21:07:36] oh [21:07:38] yeah [21:07:40] just saw [21:07:43] it wasn't done yet [21:07:53] is now [21:08:35] yep [21:08:38] thanks :) [21:08:47] no problem [21:14:15] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/JXfyu [21:14:16] [02miraheze/mediawiki] 07Universal-Omega 03177d271 - Install WMAU [21:14:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JXfyP [21:14:29] [02miraheze/mediawiki] 07Universal-Omega 03a0f0a42 - Update non-rel-extensions [21:16:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://git.io/JXf94 [21:16:15] [02miraheze/mediawiki] 07Universal-Omega 037c4c3f4 - Install WMAU [21:16:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JXf9w [21:16:23] [02miraheze/mediawiki] 07Universal-Omega 03153c101 - Update non-rel-extensions [21:17:21] !log [universalomega@test3] starting deploy of {'world': True, 'extensionlist': True} to skip [21:17:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.83, 3.57 [21:18:28] !log [universalomega@mw11] starting deploy of {'world': True, 'extensionlist': True} to all [21:18:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:20:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JXf5B [21:20:56] [02miraheze/mw-config] 07Universal-Omega 0380f1ca4 - Update extension-list [21:20:58] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [21:21:11] [02mw-config] 07Universal-Omega opened pull request 03#4203: T8211: install WMAU skin - 13https://git.io/JXf51 [21:21:33] !log [universalomega@test3] finished deploy of {'world': True, 'extensionlist': True} to skip - SUCCESS in 251s [21:21:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:59] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 111.36 ms [21:22:07] miraheze/mw-config - Universal-Omega the build passed. [21:22:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JXfd6 [21:22:30] [02miraheze/mw-config] 07Universal-Omega 0357508f9 - Update ManageWikiExtensions.php [21:22:31] [02mw-config] 07Universal-Omega synchronize pull request 03#4203: T8211: install WMAU skin - 13https://git.io/JXf51 [21:23:30] miraheze/mw-config - Universal-Omega the build passed. [21:25:49] !log [universalomega@mw11] finished deploy of {'world': True, 'extensionlist': True} to all - SUCCESS in 440s [21:25:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:43] !log [universalomega@test3] starting deploy of {'config': True} to skip [21:27:44] !log [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:27:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:28:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.57, 3.67 [21:32:41] test3 is getting annoying.... it is impossible to test anything there with it going down every 5 minutes with DB errors. Every time I make a change it seems to go down for an hour or longer sometimes... [21:35:57] I just proxied to one of my own wikis to test it. [21:36:07] [02mw-config] 07Universal-Omega closed pull request 03#4203: T8211: install WMAU skin - 13https://git.io/JXf51 [21:36:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JXJJY [21:36:10] [02miraheze/mw-config] 07Universal-Omega 0333999ae - T8211: install WMAU skin (#4203) [21:36:11] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [21:36:13] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:37:06] miraheze/mw-config - Universal-Omega the build passed. [21:38:32] !log [@test3] starting deploy of {'config': True} to skip [21:38:33] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:38:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:38] !log [@mw11] starting deploy of {'config': True} to all [21:38:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:48] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 9s [21:38:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:57] !log [universalomega@test3] starting deploy of {'l10n': True} to skip [21:39:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:11] !log [universalomega@mw11] starting deploy of {'l10n': True} to all [21:39:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:48] CosmicAlpha: the only log I found involved echo [21:41:55] But I still don't get why [21:42:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.89, 21.45, 19.82 [21:42:14] I think we need JohnLewis or paladox to look db side [21:43:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 18.43, 10.82, 7.27 [21:43:56] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.69, 23.38, 20.65 [21:44:17] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.52, 18.98, 16.72 [21:44:36] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.15, 6.79, 5.62 [21:45:54] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/MatomoAnalytics/maintenance/addMissingMatomos.php --wiki=loginwiki (END - exit=0) [21:45:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.51, 3.53 [21:46:01]