[00:00:25] <icinga-miraheze>	 PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.fbpml.org All nameservers failed to answer the query.
[00:03:26] <icinga-miraheze>	 PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:03:47] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.00, 20.38, 18.02
[00:04:20] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.43, 6.10, 5.89
[00:05:14] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.12, 6.16, 5.13
[00:05:18] <icinga-miraheze>	 PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.44, 2.50, 1.62
[00:06:32] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 3.74, 3.31
[00:06:50] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.73, 7.34, 4.12
[00:07:17] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.14, 6.99, 5.53
[00:07:19] <icinga-miraheze>	 PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.09, 3.22, 2.00
[00:07:22] <icinga-miraheze>	 PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.hostnetbv.com.', 'ns1.hostnet.nl.', 'ns3.hostnetbv.nl.'], 'CNAME': 'fbpml.miraheze.org.'}
[00:09:14] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.89, 7.41, 5.93
[00:09:21] <icinga-miraheze>	 RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.55, 2.93, 2.04
[00:10:20] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.35, 6.75, 6.27
[00:10:45] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.25, 6.82, 6.54
[00:10:49] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.26, 7.63, 5.02
[00:10:51] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.18, 5.88, 3.93
[00:11:14] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.29, 7.47, 6.11
[00:11:17] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 7.23, 5.97
[00:12:40] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.83, 6.12, 6.32
[00:12:48] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.08, 5.53, 4.05
[00:13:14] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 7.03, 6.10
[00:14:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.88, 3.60
[00:14:45] <icinga-miraheze>	 RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.93, 5.00, 4.02
[00:14:49] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.78, 6.53, 5.16
[00:15:10] <CosmicAlpha>	 paladox: do you have any ideas on this one? https://www.irccloud.com/pastebin/v4n62iNQ/
[00:15:15] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.63, 6.12, 5.88
[00:15:17] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.03, 6.05, 5.82
[00:15:56] <paladox>	 Guess you need to insert data into a table?
[00:16:05] <paladox>	 Content table or that slots table
[00:17:48] <CosmicAlpha>	 paladox: on all wikis? I tried that on 3 wikis and same exact error, I wouldn't think that would be missing from all wikis.
[00:17:56] <paladox>	 Oh
[00:18:02] <paladox>	 I’m not sure then :/
[00:20:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.60, 3.54
[00:20:51] <paladox>	 https://github.com/miraheze/mediawiki/blob/REL1_36/includes/import/WikiRevision.php#L494
[00:20:52] <MirahezeBot>	 [url] mediawiki/WikiRevision.php at REL1_36 · miraheze/mediawiki · GitHub | github.com
[00:20:59] <paladox>	 Looks like it is the slots table?
[00:21:03] <paladox>	 CosmicAlpha:
[00:21:30] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.83, 5.87, 4.79
[00:22:19] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.31, 3.44
[00:23:27] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.77, 5.93, 4.91
[00:26:16] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 6.91, 6.28
[00:26:18] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 2.95, 3.29
[00:27:22] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.33, 5.21, 4.87
[00:28:16] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.45, 6.69, 6.27
[00:29:19] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.49, 5.99, 5.16
[00:33:25] <icinga-miraheze>	 RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:37:08] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.55, 5.55, 5.45
[00:41:02] <icinga-miraheze>	 RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.69, 4.35, 4.93
[00:46:40] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.86, 3.45
[00:47:47] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 12.26, 16.25, 23.23
[00:54:18] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.21, 3.33
[00:57:47] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.10, 14.82, 19.55
[01:08:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.46, 3.34
[01:10:18] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.16, 3.07, 3.22
[01:16:39] <icinga-miraheze>	 PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.67, 1.81, 1.16
[01:18:39] <icinga-miraheze>	 PROBLEM - cp15 Current Load on cp15 is CRITICAL: CRITICAL - load average: 4.08, 3.87, 2.08
[01:22:38] <icinga-miraheze>	 PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 0.34, 1.96, 1.70
[01:24:38] <icinga-miraheze>	 RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.41, 1.45, 1.54
[01:29:35] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 4.03, 3.50
[01:33:24] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.82, 3.55
[01:35:14] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.28, 8.82, 5.66
[01:35:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.96, 4.08, 3.66
[01:38:20] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.42, 6.76, 5.23
[01:41:15] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 7.44, 6.14
[01:42:20] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.06, 6.53, 5.57
[01:42:53] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.82, 3.75
[01:43:15] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.44, 6.78, 6.06
[01:50:28] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.80, 4.49, 3.93
[01:56:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.10, 3.42, 3.68
[02:04:18] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 2.89, 3.33
[02:08:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.65, 3.56
[02:10:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.39, 3.46
[02:12:18] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 3.12, 3.37
[03:31:00] <CosmicAlpha>	 paladox: hmmm... any ideas for a fix? I don't think anything should need done on the database.
[03:40:04] <Not-4bf1>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6pI3
[03:40:05] <Not-4bf1>	 [02miraheze/mw-config] 07Universal-Omega 0308ab5b9 - Remove requirement for DynamicPageList
[03:41:12] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[04:03:35] <MirahezeLSBot>	 !log [@test3] starting deploy of {'config': True} to skip
[04:03:36] <MirahezeLSBot>	 !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
[04:03:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:03:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:04:53] <MirahezeLSBot>	 !log [@mw11] starting deploy of {'config': True} to all
[04:04:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:05:03] <MirahezeLSBot>	 !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 10s
[04:05:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:04:30] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.57, 6.86, 5.35
[06:06:30] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.73, 7.68, 5.84
[06:07:05] <icinga-miraheze>	 alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[06:09:55] <CosmicAlpha>	 Rise in exceptions by about 1.7K?
[06:10:29] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.90, 7.48, 6.19
[06:12:05] <icinga-miraheze>	 ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[06:12:28] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.30, 7.94, 6.52
[06:13:38] <CosmicAlpha>	 5000 DBConnection exceptions again
[06:14:28] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.86, 7.94, 6.69
[06:22:26] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.00, 6.57, 6.60
[07:38:16] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.78, 7.22, 6.43
[07:40:15] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.22, 6.69, 6.32
[07:54:03] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6335 files, 49.23MB, 2021-10-13 07:52:00 (1.1 weeks ago)
[08:11:21] <CosmicAlpha>	 .tell JohnLewis: any ideas on https://phabricator.miraheze.org/T8184#165279? I'm out of ideas with this script. I don't what to do now. That happened on 3/3 wikis I tried it on.
[08:11:21] <MirahezeBot>	 CosmicAlpha: I'll pass that on when JohnLewis is around.
[08:21:06] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.25, 6.87, 6.49
[08:23:05] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.30, 6.50, 6.41
[08:50:57] <MirahezeLSBot>	 !log [rhinos@test3] sudo -u www-data php /srv/mediawiki/w/maintenance/purgeParserCache.php --wiki=test3wiki --age=604800 (END - exit=0)
[08:51:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:57:40] <Spookreeeno>	 CosmicAlpha: how do we purge parser cache?
[09:03:11] <Spookreeeno>	 Reception123: oing
[09:03:13] <Spookreeeno>	 ping
[09:05:41] <Spookreeeno>	 !log run purgeParserCache everywhere in screen on mwtask1
[09:05:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:15:21] <Reception123>	 Spookreeeno: hm?
[09:17:08] <Spookreeeno>	 Reception123: parser cache is evil
[09:17:33] <Reception123>	 yup
[09:17:39] <Reception123>	 though checkstorage is 10 times worse
[09:21:47] <Spookreeeno>	 Reception123: https://phabricator.miraheze.org/T8193#165285
[09:21:48] <MirahezeBot>	 [url] ⚓ T8193 Reduce Main Page cache lifetime to 1 day | phabricator.miraheze.org
[09:22:02] <Spookreeeno>	 Reception123: it's logic is illogical
[09:22:57] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.52, 7.07, 6.56
[09:24:56] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.63, 6.46, 6.39
[09:30:58] <MirahezeLSBot>	 !log [rhinos@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/purgeParserCache.php --age=604800 (END - exit=0)
[09:31:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:07:39] <Not-4bf1>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiJUE
[11:07:40] <Not-4bf1>	 [02miraheze/CreateWiki] 07translatewiki 03cc571d7 - Localisation updates from https://translatewiki.net.
[11:07:41] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:42] <Not-4bf1>	 [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiJUu
[11:07:43] <Not-4bf1>	 [02miraheze/landing] 07translatewiki 03260f40f - Localisation updates from https://translatewiki.net.
[11:07:44] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:45] <Not-4bf1>	 [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiJUz
[11:07:46] <Not-4bf1>	 [02miraheze/DataDump] 07translatewiki 0347e8db4 - Localisation updates from https://translatewiki.net.
[11:07:47] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:48] <Not-4bf1>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±7] 13https://git.io/JiJU2
[11:07:49] <Not-4bf1>	 [02miraheze/MirahezeMagic] 07translatewiki 0394c59c6 - Localisation updates from https://translatewiki.net.
[11:07:50] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:51] <Not-4bf1>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JiJUa
[11:07:52] <Not-4bf1>	 [02miraheze/ManageWiki] 07translatewiki 03d8049c5 - Localisation updates from https://translatewiki.net.
[11:07:53] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:54] <Not-4bf1>	 [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiJUV
[11:07:55] <Not-4bf1>	 [02miraheze/WikiDiscover] 07translatewiki 034c125d0 - Localisation updates from https://translatewiki.net.
[11:07:56] <MirahezeBot>	 ...
[11:08:40] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[11:09:23] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[11:10:02] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[11:18:48] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[11:20:12] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[11:21:16] <miraheze-github>	 miraheze/WikiDiscover - translatewiki the build passed.
[11:30:40] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.59, 6.89, 5.42
[11:31:31] <icinga-miraheze>	 PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 10.01, 6.72, 3.69
[11:32:39] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.02, 7.03, 5.66
[11:33:29] <icinga-miraheze>	 PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 7.74, 7.20, 4.25
[11:34:39] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.39, 6.05, 5.47
[11:35:27] <icinga-miraheze>	 RECOVERY - db13 Current Load on db13 is OK: OK - load average: 2.29, 5.31, 3.90
[12:13:47] <icinga-miraheze>	 PROBLEM - patternarchive.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'patternarchive.online' expires in 15 day(s) (Sat 06 Nov 2021 12:05:42 GMT +0000).
[12:17:55] <icinga-miraheze>	 PROBLEM - www.puzzles.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.puzzles.wiki' expires in 15 day(s) (Sat 06 Nov 2021 12:13:14 GMT +0000).
[12:21:47] <icinga-miraheze>	 PROBLEM - puzzles.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.puzzles.wiki' expires in 15 day(s) (Sat 06 Nov 2021 12:13:14 GMT +0000).
[12:23:22] <icinga-miraheze>	 PROBLEM - www.electrowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electrowiki.org' expires in 15 day(s) (Sat 06 Nov 2021 12:16:39 GMT +0000).
[12:23:32] <icinga-miraheze>	 PROBLEM - electrowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electrowiki.org' expires in 15 day(s) (Sat 06 Nov 2021 12:16:39 GMT +0000).
[13:45:35] <icinga-miraheze>	 PROBLEM - dreamsit.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dreamsit.com.br' expires in 15 day(s) (Sat 06 Nov 2021 13:36:51 GMT +0000).
[13:52:18] <icinga-miraheze>	 PROBLEM - www.burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.burnout.wiki' expires in 15 day(s) (Sat 06 Nov 2021 13:49:27 GMT +0000).
[14:00:32] <icinga-miraheze>	 PROBLEM - reviwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'reviwiki.info' expires in 15 day(s) (Sat 06 Nov 2021 13:52:02 GMT +0000).
[14:20:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.37, 2.86
[14:22:18] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.28, 2.89
[15:16:41] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.30, 2.86
[15:18:34] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 3.09, 2.84
[15:19:36] <Not-4bf1>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiTT1
[15:19:37] <Not-4bf1>	 [02miraheze/puppet] 07paladox 03fce107d - mon2: Increase MaxConcurrentChecks to 12
[15:22:22] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.53, 3.06
[15:26:18] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 3.37, 3.13
[16:16:02] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.45, 20.61, 16.02
[16:16:05] <Reception123>	 paladox: hi, any thoughts? https://phabricator.miraheze.org/T8184#165303
[16:16:06] <MirahezeBot>	 [url] ⚓ T8184 Editing and contributions broken on FAMEPedia | phabricator.miraheze.org
[16:16:19] <paladox>	 nope :/
[16:18:01] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 11.94, 17.66, 15.51
[16:36:30] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 7.12, 5.11
[16:38:30] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.49, 6.25, 5.04
[16:46:21] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.41, 2.99
[16:48:18] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.14, 2.95
[16:54:05] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.38, 3.11
[16:56:02] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.21, 3.07
[18:05:46] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.73, 6.09, 5.11
[18:07:46] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.19, 5.73, 5.10
[18:13:12] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.81, 6.98, 5.84
[18:13:33] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 8.53, 6.30
[18:15:11] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.14, 6.33, 5.75
[18:15:31] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.75, 7.88, 6.32
[18:17:16] <MH-Discord>	 <Lake> hey @Reception123, sorry for pinging you again, but regarding [[phab:T8173]], I still don't see the domain whitelisted on CSP
[18:17:17] <MH-Discord>	 <Wiki-Bot> https://phabricator.miraheze.org/T8173
[18:17:18] <MirahezeBot>	 [url] ⚓ T8173 Whitelist GoogleAPIs | phabricator.miraheze.org
[18:17:23] <MH-Discord>	 <Lake> (nor cdn.jsdelivr.com)
[18:19:29] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.13, 6.73, 6.19
[18:21:15] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.45, 3.15
[18:23:15] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.07, 3.04
[18:23:27] <Reception123>	 @Lake Oh I didn't see you pinged me before, let me look
[18:37:15] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.06, 2.93
[18:39:15] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.17, 3.00
[18:50:05] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.48, 3.10
[18:52:02] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.33, 3.10
[19:06:37] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.16, 5.99, 5.19
[19:08:37] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.34, 5.62, 5.14
[19:19:15] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.26, 3.02
[19:21:15] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.47, 3.13
[19:23:15] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.35, 3.21, 3.07
[20:41:13] <CosmicAlpha>	 paladox: I think I figured out the slot issue. I submitted upstream patch https://gerrit.wikimedia.org/r/732702 I also submitted https://gerrit.wikimedia.org/r/c/mediawiki/tools/mwdumper/+/732699 by the way to apply the changes I made locally to get that working.
[20:43:49] <CosmicAlpha>	 That's four total patches I think (three to core and one to mwdumper) to get checkStorage working.... that script was so out-of-date.... still haven't tested latest one but will in an hour or two.
[20:44:28] <paladox>	 requires you to set role
[20:44:34] <paladox>	 so i guess the one with "1">
[21:05:54] <Not-4bf1>	 [02mw-config] 07ugochimobi opened pull request 03#4171: otherProjectsLinks on FAMEPedia and FAMEData - 13https://git.io/JiIfW
[21:08:19] <CosmicAlpha>	 MediaWiki 1.37.0-rc.0 is out now.
[21:11:49] <Not-4bf1>	 [02miraheze/mediawiki] 07Universal-Omega pushed 0313 commits to 03REL1_37 [+0/-0/±38] 13https://git.io/JiIJc
[21:11:50] <Not-4bf1>	 [02miraheze/mediawiki] 07umherirrender 0330b1757 - Use LocalFile::getHookRunner instead of LocalFile::hookRunner
[21:11:52] <Not-4bf1>	 [02miraheze/mediawiki] 07reedy 03f317cae - Upgrading zordius/lightncandy (v1.2.5 => v1.2.6)
[21:11:53] <Not-4bf1>	 [02miraheze/mediawiki] 07cicalese 03161865d - Update git submodules
[21:11:55] <Not-4bf1>	 [02miraheze/mediawiki] ... and 10 more commits.
[21:57:50] <CosmicAlpha>	 paladox: are you around and able to create a database backup of famepediawiki? I ran checkStorage.php on test3wiki, and it seemed to work there, but would rather be safe for doing it on the first production wiki.
[21:58:18] <MH-Discord>	 <paladox> I’m currently mobile
[22:00:21] <CosmicAlpha>	 paladox: alright, I might just do it since we do have a full xml backup already, just prior to the issue, and no edits since, and it did work on test3. And I really don't want to keep them waiting any longer, since that's the only wiki with such a huge affect, that prevents editing completely.
[22:00:32] <MH-Discord>	 <paladox> Ok
[22:05:05] <CosmicAlpha>	 !log [universalomega@mwtask1]:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/storage/checkStorage.php --wiki=testwiki --fix /home/reception/wikibackups16102021/wikibackups/testwiki.xml
[22:05:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:36:43] <CosmicAlpha>	 paladox: worked on testwiki and test3wiki, but on famepediawiki, it gave `[d5ca4d7c36660c9ed5348abb] [no req] Error: Call to undefined method HistoryBlobStub::uncompress()` on the script itself, which is what it's designed to fix. My only guess is because on Famepedia, there are some current revisions affected as well.
[22:37:01] <MH-Discord>	 <paladox> Oh :/
[22:48:16] <CosmicAlpha>	 paladox: oh, if editing doesn't work at all, then the importer can't commit the edit, therefore won't fix, it seems... https://www.irccloud.com/pastebin/6bQa7IMA/
[22:52:34] <CosmicAlpha>	 GOT IT! Running it again and again works, as it commits changes, then fatals edit, but still commits so keep running worked.
[22:54:29] <CosmicAlpha>	 !log universalomega@mwtask1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/storage/checkStorage.php --wiki=famepediawiki --fix /home/universalomega/famepediawiki.xml
[22:54:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:04:31] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.47, 7.72, 5.27
[23:04:32] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.67, 18.21, 14.89
[23:05:26] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 13.20, 8.05, 5.36
[23:05:35] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.33, 7.46, 5.40
[23:05:47] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.61, 7.47, 5.24
[23:05:48] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.85, 4.48, 3.22
[23:05:48] <icinga-miraheze>	 PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:05:54] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True} to skip
[23:06:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:06:17] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.01, 23.31, 16.75
[23:06:31] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.43, 20.72, 16.17
[23:06:37] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.53, 8.88, 5.87
[23:06:42] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.82, 7.66, 5.44
[23:07:35] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.49, 6.45, 5.29
[23:07:46] <icinga-miraheze>	 RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates).
[23:07:46] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.08, 6.61, 5.20
[23:08:17] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.83, 20.24, 16.46
[23:08:31] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.56, 20.39, 16.63
[23:08:42] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.73, 6.16, 5.16
[23:09:24] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.65, 6.85, 5.56
[23:09:41] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.72, 3.70, 3.20
[23:10:30] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.65, 6.76, 5.88
[23:10:42] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - FAIL: [256, 0, 0, 0] in 287s
[23:10:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:11:23] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.59, 5.80, 5.33
[23:12:36] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 2.76, 6.35, 5.99
[23:13:34] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 3.12, 3.08
[23:15:14] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True, 'force': True, 'ignoretime': True} to skip
[23:15:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:20:21] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True, 'force': True, 'ignoretime': True} to skip - FAIL: [256, 0, 0, 0] in 306s
[23:20:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:21:13] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True} to skip
[23:21:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:23:52] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - FAIL: [256, 0, 0, 0] in 159s
[23:23:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:24:14] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'gitinfo': True, 'folders': 'w/vendor'} to skip
[23:24:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:24:19] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'gitinfo': True, 'folders': 'w/vendor'} to skip - SUCCESS in 4s
[23:24:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:25:14] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True} to skip
[23:25:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:27:41] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 147s
[23:27:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:36:31] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.93, 8.43, 6.34
[23:37:23] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.01, 6.44, 5.02
[23:38:17] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.38, 19.85, 16.41
[23:38:30] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.49, 6.94, 6.04
[23:39:23] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.92, 5.54, 4.87
[23:40:17] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.23, 18.26, 16.28
[23:40:30] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.62, 5.76, 5.72
[23:51:42] <MH-Discord>	 <Agent> hmm, I got the following error while enabling CommentStreams: The following errors occurred: The extension CommentStreams failed to be installed.
[23:52:55] <CosmicAlpha>	 Agent: looking, would be an intall sql configuration error in ManageWikiExtensions.php.
[23:55:02] <CosmicAlpha>	 Agent: was the extension disabled and re-enabled?
[23:55:15] <CosmicAlpha>	 *attempted to be reenabled.
[23:56:04] <MH-Discord>	 <Agent> CosmicAlpha: yes
[23:59:41] <Not-4bf1>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JiIlV
[23:59:42] <Not-4bf1>	 [02miraheze/mw-config] 07Universal-Omega 03d0fdede - Fix enabling CommentStreams
[23:59:44] <Not-4bf1>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3
[23:59:45] <Not-4bf1>	 [02mw-config] 07Universal-Omega opened pull request 03#4172: Fix enabling CommentStreams - 13https://git.io/JiIlw
[23:59:58] <CosmicAlpha>	 Agent: ^