[00:03:19] <icinga-miraheze>	 PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:04:10] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.61, 21.16, 16.46
[00:05:31] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.13, 7.70, 5.26
[00:06:14] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.67, 9.70, 6.45
[00:06:49] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.79, 7.40, 5.34
[00:08:47] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.63, 7.24, 5.51
[00:09:30] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.47, 6.88, 5.59
[00:10:06] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.90, 21.91, 18.58
[00:10:10] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.55, 7.45, 6.31
[00:10:43] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.59, 6.48, 5.44
[00:11:30] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.48, 6.04, 5.44
[00:12:05] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.62, 6.31, 6.03
[00:14:03] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.73, 23.87, 20.03
[00:16:03] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.95, 21.35, 19.56
[00:20:08] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.85, 23.02, 20.60
[00:21:17] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.54, 3.04
[00:21:30] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 7.99, 6.47
[00:22:05] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.35, 22.67, 20.79
[00:23:17] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.24, 2.98
[00:23:30] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.83, 6.76, 6.20
[00:26:04] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.17, 19.49, 19.89
[00:27:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.53, 3.17
[00:31:16] <icinga-miraheze>	 RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[00:31:17] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.93, 2.96, 3.05
[01:45:38] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.43, 3.18
[01:47:38] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.13, 3.10
[01:51:46] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.91, 3.43
[01:53:46] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.04, 2.37, 1.67
[01:53:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.74, 3.43
[01:55:45] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.36, 1.96, 1.60
[01:55:51] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.97, 3.55
[01:57:43] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 1.57, 2.02, 1.68
[01:57:55] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.86, 3.57
[01:59:40] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.01, 1.70, 1.60
[02:04:02] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.10, 3.33
[02:21:14] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 3.64, 3.32
[02:23:14] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.61, 3.36
[02:25:16] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.21, 3.25
[02:33:22] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.52, 3.32
[02:35:21] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 2.94, 3.13
[03:21:33] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.57, 3.14
[03:23:33] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 3.07, 3.00
[04:22:42] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.11, 1.70, 1.30
[04:24:40] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.17, 1.36, 1.22
[04:39:00] <icinga-miraheze>	 PROBLEM - nocyclo.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nocyclo.tk' expires in 15 day(s) (Sat 04 Sep 2021 04:34:44 GMT +0000).
[04:44:13] <icinga-miraheze>	 PROBLEM - bn.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bn.gyaanipedia.com' expires in 15 day(s) (Sat 04 Sep 2021 04:37:53 GMT +0000).
[04:51:03] <icinga-miraheze>	 PROBLEM - incubator.nocyclo.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'incubator.nocyclo.tk' expires in 15 day(s) (Sat 04 Sep 2021 04:45:10 GMT +0000).
[05:05:48] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J0XoH
[05:05:50] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot 03a5793f8 - Bot: Update SSL cert for nocyclo.tk
[05:06:37] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J0XKp
[05:06:38] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot 0331a0ac4 - Bot: Update SSL cert for bn.gyaanipedia.com
[05:06:52] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J0X6t
[05:06:54] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot 03ded53bf - Bot: Update SSL cert for incubator.nocyclo.tk
[05:25:18] <icinga-miraheze>	 RECOVERY - incubator.nocyclo.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'incubator.nocyclo.tk' will expire on Wed 17 Nov 2021 04:06:46 GMT +0000.
[05:26:29] <icinga-miraheze>	 RECOVERY - nocyclo.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'nocyclo.tk' will expire on Wed 17 Nov 2021 04:05:10 GMT +0000.
[05:38:17] <icinga-miraheze>	 RECOVERY - bn.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'bn.gyaanipedia.com' will expire on Wed 17 Nov 2021 04:06:31 GMT +0000.
[07:51:24] <RhinosF1>	 paladox: if you're awake, there's a ticket about broken thumbnails
[10:39:46] <RhinosF1>	 JohnLewis: if doing with --no-updates then definately run rebuildall
[10:39:56] <JohnLewis>	 RhinosF1: I am
[10:40:01] <RhinosF1>	 Ok
[10:47:30] <RhinosF1>	 JohnLewis: does the actual command syntax look ok for https://github.com/miraheze/puppet/pull/1885/files
[10:47:30] <MirahezeBot>	 [url] mwdeploy: add rsync stuff by RhinosF1 · Pull Request #1885 · miraheze/puppet · GitHub | github.com
[10:48:54] <Not-da3c>	 [02puppet] 07RhinosF1 edited pull request 03#1885: mwdeploy: add rsync stuff - 13https://git.io/J0Rrq
[10:50:04] <JohnLewis>	 It looks okay to me
[10:50:45] <RhinosF1>	 Cool
[10:59:34] <Not-da3c>	 [02puppet] 07RhinosF1 reviewed pull request 03#1885 commit - 13https://git.io/J0HWu
[10:59:55] <Not-da3c>	 [02puppet] 07RhinosF1 reviewed pull request 03#1885 commit - 13https://git.io/J0HWb
[11:07:58] <Not-da3c>	 [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J0HgA
[11:07:59] <Not-da3c>	 [02miraheze/IncidentReporting] 07translatewiki 0303cd27b - Localisation updates from https://translatewiki.net.
[11:08:00] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:08:01] <Not-da3c>	 [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+2/-0/±0] 13https://git.io/J0Hgj
[11:08:02] <Not-da3c>	 [02miraheze/MatomoAnalytics] 07translatewiki 03e657f1a - Localisation updates from https://translatewiki.net.
[11:08:03] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:08:04] <Not-da3c>	 [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+2/-0/±0] 13https://git.io/J0H2e
[11:08:05] <Not-da3c>	 [02miraheze/DataDump] 07translatewiki 03cb95119 - Localisation updates from https://translatewiki.net.
[11:08:06] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:08:07] <Not-da3c>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±7] 13https://git.io/J0H2f
[11:08:08] <Not-da3c>	 [02miraheze/CreateWiki] 07translatewiki 038cf4ab7 - Localisation updates from https://translatewiki.net.
[11:08:09] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:08:10] <Not-da3c>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±3] 13https://git.io/J0H2J
[11:08:11] <Not-da3c>	 [02miraheze/ManageWiki] 07translatewiki 03e74f381 - Localisation updates from https://translatewiki.net.
[11:08:12] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:08:13] <Not-da3c>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+3/-0/±5] 13https://git.io/J0H2U
[11:08:14] <Not-da3c>	 [02miraheze/MirahezeMagic] 07translatewiki 03c2f44de - Localisation updates from https://translatewiki.net.
[11:08:15] <MirahezeBot>	 ...
[11:08:16] <Not-da3c>	 [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+3/-0/±1] 13https://git.io/J0H2T
[11:08:17] <Not-da3c>	 [02miraheze/landing] 07translatewiki 036bbdff6 - Localisation updates from https://translatewiki.net.
[11:08:18] <MirahezeBot>	 ...
[11:08:19] <Not-da3c>	 [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/J0H2k
[11:08:20] <Not-da3c>	 [02miraheze/WikiDiscover] 07translatewiki 0300b8f5c - Localisation updates from https://translatewiki.net.
[11:08:21] <MirahezeBot>	 ...
[11:08:50] <miraheze-github>	 miraheze/MatomoAnalytics - translatewiki the build passed.
[11:08:54] <miraheze-github_>	 miraheze/CreateWiki - translatewiki the build passed.
[11:08:59] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[11:09:01] <miraheze-github_>	 miraheze/WikiDiscover - translatewiki the build passed.
[11:09:08] <miraheze-github_>	 miraheze/ManageWiki - translatewiki the build passed.
[11:09:15] <miraheze-github_>	 miraheze/DataDump - translatewiki the build passed.
[11:09:22] <miraheze-github_>	 miraheze/IncidentReporting - translatewiki the build passed.
[11:09:29] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[11:11:34] <icinga-miraheze>	 PROBLEM - cp14 Current Load on cp14 is CRITICAL: CRITICAL - load average: 3.80, 3.07, 1.60
[11:17:26] <icinga-miraheze>	 PROBLEM - cp14 Current Load on cp14 is WARNING: WARNING - load average: 0.65, 1.99, 1.61
[11:23:15] <icinga-miraheze>	 PROBLEM - cp14 Current Load on cp14 is CRITICAL: CRITICAL - load average: 6.06, 2.80, 1.98
[11:25:10] <icinga-miraheze>	 PROBLEM - cp14 Current Load on cp14 is WARNING: WARNING - load average: 0.93, 1.93, 1.76
[11:27:14] <icinga-miraheze>	 RECOVERY - cp14 Current Load on cp14 is OK: OK - load average: 0.46, 1.42, 1.59
[11:27:43] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.75, 21.35, 17.27
[11:29:38] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.85, 19.16, 16.93
[12:31:30] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.55, 17.29, 14.61
[12:33:28] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.62, 16.83, 14.76
[12:49:03] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 2.95, 2.62
[12:51:02] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.28, 2.97, 2.66
[12:52:18] <RhinosF1>	 JohnLewis: any idea on https://phabricator.miraheze.org/T7837#158007
[12:52:19] <MirahezeBot>	 [url] ⚓ T7837 Images and thumbnails failing on uploaded graphics. | phabricator.miraheze.org
[12:53:03] <JohnLewis>	 Nope, not looked into it as I'm busy clearing the import backlog
[12:53:27] <JohnLewis>	 It would be paladox's best area anyway
[12:53:39] <RhinosF1>	 Ok
[12:53:46] <RhinosF1>	 I can't see any icinga alert
[12:54:15] <RhinosF1>	 So my best guess is it was a very unlucky upload right as it was grumpy
[12:59:08] <JohnLewis>	 My guess would be MW just failed to generate it
[12:59:37] <JohnLewis>	 Icinga wouldn't be of use to you, it would be graylog logs for mw
[13:03:29] <RhinosF1>	 More whether it managed to get the mount weird
[13:03:43] <RhinosF1>	 As we have the cron hack still trying to stop impact of that
[13:05:42] <JohnLewis>	 Also a possibility
[13:28:09] <icinga-miraheze>	 PROBLEM - ethene.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ethene.wiki' expires in 15 day(s) (Sat 04 Sep 2021 13:23:33 GMT +0000).
[13:30:42] <icinga-miraheze>	 PROBLEM - www.ethene.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ethene.wiki' expires in 15 day(s) (Sat 04 Sep 2021 13:23:33 GMT +0000).
[13:33:57] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J076d
[13:33:59] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot 03c33e2b5 - Bot: Update SSL cert for ethene.wiki
[13:36:18] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/J07Pg
[13:36:20] <Not-da3c>	 [02miraheze/ssl] 07MirahezeSSLBot 033db943d - Bot: Update SSL cert for www.ethene.wiki
[13:50:59] <RhinosF1>	 JohnLewis: are you doing an image upload?
[13:51:10] <JohnLewis>	 RhinosF1: yes
[13:51:22] <RhinosF1>	 Good
[13:51:29] <RhinosF1>	 RC feed is laggy always with them
[14:05:22] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.98, 6.81, 5.65
[14:05:31] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.00, 7.89, 5.80
[14:05:40] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.33, 8.57, 6.44
[14:05:59] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.89, 6.88, 5.71
[14:07:20] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.17, 6.24, 5.58
[14:07:31] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.50, 6.74, 5.62
[14:07:38] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.36, 7.37, 6.26
[14:07:57] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.10, 6.27, 5.63
[14:08:33] <icinga-miraheze>	 RECOVERY - ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Wed 17 Nov 2021 12:33:50 GMT +0000.
[14:09:39] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.67, 6.02, 5.89
[14:11:43] <icinga-miraheze>	 RECOVERY - www.ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Wed 17 Nov 2021 12:33:50 GMT +0000.
[14:26:10] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 6.88, 6.12
[14:28:08] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.38, 6.33, 6.01
[14:51:50] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.47, 3.17
[14:53:52] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.18, 3.09
[15:07:13] <Not-da3c>	 [02puppet] 07Universal-Omega reviewed pull request 03#1876 commit - 13https://git.io/J05P6
[15:10:26] <Not-da3c>	 [02puppet] 07Universal-Omega reviewed pull request 03#1876 commit - 13https://git.io/J05MJ
[15:41:33] <Not-da3c>	 [02puppet] 07RhinosF1 reviewed pull request 03#1876 commit - 13https://git.io/J0dUV
[15:42:05] <Not-da3c>	 [02puppet] 07RhinosF1 reviewed pull request 03#1876 commit - 13https://git.io/J0dUN
[15:53:14] <RhinosF1>	 JohnLewis: what's our biggest wiki with flagged revs on?
[15:53:24] <RhinosF1>	 Like in table size
[15:53:30] <RhinosF1>	 So revisions
[15:56:24] <JohnLewis>	 most revisions is the measure for biggest you want?
[15:57:20] <RhinosF1>	 JohnLewis: that's gonna get the best. I want one that will show a stupid flaggedtemplates table so I can do so maths on it.
[15:57:38] <RhinosF1>	 It will eventually need cleaning up if we want to save space
[16:05:30] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.15, 6.01, 4.56
[16:07:32] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.81, 5.09, 4.41
[16:12:20] <JohnLewis>	 RhinosF1: biggest flaggedtemplates table is barishalwiki at 20M
[16:14:04] <RhinosF1>	 JohnLewis: can you run select count(*) from flaggedtemplates where ft_rev_id not in (select fp_stable from flaggedpages); and without the where?
[16:14:17] <RhinosF1>	 What's that table like compared to others on that wiki too?
[16:14:32] <JohnLewis>	 You mean size wise?
[16:16:41] <RhinosF1>	 Yes
[16:17:47] <JohnLewis>	 https://www.irccloud.com/pastebin/X8WTds7k/
[16:19:32] <JohnLewis>	 RhinosF1: 7th largest table
[16:20:28] <RhinosF1>	 66% potential garbage is not too bad
[16:20:34] <RhinosF1>	 JohnLewis: what's bigger?
[16:22:26] <JohnLewis>	 searchindex, searchindex-index, pagelinks, recentchanges, templatelinks, text
[16:23:29] <RhinosF1>	 That's not surprising actually because 3 of them wikimedia don't use
[16:25:04] <RhinosF1>	 JohnLewis: 1.37 comes with a lot of data clean up
[16:25:14] <RhinosF1>	 Image table + flagged revs stuff mainly
[16:39:17] <JohnLewis>	 RhinosF1: https://phabricator.miraheze.org/T7740#158015 that for data cleanup too
[16:39:18] <MirahezeBot>	 [url] ⚓ T7740 Reconsider production usage of Elastic/CirrusSearch | phabricator.miraheze.org
[16:40:05] <RhinosF1>	 JohnLewis: yes that too
[17:04:44] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.82, 7.18, 5.91
[17:06:41] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.89, 5.89, 5.59
[17:21:40] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.04, 5.95, 5.28
[17:22:41] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.48, 1.71, 1.28
[17:23:37] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.12, 5.72, 5.28
[17:24:39] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.99, 1.51, 1.27
[18:20:32] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 4.27, 3.21
[18:22:34] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.66, 3.11
[18:24:35] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.19, 3.14, 2.98
[19:29:19] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.16, 7.46, 5.89
[19:30:20] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 6.98, 5.45
[19:30:46] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.17, 8.75, 6.28
[19:31:18] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.95, 7.23, 5.99
[19:32:01] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.17, 6.46, 5.26
[19:32:02] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.55, 3.18
[19:32:19] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.98, 7.44, 5.81
[19:34:17] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.71, 8.06, 6.21
[19:35:30] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.16, 7.67, 5.83
[19:36:17] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.49, 7.43, 6.21
[19:36:45] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.26, 7.56, 6.69
[19:36:55] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.35, 7.06, 5.81
[19:37:13] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.90, 6.80, 6.24
[19:37:32] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.71, 6.57, 5.66
[19:38:16] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.74, 6.42, 5.99
[19:38:44] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.93, 6.34, 6.35
[19:38:51] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.03, 6.33, 5.69
[19:39:45] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.99, 6.68, 6.22
[19:42:06] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.14, 3.18, 3.28
[19:52:24] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[20:37:19] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.37, 5.97, 5.13
[20:37:50] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.08, 20.64, 17.79
[20:39:16] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.39, 5.97, 5.22
[20:39:46] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.23, 19.08, 17.55
[20:43:07] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.67, 3.23
[20:47:10] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.20, 3.13
[21:00:10] <paladox>	 Let's begin the maintenance!
[21:00:24] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[21:00:46] <paladox>	 !log restart mariadb on db11
[21:02:18] <paladox>	 !log restart mariadb on db12
[21:03:30] <paladox>	 !log restart mariadb on db13
[21:03:50] <RhinosF1>	 Icinga dead too
[21:04:27] <RhinosF1>	 And everything green paladox
[21:04:29] <paladox>	 Maintenance over
[21:04:53] <RhinosF1>	 paladox: https://github.com/miraheze/mw-config/pull/4057
[21:04:53] <MirahezeBot>	 [url] Revert "Set all wikis into read only" by RhinosF1 · Pull Request #4057 · miraheze/mw-config · GitHub | github.com
[21:05:01] <paladox>	 oh
[21:05:01] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[21:05:05] <paladox>	 i've reverted already
[21:05:43] <miraheze-github>	 miraheze/mw-config - RhinosF1 the build passed.
[21:06:41] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[21:07:09] <MH-Discord>	 <Agent> wow, that was fast, I was expecting something akin to heart surgery or something
[21:08:09] <RhinosF1>	 Yeah no notifico is annoying
[21:08:36] <RhinosF1>	 paladox: do you have access to tweet maint over?
[21:08:50] <RhinosF1>	 @agent: it's fairly fast now
[21:09:01] <RhinosF1>	 It's just not the safest to do with RW traffic
[21:09:25] <RhinosF1>	 4m 9s of no edits
[21:09:57] * Voidwalker tweets
[21:10:16] <RhinosF1>	 Ty Voidwalker
[21:10:36] <paladox>	 i don't have access to the miraheze twitter account
[21:11:09] <RhinosF1>	 Void did it
[21:11:16] <RhinosF1>	 Voidwalker: did you do that CU
[21:11:41] <Voidwalker>	 not yet, been working on T7322
[21:13:39] <RhinosF1>	 Voidwalker: ack, I can guess what it'll say and if it does then feel free to relock for VOA
[21:15:49] <Voidwalker>	 Hmm, puppet looks broken on mw12 but still downtimed for something else I believe
[21:16:16] <paladox>	 !log 17:00:46: !log restart mariadb on db11
[21:16:19] <Voidwalker>	 also puppet disabled on test3 for the past 6/7 days, someone should probably put their name on that
[21:16:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:16:25] <paladox>	 !log 17:02:18: !log restart mariadb on db12
[21:16:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:16:38] <paladox>	 !log 17:03:30: !log restart mariadb on db13
[21:16:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:16:53] <paladox>	 Voidwalker: oh?
[21:17:31] <Voidwalker>	 yeah, check icinga
[21:17:44] <paladox>	 i'm having a look re mw12
[21:18:39] <paladox>	 ah
[21:26:13] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.58, 3.08
[21:26:22] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.07, 6.81, 5.29
[21:26:49] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.99, 7.49, 5.53
[21:28:20] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.67, 6.47, 5.36
[21:28:45] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.27, 6.27, 5.33
[21:30:15] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.84, 2.83, 2.91
[21:36:37] <icinga-miraheze>	 PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 seconds ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:36:45] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 28 seconds ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:37:18] <icinga-miraheze>	 PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:37:27] <Voidwalker>	 that looks odd
[21:37:33] <icinga-miraheze>	 PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:37:36] <icinga-miraheze>	 PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:37:39] <RhinosF1>	 Looking
[21:38:14] <RhinosF1>	 paladox:
[21:38:18] <RhinosF1>	 See ssl
[21:38:22] <paladox>	 that's me
[21:38:24] <RhinosF1>	 Notifco is broke
[21:38:25] <paladox>	 i fixed it
[21:38:33] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:38:35] <icinga-miraheze>	 PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:38:46] <icinga-miraheze>	 PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:39:59] <RhinosF1>	 I asked for notifico to be restarted
[21:40:24] <icinga-miraheze>	 PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.chouverse.cludisciples.cyou]
[21:40:26] <Voidwalker>	 who has access to notifico?
[21:40:46] <icinga-miraheze>	 RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[21:41:10] <RhinosF1>	 Voidwalker: it's 3rd party service
[21:41:24] <RhinosF1>	 I just pinged tk tech in #notifico
[21:41:26] <Voidwalker>	 ah, not something we can restart then
[21:41:33] <icinga-miraheze>	 RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[21:41:35] <icinga-miraheze>	 RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[21:41:48] <RhinosF1>	 Voidwalker: nope
[21:41:56] <RhinosF1>	 They responded in like 10 seconds
[21:41:59] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.87, 6.25, 5.24
[21:42:07] <RhinosF1>	 They reset it for us so it could rejoin
[21:42:24] <icinga-miraheze>	 RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[21:42:34] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:42:35] <icinga-miraheze>	 RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[21:42:35] <icinga-miraheze>	 RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:42:40] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:42:41] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[21:43:17] <icinga-miraheze>	 RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:43:57] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.10, 5.96, 5.24
[21:46:21] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.19, 7.17, 5.66
[21:47:14] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.35, 7.94, 6.41
[21:48:15] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.21, 6.69, 5.65
[21:48:38] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[21:48:49] <icinga-miraheze>	 PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[21:48:56] <RhinosF1>	 Night!
[21:49:20] <icinga-miraheze>	 PROBLEM - cp14 Puppet on cp14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[21:50:23] <icinga-miraheze>	 PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[21:51:42] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.52, 5.59, 5.82
[21:53:47] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.70, 3.30
[21:54:32] <icinga-miraheze>	 RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:55:46] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 3.33, 3.22
[21:59:20] <icinga-miraheze>	 RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:04:39] <icinga-miraheze>	 RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[22:04:49] <icinga-miraheze>	 RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Wed 17 Nov 2021 20:33:31 GMT +0000.
[22:04:53] <icinga-miraheze>	 RECOVERY - ref.undu.st - LetsEncrypt on sslhost is OK: OK - Certificate 'ref.undu.st' will expire on Wed 17 Nov 2021 20:28:20 GMT +0000.
[22:04:57] <icinga-miraheze>	 RECOVERY - meta.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 17 Nov 2021 20:31:21 GMT +0000.
[22:05:28] <icinga-miraheze>	 RECOVERY - www.fossilfighterswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com' will expire on Wed 17 Nov 2021 20:27:16 GMT +0000.
[22:05:29] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.04, 1.97, 1.48
[22:05:31] <icinga-miraheze>	 RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 17 Nov 2021 20:31:21 GMT +0000.
[22:05:47] <icinga-miraheze>	 RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 17 Nov 2021 20:31:21 GMT +0000.
[22:05:53] <icinga-miraheze>	 RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Wed 17 Nov 2021 20:27:45 GMT +0000.
[22:05:55] <icinga-miraheze>	 RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 17 Nov 2021 20:31:21 GMT +0000.
[22:05:57] <icinga-miraheze>	 RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 17 Nov 2021 20:31:21 GMT +0000.
[22:05:59] <icinga-miraheze>	 RECOVERY - privacy-wiki.0x.no - LetsEncrypt on sslhost is OK: OK - Certificate 'privacy-wiki.0x.no' will expire on Wed 17 Nov 2021 20:28:02 GMT +0000.
[22:06:01] <icinga-miraheze>	 RECOVERY - pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pornwiki.org' will expire on Wed 17 Nov 2021 20:28:35 GMT +0000.
[22:07:26] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.93, 1.62, 1.41
[22:27:09] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.02, 6.61, 5.95
[22:29:05] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.45, 6.27, 5.91
[23:34:57] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.74, 6.29, 5.29
[23:36:57] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.02, 5.84, 5.26
[23:39:45] <icinga-miraheze>	 PROBLEM - rupowerrangersfanon.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rupowerrangersfanon.tk could not be found
[23:52:57] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.47, 2.04, 1.47
[23:56:50] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.28, 1.82, 1.52
[23:58:45] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.79, 1.48, 1.43