[00:01:28] <icinga-miraheze>	 RECOVERY - es.nocyclo.tk - reverse DNS on sslhost is OK: SSL OK - es.nocyclo.tk reverse DNS resolves to cp12.miraheze.org - CNAME FLAT
[00:10:05] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.44, 22.60, 19.42
[00:14:58] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JX6IH
[00:15:00] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega 037f08d37 - Fix tabbing
[00:15:55] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.58, 3.26
[00:16:10] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:17:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.35, 3.21
[00:18:22] <icinga-miraheze>	 PROBLEM - meta.nocyclo.tk - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3
[00:18:22] <icinga-miraheze>	 -packages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: meta.nocyclo.tk., meta.nocyclo.tk.
[00:18:51] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.14, 6.21, 4.96
[00:24:54] <MirahezeLSBot>	 !log [@test3] starting deploy of {'config': True} to skip
[00:24:55] <MirahezeLSBot>	 !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
[00:24:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:25:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:25:29] <icinga-miraheze>	 PROBLEM - es.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.nocyclo.tk could not be found
[00:25:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.44, 3.25
[00:29:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.16, 3.19
[00:32:12] <icinga-miraheze>	 PROBLEM - en.nocyclo.tk - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/d
[00:32:12] <icinga-miraheze>	 ackages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: en.nocyclo.tk., en.nocyclo.tk.
[00:32:30] <icinga-miraheze>	 RECOVERY - es.nocyclo.tk - reverse DNS on sslhost is OK: SSL OK - es.nocyclo.tk reverse DNS resolves to cp12.miraheze.org - CNAME FLAT
[00:38:38] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.37, 3.26
[00:39:05] <icinga-miraheze>	 RECOVERY - en.nocyclo.tk - reverse DNS on sslhost is OK: SSL OK - en.nocyclo.tk reverse DNS resolves to cp12.miraheze.org - CNAME FLAT
[00:39:18] <icinga-miraheze>	 RECOVERY - meta.nocyclo.tk - reverse DNS on sslhost is OK: SSL OK - meta.nocyclo.tk reverse DNS resolves to cp12.miraheze.org - CNAME FLAT
[00:40:32] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.50, 3.31
[00:41:25] <MirahezeLSBot>	 !log [@mw11] starting deploy of {'config': True} to all
[00:41:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:41:46] <MirahezeLSBot>	 !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 20s
[00:41:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:42:27] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.46, 3.32
[00:46:15] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.69, 4.36, 3.66
[00:47:30] <Not-616a>	 [02puppet] 07Universal-Omega opened pull request 03#2120: deploy-mediawiki: use `--save-gitinfo` for rebuildVersionCache - 13https://git.io/JX6Oi
[00:48:17] <icinga-miraheze>	 PROBLEM - meta.nocyclo.tk - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3
[00:48:17] <icinga-miraheze>	 -packages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: meta.nocyclo.tk., meta.nocyclo.tk.
[00:51:58] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 4.00, 3.74
[00:53:53] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.69, 4.84, 4.09
[00:56:03] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.40, 3.90, 5.79
[01:00:02] <icinga-miraheze>	 PROBLEM - incubator.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for incubator.nocyclo.tk could not be found
[01:02:03] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.01, 4.45, 5.38
[01:02:05] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.75, 19.45, 23.46
[01:04:03] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.10, 4.21, 5.18
[01:06:03] <icinga-miraheze>	 RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.87, 4.00, 4.98
[01:08:05] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.35, 15.14, 20.17
[01:09:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.75, 3.96
[01:11:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 4.16, 4.10
[01:19:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.87, 4.00
[01:21:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.23, 4.54, 4.24
[01:24:15] <icinga-miraheze>	 PROBLEM - en.nocyclo.tk - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/d
[01:24:16] <icinga-miraheze>	 ackages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: en.nocyclo.tk., en.nocyclo.tk.
[01:37:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.66, 3.90
[01:39:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.84, 3.93
[01:40:01] <icinga-miraheze>	 PROBLEM - nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nocyclo.tk could not be found
[01:41:31] <icinga-miraheze>	 PROBLEM - es.nocyclo.tk - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/d
[01:41:31] <icinga-miraheze>	 ackages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: es.nocyclo.tk., es.nocyclo.tk.
[01:43:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.82, 3.92
[01:44:52] <icinga-miraheze>	 RECOVERY - en.nocyclo.tk - reverse DNS on sslhost is OK: SSL OK - en.nocyclo.tk reverse DNS resolves to cp12.miraheze.org - CNAME FLAT
[01:45:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.95, 3.95
[01:46:51] <icinga-miraheze>	 RECOVERY - incubator.nocyclo.tk - reverse DNS on sslhost is OK: SSL OK - incubator.nocyclo.tk reverse DNS resolves to cp12.miraheze.org - CNAME FLAT
[01:53:43] <icinga-miraheze>	 PROBLEM - en.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.nocyclo.tk could not be found
[01:55:30] <icinga-miraheze>	 PROBLEM - incubator.nocyclo.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for incubator.nocyclo.tk could not be found
[02:01:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.83, 4.00
[02:05:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 4.22, 4.12
[02:11:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.83, 4.00
[02:13:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.23, 4.23, 4.11
[02:17:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.15, 3.29, 3.77
[02:20:03] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.46, 4.82, 3.58
[02:20:05] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.61, 17.19, 14.68
[02:22:03] <icinga-miraheze>	 RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.93, 4.42, 3.58
[02:22:05] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.43, 17.59, 15.13
[02:27:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.67, 2.65, 3.28
[03:10:05] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.49, 20.18, 15.44
[03:12:05] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.18, 17.86, 15.16
[04:37:13] <icinga-miraheze>	 PROBLEM - nl.xliving.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nl.xliving.tk could not be found
[06:08:21] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.60, 6.08, 3.85
[06:14:21] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.16, 6.96, 5.07
[06:16:21] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.62, 6.07, 4.97
[06:41:03] <icinga-miraheze>	 PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.90, 1.66, 1.00
[06:43:03] <icinga-miraheze>	 RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.89, 1.35, 0.97
[07:03:22] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JXii1
[07:03:24] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega 0379b4cc9 - Update LocalSettings.php
[07:03:25] <Not-616a>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3
[07:03:58] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.04, 6.96, 6.27
[07:04:21] <Not-616a>	 [02mw-config] 07Universal-Omega opened pull request 03#4213: T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki - 13https://git.io/JXiPJ
[07:05:22] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[07:05:54] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.84, 6.66, 6.25
[07:06:09] <Not-616a>	 [02mw-config] 07Universal-Omega closed pull request 03#4213: T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki - 13https://git.io/JXiPJ
[07:06:10] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1
[07:06:12] <Not-616a>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3
[07:06:13] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXiPp
[07:06:15] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega 037f6450e - T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki (#4213)
[07:07:18] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[07:11:25] <MirahezeLSBot>	 !log [@mw11] starting deploy of {'config': True} to all
[07:11:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:11:33] <MirahezeLSBot>	 !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 8s
[07:11:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:21:19] <Not-616a>	 [02puppet] 07Universal-Omega closed pull request 03#2118: Run l10nupdate once a week rather than every day - 13https://git.io/JXKpu
[07:21:21] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.78, 7.48, 6.84
[07:23:18] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.92, 6.82, 6.68
[07:24:16] <MirahezeLSBot>	 !log [@test3] starting deploy of {'config': True} to skip
[07:24:17] <MirahezeLSBot>	 !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
[07:24:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:24:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:27:11] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.38, 6.40, 6.56
[07:55:14] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.55, 7.03, 6.61
[07:59:08] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.48, 6.76, 6.60
[08:30:53] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4213-Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JXPq8
[08:30:55] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega 030836981 - Revert "T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki (#4213)"
[08:30:56] <Not-616a>	 [02mw-config] 07Universal-Omega created branch 03revert-4213-Universal-Omega-patch-1 - 13https://git.io/vbvb3
[08:30:58] <Not-616a>	 [02mw-config] 07Universal-Omega opened pull request 03#4214: Revert "T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki" - 13https://git.io/JXPqR
[08:31:04] <Not-616a>	 [02mw-config] 07Universal-Omega closed pull request 03#4214: Revert "T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki" - 13https://git.io/JXPqR
[08:31:06] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4213-Universal-Omega-patch-1
[08:31:07] <Not-616a>	 [02mw-config] 07Universal-Omega deleted branch 03revert-4213-Universal-Omega-patch-1 - 13https://git.io/vbvb3
[08:31:09] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXPqE
[08:31:10] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega 03fabe13f - Revert "T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki (#4213)" (#4214)
[08:31:12] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4214-revert-4213-Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JXPqz
[08:31:13] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega 03eaf8b6d - Revert "Revert "T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki (#4213)" (#4214)"
[08:31:15] <Not-616a>	 [02mw-config] 07Universal-Omega created branch 03revert-4214-revert-4213-Universal-Omega-patch-1 - 13https://git.io/vbvb3
[08:31:16] <Not-616a>	 [02mw-config] 07Universal-Omega opened pull request 03#4215: Revert "Revert "T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki"" - 13https://git.io/JXPqw
[08:32:09] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[08:32:18] <miraheze-github_>	 miraheze/mw-config - Universal-Omega the build passed.
[08:32:23] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[08:36:14] <MirahezeLSBot>	 !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all
[08:36:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:36:20] <MirahezeLSBot>	 !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 5s
[08:36:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:42:41] <RhinosF1>	 CosmicAlpha: if pull == repos should we keep it in the log
[08:45:04] <CosmicAlpha>	 RhinosF1: what do you mean? You mean with 'pull': 'config' for example? If that's what you mean then yeah, eventually I want to adapt the pull argument to be able to be used to pull submodules within world, though I'm not really sure how to go about a good implementation for that right now, if we can't do that then it can probably be removed.
[08:47:30] <RhinosF1>	 CosmicAlpha: if you're only pulling the repo you are deploying, drop saying pull in !log
[08:47:51] <RhinosF1>	 We can think about submodules
[08:48:03] <RhinosF1>	 I want to make the rsync logic more expandable first
[08:48:10] <RhinosF1>	 That should make anything else easier
[08:49:36] <RhinosF1>	 It's
[08:49:47] <RhinosF1>	 It's probably got a few command injection issues
[08:50:02] <RhinosF1>	 But I don't think it matters because it runs as your user
[08:52:04] <CosmicAlpha>	 RhinosF1: yeah I guess we can do that with the log then, though I don't think it's a necessity either.
[08:53:36] <RhinosF1>	 CosmicAlpha: It seems cleaner in my head
[08:54:33] <MirahezeLSBot>	 !log [@test3] starting deploy of {'config': True} to skip
[08:54:34] <MirahezeLSBot>	 !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
[08:54:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:54:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:08:57] <icinga-miraheze>	 PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.66, 6.92, 6.35
[09:10:54] <icinga-miraheze>	 RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.61, 6.44, 6.24
[09:46:01] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:46:36] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.
[09:46:36] <icinga-miraheze>	 (hostname, 'A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.000712394714355 seconds
[09:53:36] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - vedopedia.witches-empire.com All nameservers failed to answer the query.
[09:57:03] <icinga-miraheze>	 PROBLEM - cp15 Current Load on cp15 is CRITICAL: CRITICAL - load average: 2.36, 1.62, 0.81
[09:59:03] <icinga-miraheze>	 RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.61, 1.21, 0.76
[10:14:22] <icinga-miraheze>	 RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Fri 14 Jan 2022 05:02:37 GMT +0000.
[10:23:44] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:30:47] <icinga-miraheze>	 RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Fri 14 Jan 2022 05:02:37 GMT +0000.
[10:47:13] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:54:11] <icinga-miraheze>	 RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Fri 14 Jan 2022 05:02:37 GMT +0000.
[11:24:44] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.
[11:24:44] <icinga-miraheze>	 (hostname, 'A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.00400710105896 seconds
[11:31:48] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - vedopedia.witches-empire.com All nameservers failed to answer the query.
[11:38:44] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:39:04] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.
[11:39:05] <icinga-miraheze>	 (hostname, 'A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.001636505126953 seconds
[11:52:53] <icinga-miraheze>	 RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Fri 14 Jan 2022 05:02:37 GMT +0000.
[11:53:17] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - vedopedia.witches-empire.com All nameservers failed to answer the query.
[12:00:36] <icinga-miraheze>	 (hostname, 'A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.001710414886475 seconds
[12:07:30] <icinga-miraheze>	 PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - vedopedia.witches-empire.com All nameservers failed to answer the query.
[13:00:35] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.21, 7.54, 5.76
[13:02:32] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.89, 6.23, 5.49
[13:38:27] <icinga-miraheze>	 PROBLEM - wiki.aridia.space - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.aridia.space' expires in 7 day(s) (Thu 18 Nov 2021 13:36:19 GMT +0000).
[13:47:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.36, 3.04
[13:49:47] <icinga-miraheze>	 PROBLEM - wiki.wilderyogi.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.wilderyogi.eu' expires in 7 day(s) (Thu 18 Nov 2021 13:46:36 GMT +0000).
[13:49:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.47, 3.11
[13:51:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.30, 3.09
[14:20:10] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.51, 3.08
[14:22:05] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.40, 3.34, 3.06
[14:31:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.46, 3.17
[14:33:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.18, 3.11
[14:47:03] <icinga-miraheze>	 PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.70, 1.93, 1.16
[14:49:03] <icinga-miraheze>	 RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.47, 1.36, 1.04
[15:42:17] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.77, 3.18
[15:44:11] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.79, 3.26
[15:46:06] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.35, 3.15
[16:08:24] <Not-616a>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JX1xh
[16:08:25] <Not-616a>	 [02miraheze/dns] 07Reception123 038c9c5eb - add conlang.tk zone
[16:12:09] <Not-616a>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JX1hf
[16:12:11] <Not-616a>	 [02miraheze/dns] 07Reception123 039653d88 - add docse.tk zone
[16:12:16] <Not-616a>	 [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/JX1hT
[16:12:18] <Not-616a>	 [02miraheze/ssl] 07Reception123 037590835 - add grayravens.com cert
[16:12:19] <Not-616a>	 [02miraheze/ssl] 07Reception123 032adce7f - add wiki.3805.co.uk cert
[16:12:21] <Not-616a>	 [02miraheze/ssl] 07Reception123 03dbb22fb - add wiki.parappa.dev cert
[16:20:04] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.25, 6.88, 5.08
[16:21:49] <icinga-miraheze>	 PROBLEM - wiki.3805.co.uk - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ultra104.uk2.net.', 'ultra103.uk2.net.'], 'CNAME': '3805.miraheze.org.'}
[16:22:04] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.04, 6.41, 5.12
[16:24:00] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:26:06] <icinga-miraheze>	 PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:27:12] <icinga-miraheze>	 PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:30:47] <icinga-miraheze>	 PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:31:04] <icinga-miraheze>	 PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:39:57] <icinga-miraheze>	 PROBLEM - cp14 Puppet on cp14 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:40:36] <icinga-miraheze>	 PROBLEM - cp15 Puppet on cp15 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:42:40] <icinga-miraheze>	 PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:43:54] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:45:24] <icinga-miraheze>	 PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:45:24] <icinga-miraheze>	 PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:52:03] <icinga-miraheze>	 PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:52:39] <icinga-miraheze>	 PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[grayravens.com_private],File[wiki.3805.co.uk_private],File[wiki.parappa.dev_private]
[16:55:02] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.03, 3.92, 3.50
[16:58:52] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.87, 3.57
[17:00:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.30, 3.32, 3.40
[17:04:31] <icinga-miraheze>	 PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.42, 1.89, 1.14
[17:04:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.23, 4.18, 3.73
[17:06:30] <icinga-miraheze>	 RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 1.10, 1.66, 1.15
[17:06:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.73, 3.63
[17:14:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.08, 3.35
[17:52:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.48, 3.24
[17:56:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 2.95, 3.08
[18:15:11] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.40, 3.21
[18:17:06] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.18, 3.15
[18:22:52] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.63, 3.83, 3.42
[18:24:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.87, 3.48
[18:28:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.86, 3.00, 3.23
[18:45:44] <Not-616a>	 [02miraheze/mediawiki] 07Universal-Omega pushed 0310 commits to 03REL1_37 [+0/-0/±19] 13https://git.io/JXMDz
[18:45:45] <Not-616a>	 [02miraheze/mediawiki] 07cicalese 03eb5d7c4 - Allow populateContentTables to continue when there are bad blobs
[18:45:47] <Not-616a>	 [02miraheze/mediawiki] 07reedy 0329726bc - Update git submodules
[18:45:48] <Not-616a>	 [02miraheze/mediawiki] 07reedy 035d46925 - Update git submodules
[18:45:50] <Not-616a>	 [02miraheze/mediawiki] ... and 7 more commits.
[19:04:55] <CosmicAlpha>	 Reception123: puppet failures on mw*, mwtask1, test3, and cp* for certs.
[19:07:18] <icinga-miraheze>	 PROBLEM - celeste.ink - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'celeste.ink' expires in 7 day(s) (Thu 18 Nov 2021 19:02:04 GMT +0000).
[19:14:02] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.49, 3.15
[19:15:33] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'world': True} to skip
[19:15:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:17:52] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.68, 2.84, 2.98
[19:19:52] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'world': True} to skip - FAIL: [256, 0, 0, 0] in 259s
[19:19:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:21:43] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.48, 3.22
[19:23:38] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.33, 3.20
[19:48:58] <Reception123>	 CosmicAlpha: should be fixed now
[19:49:32] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'world': True, 'force': True} to skip
[19:49:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:50:39] <icinga-miraheze>	 RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:02] <icinga-miraheze>	 RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:08] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'world': True, 'force': True} to skip - FAIL: [256, 0, 0, 0] in 155s
[19:52:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:52:21] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[19:53:27] <icinga-miraheze>	 RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:53:30] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 3.80, 3.23
[19:55:07] <icinga-miraheze>	 RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[19:57:20] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.52, 3.26
[19:59:15] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.65, 3.33
[19:59:50] <icinga-miraheze>	 RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[20:00:18] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'world': True, 'force': True} to skip
[20:00:21] <icinga-miraheze>	 RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:00:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:01:07] <CosmicAlpha>	 Thanks Reception123!
[20:01:10] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.26, 3.22
[20:02:27] <Not-616a>	 [02mw-config] 07Universal-Omega closed pull request 03#4215: Revert "Revert "T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki"" - 13https://git.io/JXPqw
[20:02:29] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXDJg
[20:02:30] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega 038d5141e - Revert "Revert "T8270: set `$wgCargoPageDataColumns` for dmlwikiwiki (#4213)" (#4214)" (#4215)
[20:02:32] <Not-616a>	 [02mw-config] 07Universal-Omega deleted branch 03revert-4214-revert-4213-Universal-Omega-patch-1 - 13https://git.io/vbvb3
[20:02:33] <Not-616a>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4214-revert-4213-Universal-Omega-patch-1
[20:02:45] <MirahezeLSBot>	 !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all
[20:02:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:02:49] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'world': True, 'force': True} to skip - FAIL: [256, 0, 0, 0] in 151s
[20:02:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:03:04] <MirahezeLSBot>	 !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 19s
[20:03:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:03:46] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:07:38] <MirahezeLSBot>	 !log [universalomega@test3] starting deploy of {'world': True, 'force': True} to skip
[20:07:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:07:55] <icinga-miraheze>	 RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[20:10:08] <icinga-miraheze>	 RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:10:09] <MirahezeLSBot>	 !log [universalomega@test3] finished deploy of {'world': True, 'force': True} to skip - SUCCESS in 151s
[20:10:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:10:50] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.57, 3.35
[20:12:09] <icinga-miraheze>	 RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:12:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 3.07, 3.20
[20:13:03] <icinga-miraheze>	 RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:13:16] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:14:25] <MirahezeLSBot>	 !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/Cargo/maintenance/setCargoPageData.php --wiki=dmlwikiwiki (END - exit=0)
[20:14:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:23:57] <MirahezeLSBot>	 !log [@test3] starting deploy of {'config': True} to skip
[20:23:58] <MirahezeLSBot>	 !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
[20:24:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:24:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:50:23] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.39, 3.18
[20:58:03] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 3.31, 3.29
[21:18:19] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.32, 5.98, 4.78
[21:20:19] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.42, 6.18, 5.00
[21:29:43] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.28, 3.07
[21:31:38] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.24, 3.02, 3.01
[21:31:41] <MH-Discord>	 <jmk> Hi guys
[21:31:43] <MH-Discord>	 <jmk> weird thing happene
[21:31:46] <MH-Discord>	 <jmk> happened**
[21:31:55] <MH-Discord>	 <jmk> We have an open Phabricator request (T8264) for a custom domain
[21:32:10] <MH-Discord>	 <jmk> Earlier today we noticed that the custom domain we had chosen was now returning an "invalid wiki" notice
[21:32:16] <MH-Discord>	 <jmk> https://wiki.parappa.dev/wiki/
[21:32:17] <MirahezeBot>	 [url] Wiki not Found | wiki.parappa.dev
[21:32:22] <MH-Discord>	 <jmk> Yep ^^
[21:32:39] <MH-Discord>	 <jmk> I assume that it's just being weird because it isn't set up yet, but I thought I'd check in
[21:36:15] <RhinosF1>	 If you're
[21:36:29] <RhinosF1>	 If your custom domain isn't set up yet, it won't work
[21:36:33] <RhinosF1>	 .task 8264
[21:36:36] <MirahezeBot>	 https://phabricator.miraheze.org/T8264 - custom domain request, authored by Pips, assigned to None, Priority: Normal, Status: Open
[21:36:38] <RhinosF1>	 Ta
[21:36:47] <RhinosF1>	 Task is still open
[21:38:26] <RhinosF1>	 Sario: pester me to do a new prod release
[21:42:52] <MH-Discord>	 <jmk> Yep
[21:43:57] <MH-Discord>	 <Joseph> It means the setup hasn't finished
[21:45:26] <MH-Discord>	 <jmk> Ah alright
[21:45:31] <MH-Discord>	 <jmk> thanks for letting me know
[21:46:30] <MH-Discord>	 <Joseph Bukkit> I think they need to change a setting on Special:ManageWiki?
[21:46:59] <RhinosF1>	 No
[21:47:07] <RhinosF1>	 Sysadmins do
[21:47:31] <MH-Discord>	 <Joseph Bukkit> I meant sysadmins, should have been clearer bout that
[21:47:31] <RhinosF1>	 It looks like Reception123 stopped half way
[21:51:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.82, 3.19
[21:57:34] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.89, 3.47
[21:59:29] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 3.40, 3.34
[22:05:21] <Sario>	 RhinosF1: a k
[22:05:22] <Sario>	 ack*
[22:48:19] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.14, 7.54, 5.49
[22:50:19] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.10, 6.66, 5.42
[22:52:23] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.66, 21.20, 18.07
[22:54:23] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.26, 19.13, 17.69
[23:00:01] <MirahezeLSBot>	 !log [@test3] starting deploy of {'l10nupdate': True} to skip
[23:00:05] <MirahezeLSBot>	 !log [@mw11] starting deploy of {'l10nupdate': True} to all
[23:00:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:00:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:02:23] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.19, 23.59, 19.56
[23:02:39] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.65, 4.99, 3.79
[23:04:36] <icinga-miraheze>	 RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.77, 4.94, 3.92
[23:05:12] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.81, 7.67, 5.29
[23:07:12] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.41, 6.31, 5.08
[23:08:23] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.37, 23.58, 21.57
[23:12:23] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.93, 19.47, 20.38
[23:20:12] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.23, 7.50, 5.79
[23:21:14] <Agent>	 staratlaswiki.org's SSL certificate seems to have expired today
[23:21:19] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.65, 21.30, 20.71
[23:21:33] <icinga-miraheze>	 PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:22:59] <icinga-miraheze>	 PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:23:30] <icinga-miraheze>	 RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[23:24:11] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.02, 7.57, 6.35
[23:24:19] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.76, 7.36, 5.71
[23:24:56] <icinga-miraheze>	 RECOVERY - mw10 APT on mw10 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[23:26:11] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.68, 6.26, 6.02
[23:26:18] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.87, 6.36, 5.54
[23:29:32] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.60, 6.81, 6.18
[23:29:52] <MirahezeLSBot>	 !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1790s
[23:29:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:31:11] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.07, 19.95, 20.38
[23:31:27] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.53, 5.59, 5.80
[23:39:03] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.08, 24.00, 21.37
[23:41:49] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.32, 3.02
[23:41:59] <MirahezeLSBot>	 !log [@mw11] finished deploy of {'l10nupdate': True} to all - SUCCESS in 2513s
[23:42:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:43:00] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.68, 22.17, 21.37
[23:45:39] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.32, 3.09
[23:46:57] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.87, 17.77, 19.76
[23:49:31] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.44, 3.19
[23:53:21] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.31, 3.22