[00:02:29] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/etc/nginx/fastcgi_params],File[/usr/local/bin/nginx-prometheus-exporter],File[/etc/nginx/sites-available/graylog-proxy],File[/etc/sysctl.d] [00:14:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.93, 20.86, 17.60 [00:17:00] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.44, 22.52, 18.60 [00:18:59] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.44, 5.25, 3.97 [00:21:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.89, 4.58, 3.87 [00:21:08] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.13, 22.33, 19.63 [00:23:01] PROBLEM - test4 Puppet on test4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [00:23:12] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.24, 19.91, 19.09 [00:31:11] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:36:02] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.95, 21.31, 19.73 [00:40:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.90, 20.75, 19.93 [00:42:06] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.46, 17.77, 18.97 [00:51:34] RECOVERY - test4 Puppet on test4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:56:14] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.03, 6.02, 4.51 [01:00:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.52, 3.32 [01:00:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.61, 4.75, 4.38 [01:02:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.58, 6.46, 5.35 [01:02:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.72, 3.97, 3.51 [01:04:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.66, 3.45 [01:06:23] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JrqZa [01:06:25] [02miraheze/puppet] 07The-Voidwalker 033db80c6 - update discord link [01:06:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.93, 8.89, 6.50 [01:06:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.95, 3.40, 3.38 [01:08:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.57, 6.70, 5.99 [01:36:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.45, 3.26 [01:38:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.11, 3.09, 3.16 [02:23:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.55, 3.32 [02:31:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.31, 3.34 [02:36:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.62, 7.55, 5.42 [02:38:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.52, 6.87, 5.43 [02:38:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.10, 6.08, 5.15 [02:39:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.41, 3.35 [02:39:58] PROBLEM - wikibeats.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:40:28] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.25, 5.83, 5.21 [02:41:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.61, 3.16, 3.27 [03:08:46] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:41] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.landev.vn' expires in 13 day(s) (Thu 21 Oct 2021 04:18:15 GMT +0000). [03:50:16] PROBLEM - mabi.myomi.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'mabi.myomi.net' expires in 7 day(s) (Sat 16 Oct 2021 03:48:34 GMT +0000). [03:50:18] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.consid.vn' expires in 7 day(s) (Sat 16 Oct 2021 03:49:06 GMT +0000). [03:51:59] PROBLEM - wiki.jumpbound.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jumpbound.com' expires in 15 day(s) (Sun 24 Oct 2021 03:49:03 GMT +0000). [03:55:43] PROBLEM - thepolicyhub.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'thepolicyhub.org.uk' expires in 7 day(s) (Sat 16 Oct 2021 03:50:30 GMT +0000). [04:07:47] RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bikinibottomdiscord.org' will expire on Sat 01 Jan 2022 16:29:36 GMT +0000. [04:08:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.06, 2.79 [04:10:16] PROBLEM - wikibeats.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns-cloud-e3.googledomains.com.', 'ns-cloud-e2.googledomains.com.', 'ns-cloud-e4.googledomains.com.', 'ns-cloud-e1.googledomains.com.'], 'CNAME': None} [04:11:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.31, 3.33, 2.98 [04:16:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.53, 3.17 [04:17:16] PROBLEM - wikibeats.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikibeats.net reverse DNS resolves to any-in-2015.1e100.net [04:18:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.19, 3.09 [04:24:16] PROBLEM - biblestrength.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'biblestrength.net' expires in 15 day(s) (Sun 24 Oct 2021 04:16:04 GMT +0000). [04:31:12] RECOVERY - wikibeats.net - reverse DNS on sslhost is OK: SSL OK - wikibeats.net reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [04:40:11] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns2.registrar-servers.com.', 'dns1.registrar-servers.com.', 'dns101.registrar-servers.com.', 'dns102.registrar-servers.com.'], 'CNAME': None} [04:40:17] PROBLEM - wikibeats.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikibeats.net reverse DNS resolves to any-in-2015.1e100.net [04:47:10] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.bikinibottomdiscord.org reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com [05:01:05] RECOVERY - wikibeats.net - reverse DNS on sslhost is OK: SSL OK - wikibeats.net reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [05:08:06] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns102.registrar-servers.com.', 'dns2.registrar-servers.com.', 'dns1.registrar-servers.com.', 'dns101.registrar-servers.com.'], 'CNAME': None} [05:09:59] PROBLEM - wikibeats.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikibeats.net reverse DNS resolves to any-in-2215.1e100.net [05:15:04] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.bikinibottomdiscord.org reverse DNS resolves to a51062ecadbb5a26e.awsglobalaccelerator.com [05:16:56] PROBLEM - wikibeats.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns-cloud-e2.googledomains.com.', 'ns-cloud-e3.googledomains.com.', 'ns-cloud-e4.googledomains.com.', 'ns-cloud-e1.googledomains.com.'], 'CNAME': None} [05:23:51] PROBLEM - wikibeats.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikibeats.net reverse DNS resolves to any-in-2215.1e100.net [05:52:20] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:52:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.05, 6.40, 4.59 [05:54:21] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:54:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.66, 5.94, 4.64 [06:01:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.62, 8.87, 5.89 [06:01:57] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:02:15] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.63, 21.99, 17.50 [06:03:12] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.03, 7.20, 5.67 [06:04:02] RECOVERY - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is OK: SSL OK - wiki.bikinibottomdiscord.org reverse DNS resolves to cp14.miraheze.org - CNAME FLAT [06:04:05] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:08:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.05, 8.80, 6.51 [06:08:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.07, 23.98, 20.28 [06:11:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.49, 7.36, 7.10 [06:12:29] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.30, 8.52, 5.93 [06:12:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.51, 7.46, 6.47 [06:13:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.80, 6.40, 6.77 [06:14:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.86, 7.39, 7.54 [06:15:13] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.06, 19.10, 19.42 [06:16:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.86, 7.65, 6.16 [06:16:49] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.25, 5.99, 6.13 [06:18:08] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.73, 5.37, 6.67 [06:20:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.74, 5.98, 5.88 [06:41:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.47, 19.76, 18.64 [06:42:22] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.64, 4.70, 3.59 [06:43:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.51, 18.08, 18.17 [06:44:19] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.27, 4.45, 3.63 [06:50:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.49, 5.22, 4.14 [06:52:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.00, 4.32, 3.94 [07:07:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 7.19, 5.28 [07:08:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.21, 5.99, 5.07 [07:37:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.72, 6.60, 4.95 [07:39:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.64, 5.56, 4.78 [08:07:58] [02mediawiki] 07RhinosF1 labeled pull request 03#3229: Bump extensions/UniversalLanguageSelector from `7374540` to `b5ec74d` - 13https://git.io/JaGCR [08:13:10] [02mediawiki] 07RhinosF1 labeled pull request 03#3223: Bump extensions/PageSchemas from `5e10d6a` to `a698ba2` - 13https://git.io/Jg7JH [08:13:53] [02mediawiki] 07RhinosF1 labeled pull request 03#3222: Bump extensions/Cargo from `19b7b42` to `14c497b` - 13https://git.io/Jg7Jw [08:14:20] [02mediawiki] 07RhinosF1 labeled pull request 03#3219: Bump skins/liberty from `4d614e9` to `3687268` - 13https://git.io/Jg7f5 [08:14:38] [02mediawiki] 07RhinosF1 labeled pull request 03#3218: Bump extensions/GlobalNewFiles from `5a0fd36` to `6c75f13` - 13https://git.io/Jg7fZ [08:15:23] [02mediawiki] 07RhinosF1 labeled pull request 03#3216: Bump extensions/WikiForum from `ab21d72` to `2ccdb80` - 13https://git.io/Jg7v9 [08:15:49] [02mediawiki] 07RhinosF1 labeled pull request 03#3215: Bump extensions/DiscordNotifications from `b19d11b` to `faacede` - 13https://git.io/Jg7vg [08:16:26] [02mediawiki] 07RhinosF1 assigned pull request 03#3213: Bump extensions/MirahezeMagic from `1fccf46` to `37f09ac` - 13https://git.io/Jg7vU [08:16:30] [02mediawiki] 07RhinosF1 labeled pull request 03#3213: Bump extensions/MirahezeMagic from `1fccf46` to `37f09ac` - 13https://git.io/Jg7vU [08:17:04] [02mediawiki] 07RhinosF1 labeled pull request 03#3210: Bump extensions/ApprovedRevs from `aaddd36` to `2503370` - 13https://git.io/Jg7eL [08:17:24] [02mediawiki] 07RhinosF1 labeled pull request 03#3209: Bump skins/foreground from `d8fe013` to `471eaff` - 13https://git.io/Jg7eU [08:17:49] [02mediawiki] 07RhinosF1 labeled pull request 03#3208: Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` - 13https://git.io/JgQjH [08:18:21] [02mediawiki] 07RhinosF1 labeled pull request 03#3207: Bump skins/Medik from `4cd707d` to `c52dc4d` - 13https://git.io/JgQjK [08:18:46] [02mediawiki] 07RhinosF1 labeled pull request 03#3205: Bump extensions/PageForms from `f3dabf1` to `a7f59e9` - 13https://git.io/JgQjO [08:20:09] [02mediawiki] 07RhinosF1 labeled pull request 03#3058: Bump extensions/MintyDocs from `77fdf03` to `c787fc0` - 13https://git.io/JzOJh [08:20:10] [02mediawiki] 07RhinosF1 labeled pull request 03#3058: Bump extensions/MintyDocs from `77fdf03` to `c787fc0` - 13https://git.io/JzOJh [08:20:50] [02mediawiki] 07RhinosF1 labeled pull request 03#2733: Bump skins/Cosmos from `ae4f902` to `b19f881` - 13https://git.io/JumMd [08:21:02] [02mediawiki] 07RhinosF1 labeled pull request 03#2733: Bump skins/Cosmos from `ae4f902` to `b19f881` - 13https://git.io/JumMd [08:21:04] [02mediawiki] 07RhinosF1 unlabeled pull request 03#2733: Bump skins/Cosmos from `ae4f902` to `b19f881` - 13https://git.io/JumMd [08:21:05] [02mediawiki] 07RhinosF1 labeled pull request 03#2733: Bump skins/Cosmos from `ae4f902` to `b19f881` - 13https://git.io/JumMd [08:21:31] [02mediawiki] 07RhinosF1 closed pull request 03#2744: Bump extensions/RottenLinks from `09b83d7` to `535ab5f` - 13https://git.io/JumDn [08:21:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#2744: Bump extensions/RottenLinks from `09b83d7` to `535ab5f` - 13https://git.io/JrWwh [08:21:35] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/RottenLinks-535ab5f [08:21:37] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/RottenLinks-535ab5f - 13https://git.io/vbL5b [08:21:45] [02mediawiki] 07RhinosF1 closed pull request 03#2747: Bump extensions/GlobalNewFiles from `8edd36d` to `87496a2` - 13https://git.io/JumD4 [08:21:46] [02mediawiki] 07dependabot[bot] commented on pull request 03#2747: Bump extensions/GlobalNewFiles from `8edd36d` to `87496a2` - 13https://git.io/JrWrC [08:21:48] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/GlobalNewFiles-87496a2 [08:21:49] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/GlobalNewFiles-87496a2 - 13https://git.io/vbL5b [08:22:13] [02mediawiki] 07RhinosF1 closed pull request 03#2749: Bump extensions/OATHAuth from `fbba3db` to `0993878` - 13https://git.io/JumDu [08:22:14] [02mediawiki] 07dependabot[bot] commented on pull request 03#2749: Bump extensions/OATHAuth from `fbba3db` to `0993878` - 13https://git.io/JrWof [08:22:16] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/OATHAuth-0993878 [08:22:17] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/OATHAuth-0993878 - 13https://git.io/vbL5b [08:23:03] [02mediawiki] 07RhinosF1 labeled pull request 03#2751: Bump extensions/WikiTextLoggedInOut from `aa453a4` to `60155e6` - 13https://git.io/JumDa [08:23:07] [02mediawiki] 07RhinosF1 closed pull request 03#2751: Bump extensions/WikiTextLoggedInOut from `aa453a4` to `60155e6` - 13https://git.io/JumDa [08:23:08] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JrWKW [08:23:10] [02miraheze/mediawiki] 07dependabot[bot] 036e4faa9 - Bump extensions/WikiTextLoggedInOut from `aa453a4` to `60155e6` (#2751) [08:23:12] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/WikiTextLoggedInOut-60155e6 - 13https://git.io/vbL5b [08:23:13] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/WikiTextLoggedInOut-60155e6 [08:23:28] [02mediawiki] 07RhinosF1 labeled pull request 03#2752: Bump extensions/Josa from `87e0ea9` to `56ea7ac` - 13https://git.io/JumDw [08:23:48] [02mediawiki] 07RhinosF1 closed pull request 03#2752: Bump extensions/Josa from `87e0ea9` to `56ea7ac` - 13https://git.io/JumDw [08:23:50] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/Josa-56ea7ac [08:23:51] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/Josa-56ea7ac - 13https://git.io/vbL5b [08:23:53] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JrW64 [08:23:54] [02miraheze/mediawiki] 07dependabot[bot] 034e5b202 - Bump extensions/Josa from `87e0ea9` to `56ea7ac` (#2752) [08:24:20] [02mediawiki] 07RhinosF1 labeled pull request 03#2753: Bump extensions/Nuke from `36ff776` to `2dd2dab` - 13https://git.io/JumDo [08:24:24] [02mediawiki] 07RhinosF1 closed pull request 03#2753: Bump extensions/Nuke from `36ff776` to `2dd2dab` - 13https://git.io/JumDo [08:24:25] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+0/-0/±0] 13https://git.io/JrWit [08:24:27] [02miraheze/mediawiki] 07dependabot[bot] 0396de9f6 - Bump extensions/Nuke from `36ff776` to `2dd2dab` (#2753) [08:24:28] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/Nuke-2dd2dab [08:24:30] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/Nuke-2dd2dab - 13https://git.io/vbL5b [08:24:43] [02mediawiki] 07RhinosF1 closed pull request 03#2754: Bump extensions/ProofreadPage from `ad83585` to `8140595` - 13https://git.io/JumDP [08:24:44] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/ProofreadPage-8140595 - 13https://git.io/vbL5b [08:24:46] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JrWii [08:24:47] [02miraheze/mediawiki] 07dependabot[bot] 0307638f9 - Bump extensions/ProofreadPage from `ad83585` to `8140595` (#2754) [08:24:49] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/ProofreadPage-8140595 [08:24:50] [02mediawiki] 07RhinosF1 labeled pull request 03#2754: Bump extensions/ProofreadPage from `ad83585` to `8140595` - 13https://git.io/JumDP [08:25:04] [02mediawiki] 07RhinosF1 labeled pull request 03#2755: Bump extensions/MintyDocs from `77fdf03` to `72e7363` - 13https://git.io/JumD1 [08:25:26] [02mediawiki] 07RhinosF1 labeled pull request 03#2756: Bump extensions/MassMessage from `d9d1612` to `620e030` - 13https://git.io/JumDD [08:25:46] [02mediawiki] 07RhinosF1 closed pull request 03#2756: Bump extensions/MassMessage from `d9d1612` to `620e030` - 13https://git.io/JumDD [08:25:47] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/MassMessage-620e030 [08:25:49] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_36/extensions/MassMessage-620e030 - 13https://git.io/vbL5b [08:25:51] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JrWX3 [08:25:52] [02miraheze/mediawiki] 07dependabot[bot] 032eddecf - Bump extensions/MassMessage from `d9d1612` to `620e030` (#2756) [08:42:49] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [08:51:33] Go away icinga-miraheze [08:51:56] * RhinosF1 does wonder if the train style would work better anyway I'm getting rid of that [08:52:50] * RhinosF1 just doesn't like the multiversion thing because of disk soace [08:52:53] Space* [08:52:55] Reception123: ^ [08:58:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.92, 5.01, 4.21 [08:59:14] yeah, disk space is a concern [08:59:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.45, 18.09, 16.39 [09:00:57] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.68, 4.36, 4.06 [09:01:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.77, 16.57, 16.05 [09:32:20] Reception123: I can't decide whether an mw-deploy repo is too many steps and whether if we had a submodule of the mediawiki repo in there it would give us any benefit [09:32:54] it does seem a bit overcomplicating [09:33:11] It may save disk because we can pull less commits and stop puppet failures but it's also an extra step [09:33:14] Reception123: ^ [09:33:38] It also moves us closer to multiversion & trains but we don't have the disk for that [09:34:08] yeah.. [09:37:14] Reception123: it would get rid of the nightmares of git when we merge lots at once [09:37:23] Like 1.36.1 I had to reclone [09:37:35] But I don't know whether another step is worth it [09:37:59] And all the work to shift around where we serve from [10:21:32] yeah that is an advantage [11:07:42] The AI that automatically creates wikis is showing off an error like "Exception experienced creating the wiki. Error is: Redis server error: socket error on read socket". [11:09:40] The error message in question. [11:09:40] https://cdn.discordapp.com/attachments/808001911868489748/895991327249014824/bandicam_2021-10-08_07-08-44-657.jpg [11:14:17] Reception123: ^ [11:14:23] I guess redis is broken again [11:15:27] RhinosF1: I've just seen that on an import I was running [11:20:40] Reception123: must be an OOM [11:21:18] Doesn't look like it from grafana [11:21:25] I believe it's ran on jobchron1 [11:21:39] I unfortunately can't look until later today. maybe JohnLewis or paladox can earlier [11:22:01] Ack [11:27:56] Reception123: are you running it with —no-updates? [11:28:49] JohnLewis: No, I started the import last night and this morning I noticed it timed out because of that error but I didn't have time to investigate then [11:29:01] That’s why [11:29:19] Oh, was I supposed to run it with --no-updates? I've never done that before [11:29:32] I’ve told you to do it that way before [11:29:52] Oh, I don't remember. What exactly does it do? [11:30:00] I'll know to do that next time and will update the docs later to reflect it [11:30:01] Because otherwise Redis errors will occur not only for the update, but general traffic [11:30:34] Oh so the import itself is responsible? [11:30:35] It doesn’t create jobs to refresh links, etc so you’ll need to rebuildall after [11:31:19] ah so if I run an import with no updates then right after I need to run rebuildAll each time? [11:39:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 21.33, 11.78, 6.86 [11:43:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.11, 7.75, 6.46 [11:44:57] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.24, 26.44, 20.65 [11:45:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.88, 7.78, 5.13 [11:45:57] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.23, 6.65, 6.21 [11:47:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.05, 6.85, 5.11 [11:49:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.27, 23.27, 20.64 [11:49:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.77, 5.92, 4.97 [11:55:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.78, 19.31, 19.75 [12:01:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.54, 20.72, 20.08 [12:05:45] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.91, 6.41, 5.08 [12:06:16] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.33, 7.45, 5.39 [12:07:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.97, 5.85, 5.04 [12:08:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.55, 6.21, 5.19 [12:15:41] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.56, 18.70, 19.80 [12:18:30] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.38, 5.13, 4.27 [12:20:30] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.07, 4.24, 4.04 [12:38:02] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 6.10, 4.84 [12:39:59] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.72, 5.76, 4.86 [12:45:17] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.93, 5.60, 4.80 [12:47:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.07, 5.28, 4.77 [12:48:22] Why is it failing though [12:48:34] surely if redis is having issues... memcache would [12:49:20] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.75, 4.79, 4.65 [12:51:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.10, 2.65 [12:53:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.05, 2.68 [12:56:20] paladox: it only happens for jobs, which memcached doesn't handle [12:58:57] I think i may have an idea. [13:04:11] JobQueueConnectionError from line 764 of /srv/mediawiki/w/includes/jobqueue/JobQueueRedis.php: Unable to connect to redis server 51.195.236.215:6379. :wow: [13:05:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.20, 19.87, 19.10 [13:07:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.26, 19.97, 19.27 [13:25:11] Hmm, redis doesn't log to graylog after it sends "Configuration loaded" and "Running mode=standalone, port=6379." [13:28:40] https://www.irccloud.com/pastebin/cmEfCGlm/ [13:29:29] Logs do go to graylog, we've been over this in the past when you said it didn't [13:31:48] PROBLEM - jobchron1 Redis Process on jobchron1 is CRITICAL: PROCS CRITICAL: 3 processes with args 'redis-server' [13:33:19] huh [13:33:44] RECOVERY - jobchron1 Redis Process on jobchron1 is OK: PROCS OK: 1 process with args 'redis-server' [13:36:11] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:36:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.21, 8.18, 5.39 [13:38:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 6.14, 4.80 [13:38:12] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:38:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.47, 6.70, 5.20 [13:40:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.88, 5.34, 4.67 [13:43:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.67, 5.30, 4.45 [13:45:23] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.75, 5.52, 4.62 [13:49:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.57, 20.65, 19.41 [13:51:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jr2a2 [13:51:05] [02miraheze/puppet] 07paladox 03d4264a3 - redis: Unset tcp-keepalive [13:51:07] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [13:51:08] [02puppet] 07paladox opened pull request 03#2007: redis: Unset tcp-keepalive - 13https://git.io/Jr2aD [13:51:10] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.17, 18.29, 18.70 [13:51:21] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.59, 5.31, 4.99 [13:51:36] [02puppet] 07paladox edited pull request 03#2007: redis: Unset tcp-keepalive - 13https://git.io/Jr2aD [13:51:38] [02puppet] 07paladox closed pull request 03#2007: redis: Unset tcp-keepalive - 13https://git.io/Jr2aD [13:51:40] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jr2VM [13:51:41] [02miraheze/puppet] 07paladox 03d4264a3 - redis: Unset tcp-keepalive [13:51:43] [02miraheze/puppet] 07paladox 0366ae32c - Merge pull request #2007 from miraheze/paladox-patch-1 [13:51:44] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [13:51:46] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [13:54:14] [02puppet] 07paladox commented on pull request 03#1994: mwdeploy: add --inplace when syncing without timestamp - 13https://git.io/Jr2iB [13:55:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.71, 4.68, 4.82 [13:56:22] [02puppet] 07paladox closed pull request 03#1991: csp: add blob to script-src - 13https://git.io/Jz0bf [13:56:23] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Jr2DI [13:56:25] [02miraheze/puppet] 07R4356th 03b54c8cc - csp: add blob to script-src [13:56:26] [02miraheze/puppet] 07paladox 039d44270 - Merge pull request #1991 from R4356th/patch-3 [13:59:34] [02puppet] 07paladox commented on pull request 03#1983: csp: add *.wikipedia.org to connect-src - 13https://git.io/Jr27P [14:08:08] Reception123: where did you run the import [14:09:10] [02puppet] 07RhinosF1 commented on pull request 03#1994: mwdeploy: add --inplace when syncing without timestamp - 13https://git.io/JraqO [14:09:42] [02puppet] 07paladox closed pull request 03#1994: mwdeploy: add --inplace when syncing without timestamp - 13https://git.io/Jz6vf [14:09:43] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JramB [14:09:45] [02miraheze/puppet] 07RhinosF1 03d5a7850 - mwdeploy: add --inplace when syncing without timestamp [14:09:46] [02miraheze/puppet] 07paladox 03bec6907 - Merge pull request #1994 from RhinosF1/patch-93 [14:10:02] Ty paladox [14:10:13] [02puppet] 07paladox commented on pull request 03#1992: Fix Flake8 violations - 13https://git.io/JraY0 [14:10:40] [02puppet] 07paladox commented on pull request 03#1876: mwscript: add some logging for 'long' scripts - 13https://git.io/JraOz [14:11:11] paladox: 1992 can be merged [14:11:20] I'll rebase 1876 later [14:11:28] ok [14:11:58] [02puppet] 07paladox closed pull request 03#1992: Fix Flake8 violations - 13https://git.io/JzEwl [14:11:59] [02miraheze/puppet] 07paladox pushed 038 commits to 03master [+0/-0/±12] 13https://git.io/JraGU [14:12:01] [02miraheze/puppet] 07Universal-Omega 034eae8f3 - Fix Flake8 violations [14:12:02] [02miraheze/puppet] 07Universal-Omega 034103f53 - Update adminlogbot.py [14:12:04] [02miraheze/puppet] 07Universal-Omega 03e9c0cfa - Update adminlog.py [14:12:05] [02miraheze/puppet] ... and 5 more commits. [14:12:45] paladox: mwtask1 is where I ran it [14:12:56] ah, i guess not in a screen? [14:13:00] as i see none running [14:17:00] I just got back to my PC so I can restart it now but yes it previously failed [14:17:20] Yeh, can you restart it but do it how you did it before please Reception123 [14:17:25] i want to see if my change worked [14:17:29] reception@mwtask1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=undertaledeltaruneauocwiki /home/reception/undertaleaufanon_pages_full.xml --username-prefix wikia:735q4e87 --report 1 [14:17:41] it didn't log :) [14:17:57] paladox: it did when it first failed when I used mwscript [14:18:04] oh [14:20:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.14, 7.51, 5.65 [14:21:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.99, 3.74, 3.08 [14:21:47] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.44, 4.94, 4.60 [14:22:13] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.90, 6.69, 5.58 [14:23:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.40, 3.04 [14:27:43] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.23, 5.56, 4.96 [14:28:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jra5T [14:28:47] [02miraheze/puppet] 07paladox 03bd00fc1 - graylog: Upgrade elasticsearch, graylog and mongodb [14:29:42] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.12, 5.07, 4.85 [14:33:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.19, 24.52, 21.21 [14:35:35] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.94, 6.19, 4.79 [14:36:13] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.18, 5.26, 5.06 [14:37:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.24, 8.74, 5.66 [14:37:48] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.18, 9.21, 6.13 [14:39:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.20, 6.89, 5.35 [14:39:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.25, 7.68, 5.95 [14:40:20] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.70, 4.82, 4.91 [14:41:28] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.45, 6.06, 5.22 [14:41:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.76, 6.72, 5.81 [14:43:36] paladox: should I restart it then? [14:44:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.13, 23.13, 22.62 [14:44:14] Reception123: hold on, i'm going to see about increasing redis tcp connections [14:44:18] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.30, 5.16, 5.03 [14:45:38] [02puppet] 07paladox closed pull request 03#2008: redis: Increase tcp-backlog to 1024 - 13https://git.io/JrVB0 [14:45:39] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JrVBE [14:45:41] [02miraheze/puppet] 07paladox 03b9170a6 - redis: Increase tcp-backlog to 1024 [14:45:42] [02miraheze/puppet] 07paladox 033e4f6c9 - Merge pull request #2008 from miraheze/paladox-patch-1 [14:45:44] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [14:46:20] ok [14:46:22] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.33, 5.68, 5.23 [14:46:27] Reception123: now try pls [14:48:03] ok [14:48:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.08, 23.26, 22.73 [14:50:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [14:50:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JrVo4 [14:51:00] [02miraheze/puppet] 07paladox 03b9170a6 - redis: Increase tcp-backlog to 1024 [14:51:15] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [14:51:17] [02puppet] 07paladox opened pull request 03#2008: redis: Increase tcp-backlog to 1024 - 13https://git.io/JrVB0 [14:53:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.58, 3.15 [14:54:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.02, 23.39, 23.19 [14:55:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 3.03, 3.00 [14:58:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.04, 5.61, 5.66 [15:01:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.68, 3.29 [15:03:01] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.02, 5.72, 5.68 [15:05:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.83, 23.67, 23.10 [15:06:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 13.40, 9.25, 6.68 [15:07:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.54, 23.82, 23.28 [15:08:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.27, 3.27 [15:08:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.78, 7.41, 6.34 [15:08:56] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.38, 7.28, 6.50 [15:09:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.50, 5.90, 5.81 [15:11:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.96, 6.65, 6.36 [15:12:48] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.03, 6.28, 6.16 [15:15:51] Reception123: it failed, you can stop and do it the way John said [15:16:07] paladox: it's still going for me [15:16:20] ok [15:20:15] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.23, 22.44, 22.41 [15:21:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.94, 4.27, 5.01 [15:22:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.12, 21.33, 21.99 [15:23:12] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.12, 7.04, 6.45 [15:25:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.38, 6.35, 6.27 [15:29:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.84, 7.50, 6.79 [15:31:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.01, 6.79, 6.61 [15:32:25] [02puppet] 07RhinosF1 opened pull request 03#2009: mwdeploy: pull as part of the script - 13https://git.io/JrrUB [15:33:19] [02puppet] 07RhinosF1 synchronize pull request 03#2009: mwdeploy: pull as part of the script - 13https://git.io/JrrUB [15:33:28] [02puppet] 07paladox reviewed pull request 03#2009 commit - 13https://git.io/JrrkQ [15:34:05] paladox: --depth=5 is what puppet does [15:34:52] --allow-unrelated-histories is to avoid it failing if there's more than 5 to pull [15:34:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.91, 22.42, 21.68 [15:34:56] Yes and that's done so a timeout isn't hit [15:35:34] ideally with mwdeploy we don't need to do deth simply because everytime there's a large merge (e.g. merging several changes that go past >5 it messes things up) [15:35:56] which requires you to run a manual git reset and then to manually do git pull/git submodule update [15:36:17] paladox: so you think we should just take the full repo? [15:36:23] yeh [15:36:47] paladox: what about disk space? [15:37:34] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.66, 7.40, 6.58 [15:37:37] [02puppet] 07RhinosF1 synchronize pull request 03#2009: mwdeploy: pull as part of the script - 13https://git.io/JrrUB [15:37:42] i don't think we copy .git right? In which case disk space would slightly increase. [15:37:59] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.65, 10.37, 8.20 [15:38:23] paladox: we don't copy any dot file/folder outside of staging [15:38:33] The staging clone on 11 & test3 still has it [15:38:40] But tbh I doubt it's much [15:38:47] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.80, 5.22, 5.05 [15:38:57] PROBLEM - housing.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - housing.wiki All nameservers failed to answer the query. [15:39:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.76, 23.61, 22.53 [15:39:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.66, 6.72, 6.43 [15:40:25] paladox: let's wait for CosmicAlpha before merge [15:40:31] ok [15:40:46] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.13, 4.89, 4.95 [15:42:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.22, 7.45, 7.48 [15:47:40] PROBLEM - housing.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.43, 7.01, 7.03 [15:52:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.66, 7.63, 7.32 [15:52:37] paladox: hmm, if you go to https://harrypotter.miraheze.org/wiki/ you'll get an exception ID [15:52:45] but when I search it on Graylog I don't get any results [15:52:49] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.dreamhost.com.', 'ns3.dreamhost.com.', 'ns1.dreamhost.com.'], 'CNAME': None} [15:52:49] for example 4a43e1da51bb43777cd45228 [15:53:07] Reception123: it's slots [15:53:21] We should know that error by now [15:53:25] RhinosF1: that's what I thought but why's graylog silent? [15:53:37] and there's two kinds of slot related errors [15:53:52] one of them just needs a DB insert but the other is just weird [15:54:00] No idea [15:54:16] so I'd like to know why it's not showing up in graylog [15:54:37] PROBLEM - housing.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'housing.wiki' expires in 11 day(s) (Tue 19 Oct 2021 23:12:14 GMT +0000). [15:56:16] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.10, 8.59, 6.64 [15:56:17] uh [15:56:20]