[00:00:15] miraheze/mw-config - Universal-Omega the build passed. [00:03:20] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:04:50] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega - Testing Snap! Skin, last run 14 minutes ago with 0 failures [00:06:17] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.50, 20.97, 15.26 [00:09:26] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-1/±1] 13https://git.io/JnpVf [00:09:27] [02miraheze/mediawiki] 07Universal-Omega 03db0e4c0 - Remove Snap! Skin [00:10:45] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/JnpVq [00:10:47] [02miraheze/mediawiki] 07Universal-Omega 032e0de79 - Install snapwikiskin [00:11:07] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JnpVm [00:11:09] [02miraheze/mediawiki] 07Universal-Omega 0308c6d4a - ignore = dirty [00:11:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpVC [00:11:58] [02miraheze/mw-config] 07Universal-Omega 0304e5409 - Update LocalExtensions.php [00:12:00] [02mw-config] 07Universal-Omega synchronize pull request 03#3980: Install Snap! Skin - 13https://git.io/Jnp2R [00:12:22] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:12:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpVl [00:12:31] [02miraheze/mw-config] 07Universal-Omega 03feb3741 - Update LocalSettings.php [00:12:32] [02mw-config] 07Universal-Omega synchronize pull request 03#3980: Install Snap! Skin - 13https://git.io/Jnp2R [00:12:34] [02mw-config] 07Universal-Omega synchronize pull request 03#3980: Install Snap! Skin - 13https://git.io/Jnp2R [00:12:35] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:12:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpVB [00:12:54] [02miraheze/mw-config] 07Universal-Omega 03ace6a32 - Update ManageWikiExtensions.php [00:12:55] [02mw-config] 07Universal-Omega synchronize pull request 03#3980: Install Snap! Skin - 13https://git.io/Jnp2R [00:12:57] miraheze/mw-config - Universal-Omega the build passed. [00:13:22] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:13:28] miraheze/mw-config - Universal-Omega the build passed. [00:13:34] miraheze/mw-config - Universal-Omega the build passed. [00:13:48] [02mw-config] 07Universal-Omega edited pull request 03#3980: Install Snap! Wiki Skin - 13https://git.io/Jnp2R [00:13:51] miraheze/mw-config - Universal-Omega the build passed. [00:14:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.15, 22.50, 18.96 [00:14:52] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:18:50] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega - Testing Snap! Wiki Skin, last run 2 minutes ago with 0 failures [00:20:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpwL [00:20:59] [02miraheze/mw-config] 07Universal-Omega 0376627a7 - Update extension-list [00:21:00] [02mw-config] 07Universal-Omega synchronize pull request 03#3980: Install Snap! Wiki Skin - 13https://git.io/Jnp2R [00:21:38] miraheze/mw-config - Universal-Omega the build has errored. [00:21:55] miraheze/mw-config - Universal-Omega the build passed. [00:23:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.67, 19.85, 19.34 [00:24:27] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jnpw6 [00:24:28] [02miraheze/MirahezeMagic] 07Universal-Omega 03d9fec63 - Fix [00:25:23] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:29:00] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/Jnprr [00:29:02] [02miraheze/mediawiki] 07Universal-Omega 03210400a - Update MirahezeMagic [00:29:15] [02mw-config] 07Universal-Omega closed pull request 03#3980: Install Snap! Wiki Skin - 13https://git.io/Jnp2R [00:29:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JnprP [00:29:18] [02miraheze/mw-config] 07Universal-Omega 03c81d5c8 - Install Snap! Wiki Skin (#3980) [00:29:19] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:29:21] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [00:30:17] miraheze/mw-config - Universal-Omega the build passed. [00:31:20] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:33:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.78, 21.67, 20.47 [00:35:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.95, 22.96, 21.10 [00:36:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.35, 6.68, 5.74 [00:38:21] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:38] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:40:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.95, 7.51, 6.26 [00:41:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.64, 23.18, 22.04 [00:42:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.38, 7.80, 6.54 [00:43:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki on mw* / jbr* / test3 [00:43:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:22] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:43:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 23.78, 24.60, 22.78 [00:46:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.99, 6.58, 6.32 [00:49:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.40, 6.96, 6.15 [00:51:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.87, 6.23, 5.99 [00:54:02] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpiI [00:54:03] [02miraheze/ManageWiki] 07Universal-Omega 0379d2afb - Update ManageWikiFormFactoryBuilder.php [00:54:05] [02ManageWiki] 07Universal-Omega synchronize pull request 03#279: Allow to blacklist 'new' namespaces - 13https://git.io/Jnx7p [00:55:03] miraheze/ManageWiki - Universal-Omega the build passed. [00:58:30] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jnpib [00:58:31] [02miraheze/ManageWiki] 07Universal-Omega 0316feb6d - Update ManageWikiFormFactoryBuilder.php [00:58:33] [02ManageWiki] 07Universal-Omega synchronize pull request 03#279: Allow to blacklist 'new' namespaces - 13https://git.io/Jnx7p [00:59:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.26, 20.21, 23.08 [00:59:26] miraheze/ManageWiki - Universal-Omega the build passed. [01:07:16] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.45, 15.50, 19.57 [01:11:05] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpX5 [01:11:06] [02miraheze/ManageWiki] 07Universal-Omega 0350fa7c4 - Update ManageWikiFormFactoryBuilder.php [01:11:08] [02ManageWiki] 07Universal-Omega synchronize pull request 03#279: Allow to blacklist 'new' namespaces - 13https://git.io/Jnx7p [01:12:03] miraheze/ManageWiki - Universal-Omega the build passed. [01:16:39] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jnp1X [01:16:40] [02miraheze/ManageWiki] 07Universal-Omega 03882671f - Update ManageWikiFormFactoryBuilder.php [01:16:42] [02ManageWiki] 07Universal-Omega synchronize pull request 03#279: Allow to blacklist 'new' namespaces - 13https://git.io/Jnx7p [01:17:35] miraheze/ManageWiki - Universal-Omega the build passed. [01:21:36] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpMz [01:21:37] [02miraheze/ManageWiki] 07Universal-Omega 03db9fe46 - Update ManageWikiFormFactoryBuilder.php [01:21:39] [02ManageWiki] 07Universal-Omega synchronize pull request 03#279: Allow to blacklist 'new' namespaces - 13https://git.io/Jnx7p [01:22:31] miraheze/ManageWiki - Universal-Omega the build passed. [01:23:30] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpM9 [01:23:31] [02miraheze/ManageWiki] 07Universal-Omega 034d4cc10 - Update ManageWikiFormFactoryBuilder.php [01:23:33] [02ManageWiki] 07Universal-Omega synchronize pull request 03#279: Allow to blacklist 'new' namespaces - 13https://git.io/Jnx7p [01:24:29] miraheze/ManageWiki - Universal-Omega the build passed. [01:24:48] [02ManageWiki] 07Universal-Omega edited pull request 03#279: Allow to whitelist namespaces - 13https://git.io/Jnx7p [01:25:03] hmm I just grouped miraheze-github to the mirahezebots account and it didn't seem to work [01:26:24] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpDg [01:26:26] [02miraheze/ManageWiki] 07Universal-Omega 039a8aa1c - Update ManageWikiFormFactoryBuilder.php [01:26:27] [02ManageWiki] 07Universal-Omega synchronize pull request 03#279: Allow to whitelist namespaces - 13https://git.io/Jnx7p [01:26:36] [02ManageWiki] 07Universal-Omega closed pull request 03#279: Allow to whitelist namespaces - 13https://git.io/Jnx7p [01:26:38] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnpD2 [01:26:40] [02miraheze/ManageWiki] 07Universal-Omega 038b23794 - Allow to whitelist namespaces (#279) [01:26:41] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [01:26:43] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:27:32] miraheze/ManageWiki - Universal-Omega the build passed. [01:28:20] miraheze/ManageWiki - Universal-Omega the build passed. [01:29:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpDb [01:29:48] [02miraheze/mw-config] 07Universal-Omega 03c332cda - use 'whitelisted for wgMetaNamespace(Talk) [01:29:49] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:29:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JnpDA [01:29:56] [02miraheze/mediawiki] 07Universal-Omega 03d1e5983 - Update ManageWiki [01:29:57] [02mw-config] 07Universal-Omega opened pull request 03#3981: use 'whitelisted for wgMetaNamespace(Talk) - 13https://git.io/JnpDp [01:30:52] miraheze/mw-config - Universal-Omega the build passed. [01:31:37] Ohhh, I think I see the problem [01:32:37] !log updated mirahezebots github secret [01:32:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:45] miraheze/mw-config - The-Voidwalker the build passed. [01:35:30] [02mw-config] 07Universal-Omega closed pull request 03#3981: use 'whitelisted for wgMetaNamespace(Talk) - 13https://git.io/JnpDp [01:35:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnpyX [01:35:33] [02miraheze/mw-config] 07Universal-Omega 03ff2543c - use 'whitelisted for wgMetaNamespace(Talk) (#3981) [01:35:35] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:35:36] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:36:21] miraheze/mw-config - Universal-Omega the build passed. [01:39:40] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±1] 13https://git.io/JnpSO [01:39:41] [02miraheze/mediawiki] 07Universal-Omega 0331d9ab6 - Install WebP [01:40:00] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JnpSZ [01:40:01] [02miraheze/mediawiki] 07Universal-Omega 039883569 - ignore = dirty [01:40:50] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:42:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpSx [01:42:31] [02miraheze/mw-config] 07Universal-Omega 03475219c - Update LocalSettings.php [01:42:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:42:45] [02mw-config] 07Universal-Omega opened pull request 03#3982: Install WebP extension - 13https://git.io/JnpSh [01:43:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jnp9I [01:43:32] [02miraheze/mw-config] 07Universal-Omega 0348f2123 - Update LocalExtensions.php [01:43:33] [02mw-config] 07Universal-Omega synchronize pull request 03#3982: Install WebP extension - 13https://git.io/JnpSh [01:43:43] miraheze/mw-config - Universal-Omega the build passed. [01:43:57] :O it worked! [01:44:29] miraheze/mw-config - Universal-Omega the build passed. [01:45:36] Voidwalker: Great! [01:50:47] I'd hesitate to add more logins to the account though, as the github integration pushes it up to five consecutive logins, and I have no idea how much libera likes that [01:52:10] [02puppet] 07Universal-Omega opened pull request 03#1796: Install libwebp-dev package - 13https://git.io/JnpH0 [01:53:08] Voidwalker: ^ do you mind checking that and hopefully merging so I can finish installing the extension? If not it's ok also. [01:57:04] [02puppet] 07Universal-Omega synchronize pull request 03#1796: Install libwebp-dev package - 13https://git.io/JnpH0 [01:57:10] [02puppet] 07Universal-Omega edited pull request 03#1796: Install libwebp package - 13https://git.io/JnpH0 [01:57:50] [02puppet] 07Universal-Omega synchronize pull request 03#1796: Install libwebp package - 13https://git.io/JnpH0 [01:57:56] [02puppet] 07Universal-Omega edited pull request 03#1796: Install libwebp-dev package - 13https://git.io/JnpH0 [02:00:00] is that final? :P [02:00:43] yeah, I just got confused for a minute. https://packages.debian.org/buster/libwebp-dev [02:00:44] [url] Debian -- Details of package libwebp-dev in buster | packages.debian.org [02:02:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jnp7J [02:02:30] [02miraheze/mw-config] 07Universal-Omega 0321332e5 - Update ManageWikiExtensions.php [02:02:32] [02mw-config] 07Universal-Omega synchronize pull request 03#3982: Install WebP extension - 13https://git.io/JnpSh [02:02:53] yeah, also got confused as there is libwebp6 too [02:03:27] miraheze/mw-config - Universal-Omega the build passed. [02:04:32] Voidwalker: Yeah, I just used -dev because the other ones are using -dev packages. (libjpeg-dev, libgif-dev, librsvg2-dev) [02:05:36] [02puppet] 07The-Voidwalker closed pull request 03#1796: Install libwebp-dev package - 13https://git.io/JnpH0 [02:05:37] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jnp7o [02:05:39] [02miraheze/puppet] 07Universal-Omega 031042844 - Install libwebp-dev package (#1796) [02:05:50] Thanks! [02:05:57] [02puppet] 07The-Voidwalker closed pull request 03#1795: Move mwscript to bin directory - 13https://git.io/Jnpuq [02:05:58] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/Jnp7y [02:06:00] [02miraheze/puppet] 07Universal-Omega 03d5ae621 - Move mwscript to bin directory (#1795) [02:06:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jnp7d [02:06:23] [02miraheze/mw-config] 07Universal-Omega 03109edda - Update extension-list [02:06:25] [02mw-config] 07Universal-Omega synchronize pull request 03#3982: Install WebP extension - 13https://git.io/JnpSh [02:07:14] miraheze/mw-config - Universal-Omega the build passed. [02:09:12] PROBLEM - speleo.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for speleo.wiki could not be found [02:09:16] PROBLEM - www.highdeologywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.highdeologywiki.com could not be found [02:09:17] PROBLEM - wiki.invinciblewellbeing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.invinciblewellbeing.com could not be found [02:09:17] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 39.81, 24.11, 16.45 [02:09:17] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thunis.eu could not be found [02:09:19] PROBLEM - miraheze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.com could not be found [02:09:19] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [02:09:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.25, 6.02, 3.60 [02:10:52] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega - Testing WebP extension, last run 2 minutes ago with 0 failures [02:11:23] PROBLEM - ao90.pinho.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ao90.pinho.org could not be found [02:11:24] PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.villagecollaborative.net could not be found [02:12:11] PROBLEM - wikimoma.art - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikimoma.art could not be found [02:12:11] PROBLEM - froggy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for froggy.info could not be found [02:12:11] PROBLEM - miraheze.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ga could not be found [02:12:13] PROBLEM - spiral.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for spiral.wiki could not be found [02:12:32] PROBLEM - www.baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.baharna.org could not be found [02:12:32] PROBLEM - bn.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bn.gyaanipedia.com could not be found [02:12:34] PROBLEM - www.sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sdiy.info could not be found [02:12:39] PROBLEM - wiki.mastodon.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mastodon.kr could not be found [02:12:40] PROBLEM - worldsanskrit.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for worldsanskrit.net could not be found [02:13:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.00, 19.54, 16.59 [02:13:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.54, 5.43, 4.04 [02:14:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JnpdJ [02:14:16] [02miraheze/mw-config] 07Universal-Omega 038e4ba06 - Update ManageWikiExtensions.php [02:14:17] [02mw-config] 07Universal-Omega synchronize pull request 03#3982: Install WebP extension - 13https://git.io/JnpSh [02:15:14] miraheze/mw-config - Universal-Omega the build passed. [02:15:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.71, 4.16, 3.74 [02:15:53] RECOVERY - speleo.wiki - reverse DNS on sslhost is OK: rDNS OK - speleo.wiki reverse DNS resolves to cp13.miraheze.org [02:15:57] RECOVERY - www.highdeologywiki.com - reverse DNS on sslhost is OK: rDNS OK - www.highdeologywiki.com reverse DNS resolves to cp13.miraheze.org [02:15:59] RECOVERY - wiki.invinciblewellbeing.com - reverse DNS on sslhost is OK: rDNS OK - wiki.invinciblewellbeing.com reverse DNS resolves to cp13.miraheze.org [02:16:00] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.thunis.eu reverse DNS resolves to cp13.miraheze.org [02:16:05] RECOVERY - miraheze.com - reverse DNS on sslhost is OK: rDNS OK - miraheze.com reverse DNS resolves to cp13.miraheze.org [02:16:06] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org [02:18:06] RECOVERY - ao90.pinho.org - reverse DNS on sslhost is OK: rDNS OK - ao90.pinho.org reverse DNS resolves to cp13.miraheze.org [02:18:09] RECOVERY - wiki.villagecollaborative.net - reverse DNS on sslhost is OK: rDNS OK - wiki.villagecollaborative.net reverse DNS resolves to cp13.miraheze.org [02:18:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jnpd2 [02:18:16] [02miraheze/mw-config] 07Universal-Omega 034dcda8c - Update LocalWiki.php [02:18:18] [02mw-config] 07Universal-Omega synchronize pull request 03#3982: Install WebP extension - 13https://git.io/JnpSh [02:19:00] RECOVERY - wikimoma.art - reverse DNS on sslhost is OK: rDNS OK - wikimoma.art reverse DNS resolves to cp13.miraheze.org [02:19:00] RECOVERY - froggy.info - reverse DNS on sslhost is OK: rDNS OK - froggy.info reverse DNS resolves to cp13.miraheze.org [02:19:01] RECOVERY - miraheze.ga - reverse DNS on sslhost is OK: rDNS OK - miraheze.ga reverse DNS resolves to cp13.miraheze.org [02:19:05] RECOVERY - spiral.wiki - reverse DNS on sslhost is OK: rDNS OK - spiral.wiki reverse DNS resolves to cp13.miraheze.org [02:19:19] RECOVERY - www.baharna.org - reverse DNS on sslhost is OK: rDNS OK - www.baharna.org reverse DNS resolves to cp13.miraheze.org [02:19:31] RECOVERY - wiki.mastodon.kr - reverse DNS on sslhost is OK: rDNS OK - wiki.mastodon.kr reverse DNS resolves to cp13.miraheze.org [02:19:32] RECOVERY - bn.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - bn.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [02:19:32] RECOVERY - www.sdiy.info - reverse DNS on sslhost is OK: rDNS OK - www.sdiy.info reverse DNS resolves to cp13.miraheze.org [02:19:36] RECOVERY - worldsanskrit.net - reverse DNS on sslhost is OK: rDNS OK - worldsanskrit.net reverse DNS resolves to cp13.miraheze.org [02:19:39] miraheze/mw-config - Universal-Omega the build passed. [02:20:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jnpdh [02:20:49] [02miraheze/mw-config] 07Universal-Omega 038e3ee0d - Update LocalSettings.php [02:20:50] [02mw-config] 07Universal-Omega synchronize pull request 03#3982: Install WebP extension - 13https://git.io/JnpSh [02:21:42] miraheze/mw-config - Universal-Omega the build passed. [02:22:51] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [02:26:52] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega - Testing WebP extension, last run 4 minutes ago with 0 failures [02:40:08] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.47, 1.72, 1.23 [02:42:09] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.90, 1.50, 1.22 [03:24:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JnhJg [03:24:38] [02miraheze/mw-config] 07Universal-Omega 03a97169c - CommentStreams conflicts with Moderation [03:24:39] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [03:24:41] [02mw-config] 07Universal-Omega opened pull request 03#3983: CommentStreams conflicts with Moderation - 13https://git.io/JnhJw [03:25:36] miraheze/mw-config - Universal-Omega the build passed. [03:25:51] [02mw-config] 07Universal-Omega closed pull request 03#3983: CommentStreams conflicts with Moderation - 13https://git.io/JnhJw [03:25:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnhJD [03:25:54] [02miraheze/mw-config] 07Universal-Omega 03bee51e1 - CommentStreams conflicts with Moderation (#3983) [03:25:56] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [03:25:57] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [03:26:43] miraheze/mw-config - Universal-Omega the build passed. [06:11:59] Nobody could I dentify this ??? [06:13:01] Have changed that name but I still need to set the wikibase repo name to FAMEData And then the logo to FAMEData logo as well [06:13:08] Identify what? [06:49:28] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JnhKo [06:49:30] [02miraheze/MirahezeMagic] 07Universal-Omega 03c308bc7 - Also delete the setting on wiki deletion [06:49:31] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#266: Update ManageWiki database type when renaming wikis - 13https://git.io/JnKfd [06:50:25] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:51:53] [02MirahezeMagic] 07Universal-Omega edited pull request 03#266: Update ManageWiki database type when renaming or deleting wikis - 13https://git.io/JnKfd [06:52:47] How to change the wikidata logo here to FAMEData logo [06:53:34] Here [06:54:54] [02puppet] 07Universal-Omega opened pull request 03#1797: Add TransformWebPImageJob to jobQueueCollector - 13https://git.io/Jnh68 [07:07:00] [02puppet] 07Reception123 closed pull request 03#1792: Robots.php: use factory services - 13https://git.io/JnKRl [07:07:02] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnhPW [07:07:03] [02miraheze/puppet] 07Universal-Omega 036ecfd40 - Robots.php: use factory services (#1792) [07:09:48] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnhPS [07:09:49] [02miraheze/puppet] 07Universal-Omega 03eb14725 - Add TransformWebPImageJob to jobQueueCollector (#1797) [07:09:51] [02puppet] 07Reception123 closed pull request 03#1797: Add TransformWebPImageJob to jobQueueCollector - 13https://git.io/Jnh68 [07:11:04] @Universal Omega [07:11:36] Ugochimobi: yes? [07:13:05] @Universal Omega please read from here [07:13:39] Discord? [07:13:44] What's the difference? [07:15:50] Oh I see. [07:16:05] I have no idea. Sorry @Ugochimobi [09:51:58] [02ssl] 07Reception123 commented on commit 036106417a87a903b45d4bdcbc13834379a1fe5128 - 13https://git.io/JnjOb [10:06:35] No problems. [10:06:46] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 10.54, 8.89, 4.53 [10:11:40] PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:11:51] PROBLEM - cp13 Stunnel Http for mw9 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:11:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [10:11:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [10:12:05] Error 503 Backend fetch failed, forwarded for 105.112.101.190, 127.0.0.1 (Varnish XID 370119070) via cp13.miraheze.org at Sat, 26 Jun 2021 10:10:59 GMT. [10:12:14] Hope you're aware of this? [10:13:38] RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 0.304 second response time [10:13:47] RECOVERY - cp13 Stunnel Http for mw9 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.008 second response time [10:13:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:13:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:20:44] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 0.87, 6.30, 6.66 [10:24:42] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 11.37, 9.19, 7.73 [10:25:02] PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:25:15] PROBLEM - cp13 Stunnel Http for mw10 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:25:36] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [10:25:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [10:25:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [10:26:10] PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:26:11] PROBLEM - cp12 Stunnel Http for mw11 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:26:23] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:26:33] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:27:03] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 2 backends are down. mw8 mw11 [10:27:37] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is WARNING: WARNING - NGINX Error Rate is 41% [10:28:40] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 5.422 second response time [10:29:09] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 1.035 second response time [10:29:13] RECOVERY - cp13 Stunnel Http for mw10 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 0.004 second response time [10:29:35] RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 2% [10:29:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:29:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:30:08] RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.005 second response time [10:30:26] RECOVERY - cp12 Stunnel Http for mw11 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15418 bytes in 6.642 second response time [10:30:38] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15418 bytes in 7.601 second response time [10:31:00] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [10:33:14] !log running delbackups ( bash ./delbackups.sh /home/reception/delwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php) [10:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:40:08] PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:40:16] PROBLEM - cp13 Stunnel Http for mw11 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:40:40] PROBLEM - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is WARNING: WARNING - NGINX Error Rate is 47% [10:41:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [10:42:02] RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.005 second response time [10:42:13] RECOVERY - cp13 Stunnel Http for mw11 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15412 bytes in 0.081 second response time [10:42:18] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8 [10:42:40] RECOVERY - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is OK: OK - NGINX Error Rate is 2% [10:43:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:44:14] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [10:48:10] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw9 [10:48:46] PROBLEM - egoish.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for egoish.org could not be found [10:49:50] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw9 [10:50:06] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [10:51:49] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [10:52:51] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki nonciclopediawiki [10:55:34] RECOVERY - egoish.org - reverse DNS on sslhost is OK: rDNS OK - egoish.org reverse DNS resolves to cp13.miraheze.org [10:55:44] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 3 backends are down. mw8 mw9 mw11 [10:55:46] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [10:55:59] PROBLEM - cp13 Stunnel Http for mw11 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:56:00] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw8 mw9 mw11 [10:56:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:56:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [10:56:47] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw10 [10:57:56] RECOVERY - cp13 Stunnel Http for mw11 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15418 bytes in 0.004 second response time [10:58:50] PROBLEM - dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dariawiki.org could not be found [10:58:56] PROBLEM - christipedia.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for christipedia.nl could not be found [10:58:58] PROBLEM - phab-storage.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phab-storage.bots.miraheze.wiki could not be found [10:58:59] PROBLEM - wiki.scvo.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scvo.org could not be found [10:59:00] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:01] PROBLEM - baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for baharna.org could not be found [10:59:23] PROBLEM - puzzles.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for puzzles.wiki could not be found [10:59:25] PROBLEM - sopel.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sopel.bots.miraheze.wiki could not be found [10:59:28] PROBLEM - jobrunner3 MediaWiki Rendering on jobrunner3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.054 second response time [10:59:28] PROBLEM - famedata.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for famedata.org could not be found [10:59:29] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thelonsdalebattalion.co.uk could not be found [10:59:38] PROBLEM - civwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for civwiki.org could not be found [10:59:38] PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lefrenchmelee.fr could not be found [10:59:39] PROBLEM - www.fossilfighterswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fossilfighterswiki.com could not be found [10:59:39] PROBLEM - wiki.openhatch.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.openhatch.org could not be found [10:59:42] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [10:59:44] PROBLEM - runzeppelin.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for runzeppelin.ru could not be found [10:59:47] PROBLEM - zh.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.gyaanipedia.com could not be found [10:59:49] PROBLEM - wiki.thehall.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thehall.xyz could not be found [10:59:51] PROBLEM - meta.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.gyaanipedia.com could not be found [10:59:52] PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found [10:59:52] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fbpml.org could not be found [10:59:52] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for diamowiki.ga could not be found [10:59:53] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [11:00:00] PROBLEM - ethene.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ethene.wiki could not be found [11:00:00] PROBLEM - towarzystwo.prawa.kredytowego.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for towarzystwo.prawa.kredytowego.pl could not be found [11:00:02] PROBLEM - petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for petrawiki.org could not be found [11:00:02] PROBLEM - debilepedie.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for debilepedie.gq could not be found [11:00:02] PROBLEM - wiki.nevillepedia.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nevillepedia.eu could not be found [11:00:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.040 second response time [11:00:48] PROBLEM - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wildcard.bots.miraheze.wiki could not be found [11:00:53] PROBLEM - wiki.scrungecraft.gg - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scrungecraft.gg could not be found [11:00:56] PROBLEM - wiki.slimelab.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimelab.net could not be found [11:00:56] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found [11:00:56] PROBLEM - wiki.mxlinuxusers.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mxlinuxusers.de could not be found [11:00:56] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 19988 bytes in 0.137 second response time [11:00:57] PROBLEM - wiki.worldsofweary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldsofweary.com could not be found [11:00:57] PROBLEM - savage-wiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for savage-wiki.com could not be found [11:00:57] PROBLEM - wiki.kourouklides.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kourouklides.com could not be found [11:01:00] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [11:01:03] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for housing.wiki could not be found [11:01:05] PROBLEM - wiki.ciptamedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ciptamedia.org could not be found [11:01:06] PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found [11:01:06] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tl.awiki.org could not be found [11:01:13] PROBLEM - speleo.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for speleo.wiki could not be found [11:01:16] PROBLEM - www.highdeologywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.highdeologywiki.com could not be found [11:01:16] PROBLEM - sims.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sims.miraheze.org could not be found [11:01:16] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for you.r-fit.cc could not be found [11:01:16] PROBLEM - wiki.invinciblewellbeing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.invinciblewellbeing.com could not be found [11:01:17] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mineland.eu could not be found [11:01:17] PROBLEM - wiki.fourta.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fourta.org could not be found [11:01:17] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thunis.eu could not be found [11:01:18] PROBLEM - lg.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lg.gyaanipedia.com could not be found [11:01:18] PROBLEM - zw.fontainebleau-avon.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zw.fontainebleau-avon.fr could not be found [11:01:19] PROBLEM - miraheze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.com could not be found [11:01:19] PROBLEM - uncyclopediakr.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for uncyclopediakr.miraheze.org could not be found [11:01:20] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [11:01:26] RECOVERY - jobrunner3 MediaWiki Rendering on jobrunner3 is OK: HTTP OK: HTTP/1.1 200 OK - 20000 bytes in 0.116 second response time [11:01:29] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.078 second response time [11:01:34] PROBLEM - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.marinebiodiversitymatrix.org could not be found [11:01:35] PROBLEM - zh.internetpedia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.internetpedia.tk could not be found [11:01:37] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wikimedia.cat could not be found [11:01:38] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1051 bytes in 0.126 second response time [11:01:57] PROBLEM - cp13 Stunnel Http for mw11 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.003 second response time [11:02:04] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 19986 bytes in 0.188 second response time [11:02:35] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [11:02:40] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:03:29] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19986 bytes in 0.121 second response time [11:03:34] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20023 bytes in 0.162 second response time [11:03:37] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [11:03:48] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [11:03:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:03:55] RECOVERY - cp13 Stunnel Http for mw11 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 0.009 second response time [11:04:38] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 0.925 second response time [11:05:34] RECOVERY - dariawiki.org - reverse DNS on sslhost is OK: rDNS OK - dariawiki.org reverse DNS resolves to cp13.miraheze.org [11:05:47] RECOVERY - christipedia.nl - reverse DNS on sslhost is OK: rDNS OK - christipedia.nl reverse DNS resolves to cp13.miraheze.org [11:05:50] RECOVERY - phab-storage.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phab-storage.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:05:51] RECOVERY - wiki.scvo.org - reverse DNS on sslhost is OK: rDNS OK - wiki.scvo.org reverse DNS resolves to cp13.miraheze.org [11:05:55] RECOVERY - baharna.org - reverse DNS on sslhost is OK: rDNS OK - baharna.org reverse DNS resolves to cp13.miraheze.org [11:06:07] RECOVERY - famedata.org - reverse DNS on sslhost is OK: rDNS OK - famedata.org reverse DNS resolves to cp13.miraheze.org [11:06:11] RECOVERY - thelonsdalebattalion.co.uk - reverse DNS on sslhost is OK: rDNS OK - thelonsdalebattalion.co.uk reverse DNS resolves to cp13.miraheze.org [11:06:19] RECOVERY - puzzles.wiki - reverse DNS on sslhost is OK: rDNS OK - puzzles.wiki reverse DNS resolves to cp13.miraheze.org [11:06:23] RECOVERY - sopel.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - sopel.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:06:27] RECOVERY - civwiki.org - reverse DNS on sslhost is OK: rDNS OK - civwiki.org reverse DNS resolves to cp13.miraheze.org [11:06:27] RECOVERY - wiki.lefrenchmelee.fr - reverse DNS on sslhost is OK: rDNS OK - wiki.lefrenchmelee.fr reverse DNS resolves to cp13.miraheze.org [11:06:28] RECOVERY - wiki.thehall.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.thehall.xyz reverse DNS resolves to cp13.miraheze.org [11:06:29] RECOVERY - wiki.openhatch.org - reverse DNS on sslhost is OK: rDNS OK - wiki.openhatch.org reverse DNS resolves to cp13.miraheze.org [11:06:33] RECOVERY - meta.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - meta.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [11:06:35] RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - pt.graalmilitary.com reverse DNS resolves to cp13.miraheze.org [11:06:35] RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fbpml.org reverse DNS resolves to cp13.miraheze.org [11:06:36] RECOVERY - www.mh142.com - reverse DNS on sslhost is OK: rDNS OK - www.mh142.com reverse DNS resolves to cp13.miraheze.org [11:06:39] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org [11:06:40] RECOVERY - www.fossilfighterswiki.com - reverse DNS on sslhost is OK: rDNS OK - www.fossilfighterswiki.com reverse DNS resolves to cp13.miraheze.org [11:06:41] RECOVERY - runzeppelin.ru - reverse DNS on sslhost is OK: rDNS OK - runzeppelin.ru reverse DNS resolves to cp13.miraheze.org [11:06:43] RECOVERY - zh.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - zh.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [11:06:48] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.064 second response time [11:06:51] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: rDNS OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org [11:06:52] RECOVERY - ethene.wiki - reverse DNS on sslhost is OK: rDNS OK - ethene.wiki reverse DNS resolves to cp13.miraheze.org [11:06:52] RECOVERY - towarzystwo.prawa.kredytowego.pl - reverse DNS on sslhost is OK: rDNS OK - towarzystwo.prawa.kredytowego.pl reverse DNS resolves to cp13.miraheze.org [11:06:54] RECOVERY - petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - petrawiki.org reverse DNS resolves to cp13.miraheze.org [11:06:55] RECOVERY - debilepedie.gq - reverse DNS on sslhost is OK: rDNS OK - debilepedie.gq reverse DNS resolves to cp13.miraheze.org [11:06:56] RECOVERY - wiki.nevillepedia.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.nevillepedia.eu reverse DNS resolves to cp13.miraheze.org [11:07:28] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.106 second response time [11:07:34] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 2 backends are down. mw9 mw11 [11:07:42] RECOVERY - savage-wiki.com - reverse DNS on sslhost is OK: rDNS OK - savage-wiki.com reverse DNS resolves to cp12.miraheze.org [11:07:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw8 mw10 mw11 [11:07:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb [11:07:53] RECOVERY - speleo.wiki - reverse DNS on sslhost is OK: rDNS OK - speleo.wiki reverse DNS resolves to cp12.miraheze.org [11:07:54] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: rDNS OK - housing.wiki reverse DNS resolves to cp12.miraheze.org [11:07:56] RECOVERY - www.highdeologywiki.com - reverse DNS on sslhost is OK: rDNS OK - www.highdeologywiki.com reverse DNS resolves to cp12.miraheze.org [11:07:57] RECOVERY - sims.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - sims.miraheze.org reverse DNS resolves to cp12.miraheze.org [11:07:57] RECOVERY - wiki.ciptamedia.org - reverse DNS on sslhost is OK: rDNS OK - wiki.ciptamedia.org reverse DNS resolves to cp12.miraheze.org [11:07:58] RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: rDNS OK - you.r-fit.cc reverse DNS resolves to cp12.miraheze.org [11:07:59] RECOVERY - wiki.invinciblewellbeing.com - reverse DNS on sslhost is OK: rDNS OK - wiki.invinciblewellbeing.com reverse DNS resolves to cp12.miraheze.org [11:07:59] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.mineland.eu reverse DNS resolves to cp12.miraheze.org [11:08:01] RECOVERY - zw.fontainebleau-avon.fr - reverse DNS on sslhost is OK: rDNS OK - zw.fontainebleau-avon.fr reverse DNS resolves to cp12.miraheze.org [11:08:03] RECOVERY - miraheze.com - reverse DNS on sslhost is OK: rDNS OK - miraheze.com reverse DNS resolves to cp13.miraheze.org [11:08:04] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org [11:08:16] RECOVERY - zh.internetpedia.tk - reverse DNS on sslhost is OK: rDNS OK - zh.internetpedia.tk reverse DNS resolves to cp13.miraheze.org [11:08:17] RECOVERY - uncyclopediakr.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - uncyclopediakr.miraheze.org reverse DNS resolves to cp13.miraheze.org [11:08:21] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: rDNS OK - wiki.wikimedia.cat reverse DNS resolves to cp13.miraheze.org [11:08:24] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 19988 bytes in 0.114 second response time [11:08:32] RECOVERY - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: rDNS OK - www.marinebiodiversitymatrix.org reverse DNS resolves to cp13.miraheze.org [11:08:44] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 19988 bytes in 0.531 second response time [11:09:18] PROBLEM - wiki.ct777.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ct777.cf could not be found [11:09:18] PROBLEM - wiki.valkyrienskies.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.valkyrienskies.org could not be found [11:09:18] PROBLEM - reviews.thejoshmeister.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviews.thejoshmeister.com could not be found [11:09:19] PROBLEM - kk.uncyc.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kk.uncyc.tk could not be found [11:09:19] PROBLEM - wiki.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fraterniterebellion.com could not be found [11:09:19] PROBLEM - wiki.counterculturelabs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.counterculturelabs.org could not be found [11:09:28] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19986 bytes in 0.126 second response time [11:09:29] PROBLEM - thestarsareright.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thestarsareright.org could not be found [11:09:29] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [11:09:30] PROBLEM - www.opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.opendatascot.org could not be found [11:09:30] PROBLEM - wiki.yunachannel.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yunachannel.com could not be found [11:09:32] PROBLEM - www.puzzles.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.puzzles.wiki could not be found [11:09:33] PROBLEM - wiki.rpointer.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rpointer.ml could not be found [11:09:33] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for infectowiki.com could not be found [11:09:36] PROBLEM - idleon.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for idleon.info could not be found [11:09:37] PROBLEM - wiki.rebirthofthenight.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rebirthofthenight.com could not be found [11:09:37] PROBLEM - www.christipedia.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.christipedia.nl could not be found [11:09:39] PROBLEM - wiki.grottocenter.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grottocenter.org could not be found [11:09:39] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vedopedia.witches-empire.com could not be found [11:09:40] PROBLEM - wiki.worlduniversityandschool.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worlduniversityandschool.org could not be found [11:09:41] PROBLEM - history.estill.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.estill.org could not be found [11:09:45] PROBLEM - tools1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tools1.miraheze.wiki could not be found [11:09:47] PROBLEM - wiki.arnab.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.arnab.at could not be found [11:09:47] PROBLEM - wiki.jumpbound.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jumpbound.com could not be found [11:09:51] PROBLEM - wiki.xysspon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.xysspon.com could not be found [11:09:51] PROBLEM - wiki.s23.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.s23.org could not be found [11:09:56] PROBLEM - monarchists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for monarchists.wiki could not be found [11:09:57] PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minkyu.kim could not be found [11:09:57] PROBLEM - dc.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc.miraheze.org could not be found [11:09:57] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [11:10:00] PROBLEM - www.dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dariawiki.org could not be found [11:10:01] PROBLEM - www.johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.johanloopmans.nl could not be found [11:10:20] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [11:10:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:11:31] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [11:11:40] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [11:11:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:14:15] RECOVERY - wiki.scrungecraft.gg - reverse DNS on sslhost is OK: rDNS OK - wiki.scrungecraft.gg reverse DNS resolves to cp13.miraheze.org [11:14:23] RECOVERY - wiki.slimelab.net - reverse DNS on sslhost is OK: rDNS OK - wiki.slimelab.net reverse DNS resolves to cp13.miraheze.org [11:14:24] RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikiru.wildterra2.com reverse DNS resolves to cp13.miraheze.org [11:14:24] RECOVERY - wiki.mxlinuxusers.de - reverse DNS on sslhost is OK: rDNS OK - wiki.mxlinuxusers.de reverse DNS resolves to cp13.miraheze.org [11:14:27] RECOVERY - wiki.worldsofweary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.worldsofweary.com reverse DNS resolves to cp13.miraheze.org [11:14:27] RECOVERY - wiki.kourouklides.com - reverse DNS on sslhost is OK: rDNS OK - wiki.kourouklides.com reverse DNS resolves to cp13.miraheze.org [11:14:37] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: rDNS OK - wiki.zymonic.com reverse DNS resolves to cp13.miraheze.org [11:14:41] RECOVERY - wiki.fourta.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fourta.org reverse DNS resolves to cp13.miraheze.org [11:14:44] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.thunis.eu reverse DNS resolves to cp13.miraheze.org [11:14:44] RECOVERY - lg.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - lg.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [11:14:45] RECOVERY - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - wildcard.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:14:55] RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:14:55] RECOVERY - tl.awiki.org - reverse DNS on sslhost is OK: rDNS OK - tl.awiki.org reverse DNS resolves to cp13.miraheze.org [11:16:00] RECOVERY - wiki.ct777.cf - reverse DNS on sslhost is OK: rDNS OK - wiki.ct777.cf reverse DNS resolves to cp13.miraheze.org [11:16:00] RECOVERY - reviews.thejoshmeister.com - reverse DNS on sslhost is OK: rDNS OK - reviews.thejoshmeister.com reverse DNS resolves to cp13.miraheze.org [11:16:01] RECOVERY - wiki.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - wiki.fraterniterebellion.com reverse DNS resolves to cp13.miraheze.org [11:16:02] RECOVERY - kk.uncyc.tk - reverse DNS on sslhost is OK: rDNS OK - kk.uncyc.tk reverse DNS resolves to cp13.miraheze.org [11:16:02] RECOVERY - wiki.counterculturelabs.org - reverse DNS on sslhost is OK: rDNS OK - wiki.counterculturelabs.org reverse DNS resolves to cp13.miraheze.org [11:16:16] RECOVERY - wiki.rebirthofthenight.com - reverse DNS on sslhost is OK: rDNS OK - wiki.rebirthofthenight.com reverse DNS resolves to cp13.miraheze.org [11:16:17] RECOVERY - wiki.valkyrienskies.org - reverse DNS on sslhost is OK: rDNS OK - wiki.valkyrienskies.org reverse DNS resolves to cp13.miraheze.org [11:16:17] RECOVERY - www.christipedia.nl - reverse DNS on sslhost is OK: rDNS OK - www.christipedia.nl reverse DNS resolves to cp13.miraheze.org [11:16:20] RECOVERY - thestarsareright.org - reverse DNS on sslhost is OK: rDNS OK - thestarsareright.org reverse DNS resolves to cp13.miraheze.org [11:16:21] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: rDNS OK - vedopedia.witches-empire.com reverse DNS resolves to cp13.miraheze.org [11:16:21] RECOVERY - wiki.grottocenter.org - reverse DNS on sslhost is OK: rDNS OK - wiki.grottocenter.org reverse DNS resolves to cp13.miraheze.org [11:16:22] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp13.miraheze.org [11:16:23] RECOVERY - wiki.worlduniversityandschool.org - reverse DNS on sslhost is OK: rDNS OK - wiki.worlduniversityandschool.org reverse DNS resolves to cp13.miraheze.org [11:16:23] RECOVERY - www.opendatascot.org - reverse DNS on sslhost is OK: rDNS OK - www.opendatascot.org reverse DNS resolves to cp13.miraheze.org [11:16:24] RECOVERY - wiki.yunachannel.com - reverse DNS on sslhost is OK: rDNS OK - wiki.yunachannel.com reverse DNS resolves to cp13.miraheze.org [11:16:25] RECOVERY - history.estill.org - reverse DNS on sslhost is OK: rDNS OK - history.estill.org reverse DNS resolves to cp13.miraheze.org [11:16:27] RECOVERY - www.puzzles.wiki - reverse DNS on sslhost is OK: rDNS OK - www.puzzles.wiki reverse DNS resolves to cp13.miraheze.org [11:16:28] RECOVERY - wiki.rpointer.ml - reverse DNS on sslhost is OK: rDNS OK - wiki.rpointer.ml reverse DNS resolves to cp13.miraheze.org [11:16:29] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: rDNS OK - infectowiki.com reverse DNS resolves to cp13.miraheze.org [11:16:34] RECOVERY - tools1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - tools1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:16:34] RECOVERY - wiki.jumpbound.com - reverse DNS on sslhost is OK: rDNS OK - wiki.jumpbound.com reverse DNS resolves to cp13.miraheze.org [11:16:35] RECOVERY - idleon.info - reverse DNS on sslhost is OK: rDNS OK - idleon.info reverse DNS resolves to cp13.miraheze.org [11:16:35] RECOVERY - wiki.arnab.at - reverse DNS on sslhost is OK: rDNS OK - wiki.arnab.at reverse DNS resolves to cp13.miraheze.org [11:16:36] RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: rDNS OK - wiki.minkyu.kim reverse DNS resolves to cp13.miraheze.org [11:16:36] RECOVERY - dc.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - dc.miraheze.org reverse DNS resolves to cp13.miraheze.org [11:16:36] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp13.miraheze.org [11:16:43] RECOVERY - www.dariawiki.org - reverse DNS on sslhost is OK: rDNS OK - www.dariawiki.org reverse DNS resolves to cp13.miraheze.org [11:16:43] RECOVERY - wiki.xysspon.com - reverse DNS on sslhost is OK: rDNS OK - wiki.xysspon.com reverse DNS resolves to cp13.miraheze.org [11:16:44] RECOVERY - wiki.s23.org - reverse DNS on sslhost is OK: rDNS OK - wiki.s23.org reverse DNS resolves to cp13.miraheze.org [11:16:45] RECOVERY - www.johanloopmans.nl - reverse DNS on sslhost is OK: rDNS OK - www.johanloopmans.nl reverse DNS resolves to cp13.miraheze.org [11:16:55] RECOVERY - monarchists.wiki - reverse DNS on sslhost is OK: rDNS OK - monarchists.wiki reverse DNS resolves to cp13.miraheze.org [11:21:10] PROBLEM - jobrunner3 MediaWiki Rendering on jobrunner3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.050 second response time [11:21:25] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 3 backends are down. mw8 mw9 mw11 [11:21:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw9 mw11 [11:21:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:21:56] PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found [11:21:58] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 3 backends are down. mw8 mw9 mw10 [11:22:03] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [11:22:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:22:12] PROBLEM - wiki.openhatch.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.openhatch.org could not be found [11:22:22] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [11:22:24] PROBLEM - wiki.velaan.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.velaan.org could not be found [11:22:25] PROBLEM - files.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.pornwiki.org could not be found [11:23:08] RECOVERY - jobrunner3 MediaWiki Rendering on jobrunner3 is OK: HTTP OK: HTTP/1.1 200 OK - 20000 bytes in 0.229 second response time [11:23:24] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [11:23:30] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [11:23:33] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [11:23:35] PROBLEM - zh.internetpedia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.internetpedia.tk could not be found [11:23:37] PROBLEM - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wildcard.bots.miraheze.wiki could not be found [11:23:38] PROBLEM - www.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pornwiki.org could not be found [11:23:41] PROBLEM - wiki.globasa.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.globasa.net could not be found [11:23:46] PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found [11:23:47] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tl.awiki.org could not be found [11:23:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:23:48] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wikimedia.cat could not be found [11:23:50] PROBLEM - arru.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for arru.xyz could not be found [11:23:51] PROBLEM - hololive.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hololive.wiki could not be found [11:23:55] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [11:23:58] PROBLEM - www.trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.trollpasta.com could not be found [11:24:02] PROBLEM - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.hoolehistoryheritagesociety.org.uk could not be found [11:24:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:28:38] RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - pt.graalmilitary.com reverse DNS resolves to cp13.miraheze.org [11:28:47] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org [11:29:20] RECOVERY - www.mh142.com - reverse DNS on sslhost is OK: rDNS OK - www.mh142.com reverse DNS resolves to cp13.miraheze.org [11:29:20] RECOVERY - files.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - files.pornwiki.org reverse DNS resolves to cp13.miraheze.org [11:29:40] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [11:29:41] PROBLEM - miraheze.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.gq could not be found [11:29:42] PROBLEM - ref.undu.st - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ref.undu.st could not be found [11:29:46] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [11:29:47] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for diamowiki.ga could not be found [11:29:47] PROBLEM - www.iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.iceria.org could not be found [11:29:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:29:48] PROBLEM - wiki.pablojprieto.com.es - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pablojprieto.com.es could not be found [11:29:48] PROBLEM - ecole.science - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ecole.science could not be found [11:29:49] PROBLEM - wiki.slimelab.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimelab.net could not be found [11:29:51] PROBLEM - sarovia.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sarovia.graalmilitary.com could not be found [11:29:52] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found [11:29:52] PROBLEM - wiki.worldsofweary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldsofweary.com could not be found [11:29:52] PROBLEM - wiki.mxlinuxusers.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mxlinuxusers.de could not be found [11:29:55] PROBLEM - ndg.nenawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ndg.nenawiki.org could not be found [11:29:55] PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikien.wildterra2.com could not be found [11:29:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:29:56] PROBLEM - mr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mr.gyaanipedia.com could not be found [11:29:58] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [11:29:58] PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hr.petrawiki.org could not be found [11:29:58] PROBLEM - wiki.kourouklides.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kourouklides.com could not be found [11:29:58] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for you.r-fit.cc could not be found [11:29:59] PROBLEM - wiki.invinciblewellbeing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.invinciblewellbeing.com could not be found [11:30:00] PROBLEM - wiki.fourta.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fourta.org could not be found [11:30:07] PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimeapedia.org could not be found [11:30:16] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org [11:30:17] RECOVERY - zh.internetpedia.tk - reverse DNS on sslhost is OK: rDNS OK - zh.internetpedia.tk reverse DNS resolves to cp13.miraheze.org [11:30:19] RECOVERY - www.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - www.pornwiki.org reverse DNS resolves to cp13.miraheze.org [11:30:24] RECOVERY - wiki.globasa.net - reverse DNS on sslhost is OK: rDNS OK - wiki.globasa.net reverse DNS resolves to cp13.miraheze.org [11:30:31] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: rDNS OK - wiki.wikimedia.cat reverse DNS resolves to cp13.miraheze.org [11:30:34] RECOVERY - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - wildcard.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:30:36] RECOVERY - www.trollpasta.com - reverse DNS on sslhost is OK: rDNS OK - www.trollpasta.com reverse DNS resolves to cp13.miraheze.org [11:30:41] RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:30:41] RECOVERY - tl.awiki.org - reverse DNS on sslhost is OK: rDNS OK - tl.awiki.org reverse DNS resolves to cp13.miraheze.org [11:30:42] RECOVERY - arru.xyz - reverse DNS on sslhost is OK: rDNS OK - arru.xyz reverse DNS resolves to cp13.miraheze.org [11:30:44] RECOVERY - hololive.wiki - reverse DNS on sslhost is OK: rDNS OK - hololive.wiki reverse DNS resolves to cp13.miraheze.org [11:30:44] RECOVERY - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is OK: rDNS OK - www.hoolehistoryheritagesociety.org.uk reverse DNS resolves to cp13.miraheze.org [11:31:00] PROBLEM - wiki.digitaldesignhq.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitaldesignhq.com could not be found [11:31:01] PROBLEM - wiki.hrznstudio.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hrznstudio.com could not be found [11:31:10] PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gesamtschule-nordkirchen.de could not be found [11:31:12] PROBLEM - pastport.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pastport.org could not be found [11:31:16] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [11:31:16] PROBLEM - uncyclopediakr.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for uncyclopediakr.miraheze.org could not be found [11:31:17] PROBLEM - archive.a2b2.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.a2b2.org could not be found [11:31:19] PROBLEM - dcmultiverse.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dcmultiverse.miraheze.org could not be found [11:31:19] PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found [11:31:21] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 2 backends are down. mw10 mw11 [11:31:34] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw9 mw10 mw11 [11:31:36] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.055 second response time [11:31:42] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [11:31:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:31:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:33:03] PROBLEM - kunwok.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kunwok.org could not be found [11:33:03] PROBLEM - puardian.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for puardian.xyz could not be found [11:33:09] PROBLEM - allthetropes.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.orain.org could not be found [11:33:14] PROBLEM - tensegritywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tensegritywiki.com could not be found [11:33:14] PROBLEM - wiki.mikrodev.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mikrodev.com could not be found [11:33:16] PROBLEM - wiki.cdntennis.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cdntennis.ca could not be found [11:33:18] PROBLEM - wiki.funkey-project.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.funkey-project.com could not be found [11:33:18] PROBLEM - wiki.newnormal.fans - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.newnormal.fans could not be found [11:33:20] PROBLEM - wiki.patriam.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.patriam.cc could not be found [11:33:21] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [11:33:28] PROBLEM - iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iceria.org could not be found [11:33:29] PROBLEM - wiki.vinesh.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.vinesh.eu.org could not be found [11:33:29] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [11:33:29] PROBLEM - fr.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fr.gyaanipedia.com could not be found [11:33:29] PROBLEM - wiki.whentheycry.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.whentheycry.org could not be found [11:33:31] PROBLEM - wmworld.sktz.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wmworld.sktz.live could not be found [11:33:33] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [11:33:33] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 19988 bytes in 0.140 second response time [11:33:34] PROBLEM - wiki.wilderyogi.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wilderyogi.eu could not be found [11:35:53] RECOVERY - wiki.openhatch.org - reverse DNS on sslhost is OK: rDNS OK - wiki.openhatch.org reverse DNS resolves to cp13.miraheze.org [11:36:12] RECOVERY - wiki.velaan.org - reverse DNS on sslhost is OK: rDNS OK - wiki.velaan.org reverse DNS resolves to cp13.miraheze.org [11:36:30] RECOVERY - ref.undu.st - reverse DNS on sslhost is OK: rDNS OK - ref.undu.st reverse DNS resolves to cp13.miraheze.org [11:36:32] RECOVERY - wiki.slimelab.net - reverse DNS on sslhost is OK: rDNS OK - wiki.slimelab.net reverse DNS resolves to cp13.miraheze.org [11:36:34] RECOVERY - ndg.nenawiki.org - reverse DNS on sslhost is OK: rDNS OK - ndg.nenawiki.org reverse DNS resolves to cp13.miraheze.org [11:36:35] RECOVERY - wikien.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikien.wildterra2.com reverse DNS resolves to cp13.miraheze.org [11:36:35] RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikiru.wildterra2.com reverse DNS resolves to cp13.miraheze.org [11:36:36] RECOVERY - wiki.mxlinuxusers.de - reverse DNS on sslhost is OK: rDNS OK - wiki.mxlinuxusers.de reverse DNS resolves to cp13.miraheze.org [11:36:37] RECOVERY - wiki.worldsofweary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.worldsofweary.com reverse DNS resolves to cp13.miraheze.org [11:36:39] RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp13.miraheze.org [11:36:40] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: rDNS OK - www.lab612.at reverse DNS resolves to cp13.miraheze.org [11:36:40] RECOVERY - www.iceria.org - reverse DNS on sslhost is OK: rDNS OK - www.iceria.org reverse DNS resolves to cp13.miraheze.org [11:36:40] RECOVERY - hr.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - hr.petrawiki.org reverse DNS resolves to cp13.miraheze.org [11:36:40] RECOVERY - miraheze.gq - reverse DNS on sslhost is OK: rDNS OK - miraheze.gq reverse DNS resolves to cp13.miraheze.org [11:36:40] RECOVERY - wiki.pablojprieto.com.es - reverse DNS on sslhost is OK: rDNS OK - wiki.pablojprieto.com.es reverse DNS resolves to cp13.miraheze.org [11:36:40] RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: rDNS OK - you.r-fit.cc reverse DNS resolves to cp13.miraheze.org [11:36:41] RECOVERY - ecole.science - reverse DNS on sslhost is OK: rDNS OK - ecole.science reverse DNS resolves to cp13.miraheze.org [11:36:41] RECOVERY - wiki.invinciblewellbeing.com - reverse DNS on sslhost is OK: rDNS OK - wiki.invinciblewellbeing.com reverse DNS resolves to cp13.miraheze.org [11:36:42] RECOVERY - wiki.fourta.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fourta.org reverse DNS resolves to cp13.miraheze.org [11:36:43] RECOVERY - wiki.kourouklides.com - reverse DNS on sslhost is OK: rDNS OK - wiki.kourouklides.com reverse DNS resolves to cp13.miraheze.org [11:36:45] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: rDNS OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org [11:36:46] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is WARNING: WARNING - NGINX Error Rate is 53% [11:36:47] RECOVERY - sarovia.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - sarovia.graalmilitary.com reverse DNS resolves to cp13.miraheze.org [11:36:55] RECOVERY - mr.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - mr.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [11:37:00] RECOVERY - wiki.slimeapedia.org - reverse DNS on sslhost is OK: rDNS OK - wiki.slimeapedia.org reverse DNS resolves to cp13.miraheze.org [11:37:03] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.050 second response time [11:37:09] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.051 second response time [11:37:18] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [11:37:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [11:37:38] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [11:37:42] RECOVERY - wiki.digitaldesignhq.com - reverse DNS on sslhost is OK: rDNS OK - wiki.digitaldesignhq.com reverse DNS resolves to cp13.miraheze.org [11:37:45] RECOVERY - wiki.hrznstudio.com - reverse DNS on sslhost is OK: rDNS OK - wiki.hrznstudio.com reverse DNS resolves to cp13.miraheze.org [11:37:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:37:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:38:03] RECOVERY - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is OK: rDNS OK - wiki.gesamtschule-nordkirchen.de reverse DNS resolves to cp13.miraheze.org [11:38:05] RECOVERY - pastport.org - reverse DNS on sslhost is OK: rDNS OK - pastport.org reverse DNS resolves to cp13.miraheze.org [11:38:14] RECOVERY - archive.a2b2.org - reverse DNS on sslhost is OK: rDNS OK - archive.a2b2.org reverse DNS resolves to cp13.miraheze.org [11:38:15] RECOVERY - guia.esporo.net - reverse DNS on sslhost is OK: rDNS OK - guia.esporo.net reverse DNS resolves to cp13.miraheze.org [11:38:15] RECOVERY - uncyclopediakr.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - uncyclopediakr.miraheze.org reverse DNS resolves to cp13.miraheze.org [11:38:16] RECOVERY - dcmultiverse.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - dcmultiverse.miraheze.org reverse DNS resolves to cp13.miraheze.org [11:38:16] RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: rDNS OK - madgenderscience.wiki reverse DNS resolves to cp13.miraheze.org [11:38:44] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [11:38:58] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20023 bytes in 0.191 second response time [11:39:05] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 19988 bytes in 0.126 second response time [11:39:43] RECOVERY - puardian.xyz - reverse DNS on sslhost is OK: rDNS OK - puardian.xyz reverse DNS resolves to cp13.miraheze.org [11:39:54] RECOVERY - allthetropes.orain.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.orain.org reverse DNS resolves to cp13.miraheze.org [11:40:01] RECOVERY - kunwok.org - reverse DNS on sslhost is OK: rDNS OK - kunwok.org reverse DNS resolves to cp13.miraheze.org [11:40:02] RECOVERY - tensegritywiki.com - reverse DNS on sslhost is OK: rDNS OK - tensegritywiki.com reverse DNS resolves to cp13.miraheze.org [11:40:02] RECOVERY - wiki.mikrodev.com - reverse DNS on sslhost is OK: rDNS OK - wiki.mikrodev.com reverse DNS resolves to cp13.miraheze.org [11:40:08] RECOVERY - iceria.org - reverse DNS on sslhost is OK: rDNS OK - iceria.org reverse DNS resolves to cp13.miraheze.org [11:40:09] RECOVERY - wiki.funkey-project.com - reverse DNS on sslhost is OK: rDNS OK - wiki.funkey-project.com reverse DNS resolves to cp13.miraheze.org [11:40:11] RECOVERY - wiki.newnormal.fans - reverse DNS on sslhost is OK: rDNS OK - wiki.newnormal.fans reverse DNS resolves to cp13.miraheze.org [11:40:12] RECOVERY - wiki.vinesh.eu.org - reverse DNS on sslhost is OK: rDNS OK - wiki.vinesh.eu.org reverse DNS resolves to cp13.miraheze.org [11:40:12] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: rDNS OK - storytime.jdstroy.cf reverse DNS resolves to cp13.miraheze.org [11:40:12] RECOVERY - fr.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - fr.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [11:40:13] RECOVERY - wiki.whentheycry.org - reverse DNS on sslhost is OK: rDNS OK - wiki.whentheycry.org reverse DNS resolves to cp13.miraheze.org [11:40:14] RECOVERY - wiki.patriam.cc - reverse DNS on sslhost is OK: rDNS OK - wiki.patriam.cc reverse DNS resolves to cp13.miraheze.org [11:40:15] RECOVERY - wiki.cdntennis.ca - reverse DNS on sslhost is OK: rDNS OK - wiki.cdntennis.ca reverse DNS resolves to cp13.miraheze.org [11:40:16] RECOVERY - wmworld.sktz.live - reverse DNS on sslhost is OK: rDNS OK - wmworld.sktz.live reverse DNS resolves to cp13.miraheze.org [11:40:22] RECOVERY - wiki.wilderyogi.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.wilderyogi.eu reverse DNS resolves to cp12.miraheze.org [11:40:41] RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 10% [11:41:33] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [11:41:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:41:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:42:55] PROBLEM - wiki.scrungecraft.gg - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scrungecraft.gg could not be found [11:42:58] PROBLEM - puzzles.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for puzzles.wiki could not be found [11:43:15] PROBLEM - speleo.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for speleo.wiki could not be found [11:45:38] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [11:48:34] PROBLEM - marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marinebiodiversitymatrix.org could not be found [11:48:35] PROBLEM - allthetropes.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.orain.org could not be found [11:48:59] PROBLEM - wmworld.sktz.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wmworld.sktz.live could not be found [11:49:00] PROBLEM - trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for trollpasta.com could not be found [11:49:01] PROBLEM - wiki.newnormal.fans - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.newnormal.fans could not be found [11:49:02] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for halloween.gyaanipedia.com could not be found [11:49:05] PROBLEM - phab-storage.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phab-storage.bots.miraheze.wiki could not be found [11:49:05] PROBLEM - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phabdigests.bots.miraheze.wiki could not be found [11:49:11] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:12] PROBLEM - www.erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.erikapedia.com could not be found [11:49:14] PROBLEM - mediawiki.ecfin.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mediawiki.ecfin.eu could not be found [11:49:17] PROBLEM - matttest.mtwiki.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for matttest.mtwiki.cf could not be found [11:49:17] PROBLEM - wiki.cdntennis.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cdntennis.ca could not be found [11:49:18] PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found [11:49:18] PROBLEM - wiki.park-city.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.park-city.club could not be found [11:49:19] PROBLEM - test2.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for test2.miraheze.org could not be found [11:49:30] PROBLEM - baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for baharna.org could not be found [11:49:30] PROBLEM - wiki.warframestat.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.warframestat.us could not be found [11:49:31] PROBLEM - enc.for.uz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for enc.for.uz could not be found [11:49:36] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [11:49:36] PROBLEM - www.portalsofphereon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.portalsofphereon.com could not be found [11:49:36] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.landev.vn could not be found [11:49:37] PROBLEM - isv.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for isv.orain.org could not be found [11:49:37] RECOVERY - wiki.scrungecraft.gg - reverse DNS on sslhost is OK: rDNS OK - wiki.scrungecraft.gg reverse DNS resolves to cp13.miraheze.org [11:49:38] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [11:49:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:49:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:49:55] RECOVERY - puzzles.wiki - reverse DNS on sslhost is OK: rDNS OK - puzzles.wiki reverse DNS resolves to cp13.miraheze.org [11:49:55] RECOVERY - speleo.wiki - reverse DNS on sslhost is OK: rDNS OK - speleo.wiki reverse DNS resolves to cp13.miraheze.org [11:51:07] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 19988 bytes in 0.112 second response time [11:51:38] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [11:51:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:51:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:53:15] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [11:53:34] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [11:55:21] RECOVERY - allthetropes.orain.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.orain.org reverse DNS resolves to cp12.miraheze.org [11:55:32] RECOVERY - marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: rDNS OK - marinebiodiversitymatrix.org reverse DNS resolves to cp13.miraheze.org [11:55:46] RECOVERY - wmworld.sktz.live - reverse DNS on sslhost is OK: rDNS OK - wmworld.sktz.live reverse DNS resolves to cp13.miraheze.org [11:55:52] RECOVERY - trollpasta.com - reverse DNS on sslhost is OK: rDNS OK - trollpasta.com reverse DNS resolves to cp13.miraheze.org [11:55:53] RECOVERY - www.erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - www.erikapedia.com reverse DNS resolves to cp13.miraheze.org [11:55:53] RECOVERY - wiki.newnormal.fans - reverse DNS on sslhost is OK: rDNS OK - wiki.newnormal.fans reverse DNS resolves to cp13.miraheze.org [11:55:56] RECOVERY - halloween.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - halloween.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [11:55:57] RECOVERY - mediawiki.ecfin.eu - reverse DNS on sslhost is OK: rDNS OK - mediawiki.ecfin.eu reverse DNS resolves to cp13.miraheze.org [11:55:57] RECOVERY - phab-storage.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phab-storage.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:56:01] RECOVERY - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phabdigests.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [11:56:01] RECOVERY - matttest.mtwiki.cf - reverse DNS on sslhost is OK: rDNS OK - matttest.mtwiki.cf reverse DNS resolves to cp13.miraheze.org [11:56:02] RECOVERY - www.mcpk.wiki - reverse DNS on sslhost is OK: rDNS OK - www.mcpk.wiki reverse DNS resolves to cp13.miraheze.org [11:56:04] RECOVERY - wiki.park-city.club - reverse DNS on sslhost is OK: rDNS OK - wiki.park-city.club reverse DNS resolves to cp13.miraheze.org [11:56:08] RECOVERY - test2.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - test2.miraheze.org reverse DNS resolves to cp13.miraheze.org [11:56:16] RECOVERY - wiki.cdntennis.ca - reverse DNS on sslhost is OK: rDNS OK - wiki.cdntennis.ca reverse DNS resolves to cp13.miraheze.org [11:56:18] RECOVERY - www.portalsofphereon.com - reverse DNS on sslhost is OK: rDNS OK - www.portalsofphereon.com reverse DNS resolves to cp13.miraheze.org [11:56:21] RECOVERY - wiki.landev.vn - reverse DNS on sslhost is OK: rDNS OK - wiki.landev.vn reverse DNS resolves to cp13.miraheze.org [11:56:22] RECOVERY - isv.orain.org - reverse DNS on sslhost is OK: rDNS OK - isv.orain.org reverse DNS resolves to cp13.miraheze.org [11:56:25] RECOVERY - baharna.org - reverse DNS on sslhost is OK: rDNS OK - baharna.org reverse DNS resolves to cp13.miraheze.org [11:56:27] RECOVERY - wiki.warframestat.us - reverse DNS on sslhost is OK: rDNS OK - wiki.warframestat.us reverse DNS resolves to cp13.miraheze.org [11:56:29] RECOVERY - enc.for.uz - reverse DNS on sslhost is OK: rDNS OK - enc.for.uz reverse DNS resolves to cp13.miraheze.org [11:58:16] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.173 second response time [11:59:11] PROBLEM - jobrunner4 MediaWiki Rendering on jobrunner4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1051 bytes in 0.092 second response time [11:59:27] PROBLEM - jobrunner3 MediaWiki Rendering on jobrunner3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1029 bytes in 0.158 second response time [11:59:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [11:59:38] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [11:59:42] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [11:59:44] PROBLEM - www.opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.opendatascot.org could not be found [11:59:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:59:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [11:59:47] PROBLEM - wiki.aridia.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aridia.space could not be found [11:59:48] PROBLEM - hellointernet.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hellointernet.miraheze.org could not be found [11:59:48] PROBLEM - marthaspeaks.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marthaspeaks.miraheze.org could not be found [11:59:49] PROBLEM - wiki.wysteria.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wysteria.ga could not be found [11:59:51] PROBLEM - aman.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for aman.awiki.org could not be found [11:59:59] PROBLEM - wiki.ssij.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssij.se could not be found [11:59:59] PROBLEM - en.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.pornwiki.org could not be found [12:00:00] PROBLEM - podpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for podpedia.org could not be found [12:00:00] PROBLEM - dc-multiverse.dcwikis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc-multiverse.dcwikis.com could not be found [12:00:20] PROBLEM - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wildcard.bots.miraheze.wiki could not be found [12:00:22] PROBLEM - johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for johanloopmans.nl could not be found [12:00:25] PROBLEM - olwest.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for olwest.org could not be found [12:00:26] PROBLEM - mrlove.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mrlove.wiki could not be found [12:00:27] PROBLEM - gp.ct777.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for gp.ct777.cf could not be found [12:00:28] PROBLEM - wiki.seredrau.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seredrau.xyz could not be found [12:00:29] PROBLEM - www.baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.baharna.org could not be found [12:00:31] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.astralprojections.org could not be found [12:00:31] PROBLEM - translate.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for translate.petrawiki.org could not be found [12:00:31] PROBLEM - opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for opendatascot.org could not be found [12:00:33] PROBLEM - www.rangpurpedia.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.rangpurpedia.xyz could not be found [12:00:35] PROBLEM - www.permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.permanentfuturelab.wiki could not be found [12:00:36] PROBLEM - mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mh142.com could not be found [12:00:40] PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gesamtschule-nordkirchen.de could not be found [12:00:41] PROBLEM - worldsanskrit.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for worldsanskrit.net could not be found [12:00:41] PROBLEM - civclassic.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for civclassic.miraheze.org could not be found [12:00:45] PROBLEM - allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for allthetropes.org could not be found [12:00:45] PROBLEM - www.bluepageswiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.bluepageswiki.org could not be found [12:01:09] RECOVERY - jobrunner4 MediaWiki Rendering on jobrunner4 is OK: HTTP OK: HTTP/1.1 200 OK - 20000 bytes in 0.516 second response time [12:01:14] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 2 backends are down. mw9 mw10 [12:01:25] RECOVERY - jobrunner3 MediaWiki Rendering on jobrunner3 is OK: HTTP OK: HTTP/1.1 200 OK - 20000 bytes in 0.182 second response time [12:01:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:01:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:02:14] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20023 bytes in 0.353 second response time [12:03:15] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [12:03:33] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [12:03:38] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [12:06:35] RECOVERY - hellointernet.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - hellointernet.miraheze.org reverse DNS resolves to cp13.miraheze.org [12:06:35] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp13.miraheze.org [12:06:36] RECOVERY - marthaspeaks.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - marthaspeaks.miraheze.org reverse DNS resolves to cp13.miraheze.org [12:06:38] RECOVERY - wiki.wysteria.ga - reverse DNS on sslhost is OK: rDNS OK - wiki.wysteria.ga reverse DNS resolves to cp13.miraheze.org [12:06:38] RECOVERY - www.opendatascot.org - reverse DNS on sslhost is OK: rDNS OK - www.opendatascot.org reverse DNS resolves to cp13.miraheze.org [12:06:39] RECOVERY - podpedia.org - reverse DNS on sslhost is OK: rDNS OK - podpedia.org reverse DNS resolves to cp13.miraheze.org [12:06:40] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: rDNS OK - dc-multiverse.dcwikis.com reverse DNS resolves to cp13.miraheze.org [12:06:47] RECOVERY - wiki.aridia.space - reverse DNS on sslhost is OK: rDNS OK - wiki.aridia.space reverse DNS resolves to cp13.miraheze.org [12:06:50] RECOVERY - aman.awiki.org - reverse DNS on sslhost is OK: rDNS OK - aman.awiki.org reverse DNS resolves to cp13.miraheze.org [12:06:56] RECOVERY - wiki.ssij.se - reverse DNS on sslhost is OK: rDNS OK - wiki.ssij.se reverse DNS resolves to cp13.miraheze.org [12:06:58] RECOVERY - en.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - en.pornwiki.org reverse DNS resolves to cp13.miraheze.org [12:07:01] RECOVERY - johanloopmans.nl - reverse DNS on sslhost is OK: rDNS OK - johanloopmans.nl reverse DNS resolves to cp13.miraheze.org [12:07:08] RECOVERY - olwest.org - reverse DNS on sslhost is OK: rDNS OK - olwest.org reverse DNS resolves to cp13.miraheze.org [12:07:11] RECOVERY - mrlove.wiki - reverse DNS on sslhost is OK: rDNS OK - mrlove.wiki reverse DNS resolves to cp13.miraheze.org [12:07:11] RECOVERY - gp.ct777.cf - reverse DNS on sslhost is OK: rDNS OK - gp.ct777.cf reverse DNS resolves to cp13.miraheze.org [12:07:17] RECOVERY - wiki.seredrau.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.seredrau.xyz reverse DNS resolves to cp13.miraheze.org [12:07:19] RECOVERY - www.baharna.org - reverse DNS on sslhost is OK: rDNS OK - www.baharna.org reverse DNS resolves to cp13.miraheze.org [12:07:20] RECOVERY - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - wildcard.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [12:07:21] RECOVERY - wiki.astralprojections.org - reverse DNS on sslhost is OK: rDNS OK - wiki.astralprojections.org reverse DNS resolves to cp13.miraheze.org [12:07:21] RECOVERY - translate.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - translate.petrawiki.org reverse DNS resolves to cp13.miraheze.org [12:07:22] RECOVERY - opendatascot.org - reverse DNS on sslhost is OK: rDNS OK - opendatascot.org reverse DNS resolves to cp13.miraheze.org [12:07:25] RECOVERY - www.rangpurpedia.xyz - reverse DNS on sslhost is OK: rDNS OK - www.rangpurpedia.xyz reverse DNS resolves to cp13.miraheze.org [12:07:29] RECOVERY - allthetropes.org - reverse DNS on sslhost is OK: rDNS OK - allthetropes.org reverse DNS resolves to cp13.miraheze.org [12:07:29] RECOVERY - www.bluepageswiki.org - reverse DNS on sslhost is OK: rDNS OK - www.bluepageswiki.org reverse DNS resolves to cp13.miraheze.org [12:07:29] RECOVERY - www.permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - www.permanentfuturelab.wiki reverse DNS resolves to cp13.miraheze.org [12:07:31] RECOVERY - mh142.com - reverse DNS on sslhost is OK: rDNS OK - mh142.com reverse DNS resolves to cp13.miraheze.org [12:07:33] RECOVERY - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is OK: rDNS OK - wiki.gesamtschule-nordkirchen.de reverse DNS resolves to cp13.miraheze.org [12:07:37] RECOVERY - worldsanskrit.net - reverse DNS on sslhost is OK: rDNS OK - worldsanskrit.net reverse DNS resolves to cp13.miraheze.org [12:07:38] RECOVERY - civclassic.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - civclassic.miraheze.org reverse DNS resolves to cp13.miraheze.org [12:22:15] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnjXw [12:22:17] [02miraheze/mw-config] 07Reception123 03990546a - temporarily disable GlobalNewFiles [12:23:12] miraheze/mw-config - Reception123 the build passed. [12:35:52] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 0.94, 3.87, 7.26 [12:37:54] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 0.85, 2.88, 6.47 [12:47:56] [02GlobalNewFiles] 07urbanecm opened pull request 03#9: Add __METHOD__ to DB calls in GlobalNewFilesHooks - 13https://git.io/JnjSy [12:50:51] Reception123: please merge & deploy ^ [12:57:08] RhinosF1: will do [12:57:50] miraheze/GlobalNewFiles - Reception123 the build passed. [12:58:06] [02GlobalNewFiles] 07Reception123 closed pull request 03#9: Add __METHOD__ to DB calls in GlobalNewFilesHooks - 13https://git.io/JnjSy [12:58:07] [02miraheze/GlobalNewFiles] 07Reception123 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JnjQU [12:58:09] [02miraheze/GlobalNewFiles] 07urbanecm 03651a4ff - Add __METHOD__ to DB calls in GlobalNewFilesHooks [12:58:10] [02miraheze/GlobalNewFiles] 07Reception123 033a7825a - Merge pull request #9 from urbanecm/add-method [12:58:57] miraheze/GlobalNewFiles - Reception123 the build passed. [14:58:11] would be possible to have a dynamic list of which wikis use specific extensions? For statistical and troubleshooting purposes [14:58:43] @Lake that's probably something you could create a low priority WikiDiscover task for [14:59:01] though if you need any specific list now I can give you a paste :) [15:03:45] Reception123 could you give me one for PageForms? I want to see if my forms are really bugged or if I'm doing something wrong [15:04:51] and sure, I'll create a task. Which tags should I use? [15:05:17] oh, there's a WikiDiscover tag, nvm [15:07:37] Sure [15:11:50] https://www.irccloud.com/pastebin/EhA5UTmL/ [15:11:53] ^ @Lake [15:13:52] thanks! [15:20:11] no problem :) [16:34:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.23, 8.32, 6.27 [16:36:47] PROBLEM - wikibeats.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikibeats.net could not be found [16:36:51] PROBLEM - www.fossilfighterswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fossilfighterswiki.com could not be found [16:36:53] PROBLEM - wiki.scrungecraft.gg - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scrungecraft.gg could not be found [16:36:56] PROBLEM - wiki.slimelab.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimelab.net could not be found [16:36:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.59, 7.50, 6.22 [16:36:56] PROBLEM - wiki.mxlinuxusers.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mxlinuxusers.de could not be found [16:36:57] PROBLEM - wiki.worldsofweary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldsofweary.com could not be found [16:36:57] PROBLEM - wiki.kourouklides.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kourouklides.com could not be found [16:36:58] PROBLEM - savage-wiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for savage-wiki.com could not be found [16:37:00] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [16:37:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 6.80, 5.52 [16:38:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.40, 6.75, 6.10 [16:39:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.82, 6.14, 5.44 [16:43:34] RECOVERY - wiki.scrungecraft.gg - reverse DNS on sslhost is OK: rDNS OK - wiki.scrungecraft.gg reverse DNS resolves to cp13.miraheze.org [16:43:39] RECOVERY - wiki.slimelab.net - reverse DNS on sslhost is OK: rDNS OK - wiki.slimelab.net reverse DNS resolves to cp13.miraheze.org [16:43:40] RECOVERY - wiki.mxlinuxusers.de - reverse DNS on sslhost is OK: rDNS OK - wiki.mxlinuxusers.de reverse DNS resolves to cp13.miraheze.org [16:43:42] RECOVERY - wiki.worldsofweary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.worldsofweary.com reverse DNS resolves to cp13.miraheze.org [16:43:42] RECOVERY - wiki.kourouklides.com - reverse DNS on sslhost is OK: rDNS OK - wiki.kourouklides.com reverse DNS resolves to cp13.miraheze.org [16:43:43] RECOVERY - savage-wiki.com - reverse DNS on sslhost is OK: rDNS OK - savage-wiki.com reverse DNS resolves to cp13.miraheze.org [16:43:44] RECOVERY - wikibeats.net - reverse DNS on sslhost is OK: rDNS OK - wikibeats.net reverse DNS resolves to cp13.miraheze.org [16:43:49] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: rDNS OK - wiki.zymonic.com reverse DNS resolves to cp13.miraheze.org [16:43:51] RECOVERY - www.fossilfighterswiki.com - reverse DNS on sslhost is OK: rDNS OK - www.fossilfighterswiki.com reverse DNS resolves to cp13.miraheze.org [18:19:20] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JcvL9 [18:19:21] [02miraheze/GlobalNewFiles] 07Universal-Omega 0353f2e18 - Use DBLoadBalancerFactory [18:19:23] [02GlobalNewFiles] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/JtShO [18:19:24] [02GlobalNewFiles] 07Universal-Omega opened pull request 03#10: Use DBLoadBalancerFactory - 13https://git.io/JcvLH [18:20:21] miraheze/GlobalNewFiles - Universal-Omega the build passed. [18:56:34] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.55, 2.66, 1.75 [18:58:34] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.09, 2.65, 1.85 [19:50:51] !log published https://github.com/miraheze/GlobalNewFiles/security/advisories/GHSA-cwv5-c938-5h5h & requested CVE [19:50:52] [url] Uncontrolled Resource Consumption in GlobalNewFiles · Advisory · miraheze/GlobalNewFiles · GitHub | github.com [19:50:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:55] Reception123: ^ [19:51:07] ack, thanks [20:40:31] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jcv2T [20:40:32] [02miraheze/GlobalNewFiles] 07Universal-Omega 03251ea1c - Update GlobalNewFilesPager.php [20:40:34] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#10: Use DBLoadBalancerFactory - 13https://git.io/JcvLH [20:41:24] miraheze/GlobalNewFiles - Universal-Omega the build passed. [20:41:27] That seems less a software security issue and more a Miraheze resource problem, no? What was the scale involved? If it’s not in the thousands sounds like a system incapable of working than a security issue [20:42:15] Plus a 1:1 relationship wouldn’t necessarily be a security issue as that seems like a core usability aspect - one input leads to one output [20:44:28] Reading the advisory provides no actual details and seems like the advisory was published for the sake of an advisory and not because a real problem exists worthy of a disclosure - otherwise any small setup could claim utilising more resources than they have is a CVE-level security problem [20:51:25] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#10: Use DBLoadBalancerFactory - 13https://git.io/JcvLH [20:51:27] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JcvaX [20:51:28] [02miraheze/GlobalNewFiles] 07Universal-Omega 03fef1484 - Update GlobalNewFilesHooks.php [20:52:20] miraheze/GlobalNewFiles - Universal-Omega the build passed. [20:52:21] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jcva9 [20:52:23] [02miraheze/GlobalNewFiles] 07Universal-Omega 0326a4c45 - Update GlobalNewFilesPager.php [20:52:24] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#10: Use DBLoadBalancerFactory - 13https://git.io/JcvLH [20:53:12] miraheze/GlobalNewFiles - Universal-Omega the build passed. [21:06:19] @Reception123 see email please for follow up regarding concerns raised above. Thank you [21:14:38] @Owen: because various features in MediaWiki weren't appropriately used that would have prevented it on GlobalUsage's part [21:15:07] Yes theirs failings on Miraheze's part for not using appropiate systems to prevent it [21:15:16] But the code had faults too [21:16:02] @RhinosF1 the issue is, that advisory is extremely poor. There’s no details, looking at the task, I’m not even seeing a real investigation having taken place. I’ve asked Reception for the process that should be followed because from a T&S perspective, I have serious concerns over security practise looking at that advisory [21:17:23] @Owen: currently the extension is disabled and we're advising any third party user in the event their is one also does so unless their confident in their systems. It won't be re-enabled unless satisfactory fixes are in place. [21:18:45] Which is I’m getting from the advisory. Personally, I’m not satisfied with that. Who else reviewed the advisory? Who approved it to be published? [21:19:33] The tasks got lowered to normal for a security issue and it was announced public before a satisfactory investigation took place - which if this is a real security issue worthy of a CVE, is a dire approach [21:22:06] @Owen: Reception123 gave the approval [21:22:47] Can I ask exactly what more you would want? What kind of details you expect? [21:27:48] A proper investigation is the main thing I expect [21:28:09] Which isn’t the impression I’m getting having taken place [21:29:05] There no information as to the scale of what caused this either [21:30:14] Plus the extension seems to be a 1:1 scale, so I’m unsure how it can be deemed a security issue if a small pool of limited resources is exhausted by a processing utilising minimal resources already. If this was a 1 move = 10 actions, that’ll be different as that’s inefficient [21:31:05] RhinosF1: for the record I think it should've waited to publish the CVE until after a patch was done, and merged at the very least. I wanted to wait until my patch was at least done and fixed before doing that. I can't really complain though as you did ping all SRE members also and I didn't respond in time before it was published, but in the future maybe wait on doing that until we have an actual solution or investigation as [21:31:05] Owen said. In the end there wasn't much point to publish that, it gives hardly any information or viable solutions except "don't use it" just an opinion not criticism though. [21:31:14] I don’t understand how an advisory of so little detail can be acceptable and how an advisory can be published before the problem is actually known - which is why I asked for the process and policy as if this is the process, I’ll be having T&S work with SRE on improving that process [21:32:15] @Owen: we know what wiki was responsible and we can share that info with T&S. [21:32:29] I’m not interested in the wiki [21:32:50] I’m interested in the process, which to me is lacking and has an impact on the Trust of Miraheze Limited [21:33:42] If you could share the policy and process, that would assist in moving forward from this point however [21:34:18] I'm not aware of any formal policy tbh. There's Tech:GHSA but that's about it. [21:35:03] If there’s no policy, SRE need to establish one with urgency otherwise T&S would be required to mandate on onto SRE [21:36:59] Tech:GHSA has details on how to write an advisory [21:37:06] But that's all [21:37:28] That was after the last security issue where we realised they were process failings [21:37:46] And there still remains such failings [21:38:31] It's a much better response than last time. Im not saying there isn't but it's much improved. [21:38:52] That is questionable [21:39:08] ‘Much better’ but still poor, is poor I am afraid [21:39:31] I'm not saying there isn't lessons to learn [21:39:41] There most definitely is [21:39:51] From many perspectives [21:40:33] I’ve sent the email to Reception anyway to start the ball rolling [21:41:05] We've got a lot to look at [21:42:38] A 3 hour outage was caused by a single user carrying out legitimate actions because an extension had unknown but significant issues with performance and they were numerous ways that could have prevented it / detected it / helped mitigate it / warned about it / helped identify the issue [21:43:19] Do a post-mortem and work on it - as a priory, this would high as without a defined approach and clear policy, there is a greater risk of poor process than the actual problem itself [21:44:00] I've already filed 3 tasks with possible things that could have HellLord [21:44:12] Helped* [21:44:17] Stupid auto correct [21:44:20] Is one of those a defined policy? If not, it needs to be [21:44:32] No but you know how to create a task [21:44:43] It is not my job to do yours [21:45:13] If I do it, it won’t be a task - it would be a non negotiable policy that can’t be modified without T&S approval [21:45:29] I have 50 million lessons that we could learn from this [21:45:47] When I get the chance to sit down with the rest of SRE, many things will be raised [21:46:33] And the priority has got to be a defined policy. I’m putting the emphasis on it because it’s shocking there isn’t one after 6 years [21:47:33] We need polices for alot of things regarding SRE... but that is one that should take priority for sure. [21:47:35] It will be brought up in what probably needs to be a very soon meeting of SRE [21:47:42] Also there’s no high priority tasks for this [21:47:44] But there's many many gaps [21:47:53] It's unfortunate that today highlited them [21:48:42] The extension is disabled and will remain so until we are happy that it's resolved. A fix isn't going to be rushed because we don't need a repeat. [21:49:23] I'd rather we loose an extension for a few weeks than risk another outage like this [21:50:39] I don’t think that is being contested or challenged [21:50:48] But the lack of proper process is [21:52:15] You've emailed Reception123 so I'm sure they will get back to you [21:53:03] If I don't see anything within a few days then I'll make sure it's on my list of follows ups as a priority [21:53:38] That sounds good [21:55:09] My priority personally is the immediate stability of service though. So far of my week off though I seem to have got little of it and exams need to take priority unless there's an immediate risk to user safety.