[00:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:03:14] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 3.11, 4.23, 2.19 [00:04:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 2.50, 4.70, 2.69 [00:05:14] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 1.33, 3.30, 2.11 [00:06:14] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 1.79, 3.82, 2.62 [00:08:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.86, 3.26, 2.56 [00:11:14] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 8.83, 5.05, 3.07 [00:12:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 8.65, 6.98, 4.22 [00:19:36] RECOVERY - publictestwiki.com - reverse DNS on sslhost is OK: SSL OK - publictestwiki.com reverse DNS resolves to cp23.miraheze.org - CNAME OK [00:22:14] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 0.63, 2.86, 3.68 [00:22:33] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:24:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.18, 2.27, 3.36 [00:35:14] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.99, 1.17, 3.72 [00:39:14] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.86, 1.27, 3.20 [00:48:37] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:50:36] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.285 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [02:01:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.43, 2.87, 2.37 [02:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [02:03:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.14, 3.30, 2.59 [02:05:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.94, 3.44, 2.73 [02:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.66, 3.92, 3.08 [02:13:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.75, 3.92, 3.28 [02:23:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.10, 3.37, 3.32 [02:27:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.79, 3.67, 3.46 [02:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [02:33:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.99, 3.37, 3.40 [02:39:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.25, 3.47, 3.44 [02:41:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.72, 2.87, 3.23 [03:01:10] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 67 seconds old and 0 bytes [03:27:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:30:20] PROBLEM - history.sdtef.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query sdtef.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:56:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:00:17] RECOVERY - history.sdtef.org - reverse DNS on sslhost is OK: SSL OK - history.sdtef.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:18:52] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [04:21:40] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [04:25:40] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [04:43:34] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.altilunium.my.id All nameservers failed to answer the query. [04:56:02] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [04:59:57] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [05:13:33] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [05:16:49] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:23:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:52:34] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:09:44] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.09, 2.47 [06:11:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.72, 3.28, 2.61 [06:13:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.08, 3.48, 2.77 [06:15:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.72, 3.48, 2.85 [06:17:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.55, 3.10, 2.79 [06:26:29] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 3.72, 3.22 [06:28:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 5.46, 4.94, 2.45 [06:28:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.62, 3.61, 3.23 [06:29:14] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 14.39, 29.59, 14.53 [06:30:14] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 1.16, 3.47, 2.21 [06:30:22] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.87, 3.37 [06:32:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 3.29, 3.28, 2.28 [06:32:18] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.32, 3.69, 3.37 [06:36:11] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.64, 3.26, 3.27 [06:40:05] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.30, 3.53, 3.39 [06:42:02] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.35, 3.38, 3.35 [06:45:56] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.52, 3.42 [06:47:53] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.54, 3.77, 3.51 [06:53:14] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.70, 1.04, 3.66 [06:55:14] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 1.08, 1.01, 3.33 [07:01:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.48, 3.53, 3.85 [07:11:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.86, 2.95, 3.35 [07:28:33] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.59, 3.40, 3.23 [07:30:29] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.20, 3.27, 3.19 [07:38:17] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.56, 3.62, 3.31 [07:40:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.73, 3.41 [07:44:05] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.11, 3.08, 3.22 [07:55:01] PROBLEM - cloud13 Puppet on cloud13 is UNKNOWN: NRPE: Unable to read output [08:21:02] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [09:53:38] PROBLEM - cloud14 Puppet on cloud14 is UNKNOWN: NRPE: Unable to read output [10:21:38] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [12:03:29] PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikien.wildterra2.com All nameservers failed to answer the query. [12:11:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.28, 3.18, 2.29 [12:18:56] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/37f57dd3e864 [12:18:58] [02miraheze/MirahezeMagic] 07paladox 0337f57dd - Make sure VanishUse doesn't fatal on non-ca wikis [12:19:00] [02MirahezeMagic] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/MirahezeMagic [12:19:02] [02MirahezeMagic] 07paladox opened pull request 03#429: Make sure VanishUse doesn't fatal on non-ca wikis - 13https://github.com/miraheze/MirahezeMagic/pull/429 [12:19:15] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/37f57dd3e864...3ebbed41a8c7 [12:19:17] [02miraheze/MirahezeMagic] 07paladox 033ebbed4 - Update extension.json [12:19:19] [02MirahezeMagic] 07paladox synchronize pull request 03#429: Make sure VanishUse doesn't fatal on non-ca wikis - 13https://github.com/miraheze/MirahezeMagic/pull/429 [12:21:08] [02miraheze/MirahezeMagic] 07github-actions[bot] pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3ebbed41a8c7...59773163e201 [12:21:11] [02miraheze/MirahezeMagic] 07github-actions 035977316 - CI: lint code to MediaWiki standards [12:21:13] [02MirahezeMagic] 07github-actions[bot] synchronize pull request 03#429: Make sure VanishUse doesn't fatal on non-ca wikis - 13https://github.com/miraheze/MirahezeMagic/pull/429 [12:21:37] [02MirahezeMagic] 07paladox synchronize pull request 03#429: Make sure VanishUse doesn't fatal on non-ca wikis - 13https://github.com/miraheze/MirahezeMagic/pull/429 [12:21:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.94, 3.60, 3.04 [12:21:39] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/59773163e201...e377273df0d8 [12:21:41] [02miraheze/MirahezeMagic] 07paladox 03e377273 - Update SpecialVanishUser.php [12:22:57] [02MirahezeMagic] 07paladox edited pull request 03#429: Make sure VanishUser doesn't fatal on non-ca wikis - 13https://github.com/miraheze/MirahezeMagic/pull/429 [12:26:03] [02MirahezeMagic] 07paladox synchronize pull request 03#429: Make sure VanishUser doesn't fatal on non-ca wikis - 13https://github.com/miraheze/MirahezeMagic/pull/429 [12:26:06] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/e377273df0d8...5b281457028d [12:26:07] [02miraheze/MirahezeMagic] 07paladox 035b28145 - Update SpecialVanishUser.php [12:26:18] miraheze/MirahezeMagic - paladox the build has errored. [12:27:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.59, 3.85, 3.31 [12:31:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.29, 3.57, 3.33 [12:33:24] miraheze/MirahezeMagic - paladox the build has errored. [12:35:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.21, 4.06, 3.55 [12:37:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.75, 3.86, 3.54 [12:39:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.11, 3.88, 3.58 [12:41:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.90, 3.91, 3.63 [12:49:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.22, 3.67, 3.58 [12:53:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:57:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.90, 3.89, 3.80 [13:01:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.02, 2.85, 3.39 [13:03:24] RECOVERY - wikien.wildterra2.com - reverse DNS on sslhost is OK: SSL OK - wikien.wildterra2.com reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:22:32] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:43:34] RECOVERY - wiki.altilunium.my.id - reverse DNS on sslhost is OK: SSL OK - wiki.altilunium.my.id reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:03:05] RECOVERY - wiki.altilunium.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.altilunium.my.id' will expire on Wed 11 Oct 2023 05:49:31 GMT +0000. [14:50:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.20, 2.68 [14:54:36] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.12, 3.13, 2.77 [15:07:17] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.41, 3.36, 3.11 [15:09:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.45, 3.17 [15:11:10] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.84, 2.85, 2.98 [15:18:55] [02MirahezeMagic] 07paladox closed pull request 03#429: Make sure VanishUser doesn't fatal on non-ca wikis - 13https://github.com/miraheze/MirahezeMagic/pull/429 [15:18:56] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/45b6ea886568...bcb70af274ca [15:18:57] [02miraheze/MirahezeMagic] 07paladox 03bcb70af - Make sure VanishUser doesn't fatal on non-ca wikis (#429) [15:25:03] miraheze/MirahezeMagic - paladox the build has errored. [15:25:45] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/65dab8f25ced [15:25:47] [02miraheze/RemovePII] 07paladox 0365dab8f - Make sure RemovePII doesn't fatal on non-ca wikis [15:25:48] [02RemovePII] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/RemovePII [15:25:49] [02RemovePII] 07paladox opened pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:26:09] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/65dab8f25ced...f0c4a3063aab [15:26:12] [02miraheze/RemovePII] 07paladox 03f0c4a30 - Update extension.json [15:26:13] [02RemovePII] 07paladox synchronize pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:27:18] [02RemovePII] 07paladox synchronize pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:27:19] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/f0c4a3063aab...e8a4987f4a5b [15:27:22] [02miraheze/RemovePII] 07paladox 03e8a4987 - Update SpecialRemovePII.php [15:30:13] [02RemovePII] 07paladox synchronize pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:30:16] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/e8a4987f4a5b...37cf7c11f41a [15:30:17] [02miraheze/RemovePII] 07paladox 0337cf7c1 - Update SpecialRemovePII.php [15:30:41] miraheze/RemovePII - paladox the build has errored. [15:33:42] miraheze/RemovePII - paladox the build has errored. [15:35:11] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 [15:35:12] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/MirahezeMagic [15:36:50] [02RemovePII] 07paladox synchronize pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:36:52] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/37cf7c11f41a...87b0a702854e [15:36:55] [02miraheze/RemovePII] 07paladox 0387b0a70 - Update SpecialRemovePII.php [15:37:11] [02RemovePII] 07paladox synchronize pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:37:12] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/87b0a702854e...c7494094f0b5 [15:37:13] [02miraheze/RemovePII] 07paladox 03c749409 - Update SpecialRemovePII.php [15:40:21] miraheze/RemovePII - paladox the build has errored. [15:40:37] [02RemovePII] 07paladox synchronize pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:40:40] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/c7494094f0b5...3c104d7439b8 [15:40:42] [02miraheze/RemovePII] 07paladox 033c104d7 - Update SpecialRemovePII.php [15:42:33] [02RemovePII] 07paladox synchronize pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:42:34] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/3c104d7439b8...e6fbf517be1f [15:42:37] [02miraheze/RemovePII] 07paladox 03e6fbf51 - Update SpecialRemovePII.php [15:42:41] [02RemovePII] 07paladox synchronize pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:42:44] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/e6fbf517be1f...eec078d28d9e [15:42:47] [02miraheze/RemovePII] 07paladox 03eec078d - Update SpecialRemovePII.php [15:48:35] [02RemovePII] 07paladox closed pull request 03#72: Make sure RemovePII doesn't fatal on non-ca wikis - 13https://github.com/miraheze/RemovePII/pull/72 [15:48:38] [02miraheze/RemovePII] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/RemovePII/compare/5dfcaf735be3...b9e1f6848866 [15:48:41] [02miraheze/RemovePII] 07paladox 03b9e1f68 - Make sure RemovePII doesn't fatal on non-ca wikis (#72) [15:48:44] [02miraheze/RemovePII] 07paladox deleted branch 03paladox-patch-2 [15:48:45] [02RemovePII] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/RemovePII [15:48:59] miraheze/RemovePII - paladox the build has errored. [15:49:29] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±2] 13https://github.com/miraheze/mediawiki/compare/0bc5bcb60eb5...dbfb45035e04 [15:49:32] [02miraheze/mediawiki] 07paladox 03dbfb450 - Update MirahezeMagic and RemovePII [15:50:24] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_39 [+0/-0/±2] 13https://github.com/miraheze/mediawiki/compare/c2682b1d1afc...cd3465460bc6 [15:50:27] [02miraheze/mediawiki] 07paladox 03cd34654 - Update MirahezeMagic and RemovePII [15:51:16] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [15:51:19] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [15:51:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:51:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:59] miraheze/RemovePII - paladox the build has errored. [15:56:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4b58d5352f87...488faaacb810 [15:56:46] [02miraheze/mw-config] 07paladox 03488faaa - Unset wan for beta [15:57:19] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [15:57:25] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 5s [15:57:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:32] !log [paladox@test131] starting deploy of {'config': True} to all [15:57:34] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [15:57:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:45] miraheze/mw-config - paladox the build passed. [15:59:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.56, 3.44, 2.98 [16:01:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.91, 2.90, 2.84 [16:02:55] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 695s [16:03:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [16:06:01] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 883s [16:06:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:08:52] !log [@mwtask141] starting deploy of {'config': True} to all [16:08:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:03] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [16:09:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:25:45] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.001423805952 secs [16:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [17:21:34] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [17:52:04] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - with.cpt-ra.bid All nameservers failed to answer the query. [18:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [18:11:45] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1977953911 secs [18:19:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.23, 3.42, 2.89 [18:21:08] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cpt-ra.bid. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:21:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.18, 3.68, 3.05 [18:23:39] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.16, 3.47, 3.04 [18:29:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.78, 3.31 [18:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [18:41:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.34, 3.94, 3.74 [18:50:15] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:53:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.66, 4.12, 3.91 [18:55:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.23, 3.82, 3.83 [18:57:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [19:03:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.83, 2.62, 3.32 [19:06:57] PROBLEM - swiftproxy131 APT on swiftproxy131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:08:58] PROBLEM - swiftproxy131 APT on swiftproxy131 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (6 critical updates). [19:26:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:33:12] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [19:52:16] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query with.cpt-ra.bid. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [20:21:20] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp22.miraheze.org - CNAME OK [20:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:41:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.48, 2.97, 2.33 [20:43:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.26, 2.67, 2.30 [21:23:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:45:53] PROBLEM - ping6 on cp33 is WARNING: PING WARNING - Packet loss = 0%, RTA = 127.16 ms [21:46:04] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 123.63 ms [21:46:57] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 123.26 ms [21:47:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c05851ca26dc [21:48:00] [02miraheze/mw-config] 07paladox 03c05851c - Set wgULSNoImeSelectors [21:48:03] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:48:06] [02mw-config] 07paladox opened pull request 03#5306: Set wgULSNoImeSelectors - 13https://github.com/miraheze/mw-config/pull/5306 [21:48:15] [02mw-config] 07paladox closed pull request 03#5306: Set wgULSNoImeSelectors - 13https://github.com/miraheze/mw-config/pull/5306 [21:48:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/488faaacb810...864862451d8e [21:48:19] [02miraheze/mw-config] 07paladox 038648624 - Set wgULSNoImeSelectors (#5306) [21:48:22] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [21:48:23] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:48:31] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [21:48:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:41] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [21:48:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:55] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [21:48:59] !log [paladox@mwtask141] starting deploy of {'config': True} to all [21:49:02] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [21:49:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:03] !log [paladox@test131] starting deploy of {'config': True} to all [21:49:04] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [21:49:05] miraheze/mw-config - paladox the build passed. [21:49:07] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [21:49:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:10] miraheze/mw-config - paladox the build passed. [21:49:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:32] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:52:54] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 118.93 ms [21:53:52] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 119.04 ms [21:54:13] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 104.11 ms [22:07:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.41, 2.77 [22:09:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.12, 3.25, 2.79 [22:51:31] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [23:04:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 2.01, 4.47, 2.72 [23:04:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.21, 8.95, 4.21 [23:06:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.94, 3.25, 2.49 [23:10:30] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 1.30, 6.91, 5.24 [23:12:30] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.27, 4.67, 4.62 [23:39:45] PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.6818426251 secs