[00:01:19] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:03:19] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:05:03] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21572MiB (24% inode=98%); [00:05:15] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21596MiB (24% inode=98%); [01:57:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59% [02:01:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [02:03:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59% [02:34:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [02:42:09] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:00:03] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 2 seconds old and 0 bytes [03:04:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:30:13] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.70 ms [03:36:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.78 ms [03:40:50] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:33:19] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 49% [04:41:53] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.93 ms [04:43:56] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.03 ms [04:47:13] [02MirahezeMagic] 07Dringsim synchronize pull request 03#481: Remove 'forcesafemode' option - 13https://github.com/miraheze/MirahezeMagic/pull/481 [04:47:58] [02MirahezeMagic] 07Dringsim synchronize pull request 03#481: Remove 'forcesafemode' option - 13https://github.com/miraheze/MirahezeMagic/pull/481 [04:48:52] [02MirahezeMagic] 07Dringsim synchronize pull request 03#481: Remove 'forcesafemode' option - 13https://github.com/miraheze/MirahezeMagic/pull/481 [05:11:59] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.54 ms [05:16:06] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 206.13 ms [05:17:12] miraheze/MirahezeMagic - Dringsim the build passed. [05:29:29] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.21 ms [05:29:39] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 171.76 ms [05:31:32] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 177.86 ms [05:31:43] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.51 ms [05:55:17] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1064421535 secs [05:59:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.24 ms [06:01:50] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.19 ms [06:40:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.42 ms [06:48:42] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.06 ms [06:49:00] PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:49:25] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 68.92, 33.89, 13.91 [06:51:59] PROBLEM - db151 SSH on db151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:14] PROBLEM - db151 Puppet on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:55:27] RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 0.198 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:56:03] RECOVERY - db151 SSH on db151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [06:57:19] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:21] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.83 ms [07:12:28] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 180.74 ms [07:13:24] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 0.15, 2.00, 11.06 [07:13:37] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.01 ms [07:15:22] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.17, 1.40, 9.74 [07:17:45] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.33 ms [07:37:14] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.79 ms [07:41:21] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.04 ms [07:43:18] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.08574780822 secs [07:47:18] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1035134792 secs [07:53:17] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09884545207 secs [08:01:04] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 180.51 ms [08:03:07] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 191.31 ms [08:14:32] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 187.29 ms [08:16:34] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 192.83 ms [08:19:17] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1041163802 secs [08:27:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 200.10 ms [08:35:40] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 215.17 ms [08:39:56] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 180.61 ms [08:42:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 183.18 ms [08:44:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 191.40 ms [08:46:25] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 183.14 ms [08:49:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.07 ms [08:51:17] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09962055087 secs [08:51:27] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.47 ms [09:05:02] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.79 ms [09:07:05] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.91 ms [09:25:49] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 178.89 ms [09:27:51] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 181.47 ms [09:32:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 172.81 ms [09:34:18] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1069344282 secs [09:36:10] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.61 ms [09:40:15] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09988677502 secs [09:43:34] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.86 ms [09:44:12] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1027359962 secs [09:45:37] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 177.97 ms [09:53:48] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.97 ms [09:55:50] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.91 ms [10:04:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.88 ms [10:06:28] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 177.13 ms [10:10:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.00 ms [10:14:30] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.85 ms [10:15:53] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.13 ms [10:20:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.60 ms [10:32:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.88 ms [10:38:40] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 177.72 ms [10:42:26] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 176.51 ms [10:44:29] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 177.75 ms [10:51:43] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09977787733 secs [11:00:58] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.61 ms [11:05:05] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.87 ms [11:32:56] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.36 ms [11:34:59] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 177.50 ms [11:35:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/357a3cd74be8...0c65c373b8d0 [11:35:39] [02miraheze/ssl] 07MirahezeSSLBot 030c65c37 - Bot: Add SSL cert for drachronicles.voxel.gay [11:35:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/0c65c373b8d0...e1ba917169ac [11:35:54] [02miraheze/ssl] 07MirahezeSSLBot 03e1ba917 - Bot: Add SSL cert for wiki.thelastdimension.net [11:37:19] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [11:39:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [11:57:13] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.71 ms [11:57:41] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 189.04 ms [11:59:12] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.27 ms [12:07:54] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 182.32 ms [12:13:17] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.37 ms [12:14:42] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1017638743 secs [12:15:16] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.12 ms [12:19:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.51 ms [12:20:11] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 181.82 ms [12:24:14] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 185.33 ms [12:28:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 184.15 ms [12:33:25] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.11 ms [12:36:32] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 186.21 ms [12:48:41] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.13 ms [13:00:41] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 187.27 ms [13:04:48] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 186.97 ms [13:08:47] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.45 ms [13:18:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 168.96 ms [13:19:15] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 182.60 ms [13:24:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.55 ms [13:35:35] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 183.86 ms [13:46:50] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.74 ms [13:48:52] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 185.79 ms [13:52:24] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09941253066 secs [14:06:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 189.25 ms [14:08:24] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 192.49 ms [14:08:30] [02dns] 07MacFan4000 opened pull request 03#496: Add pso2ngs.wiki zone - 13https://github.com/miraheze/dns/pull/496 [14:11:20] [02dns] 07Reception123 closed pull request 03#496: Add pso2ngs.wiki zone - 13https://github.com/miraheze/dns/pull/496 [14:11:21] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/1914cc3906b4...7a006f80a8ab [14:11:22] [02miraheze/dns] 07MacFan4000 037a006f8 - Add pso2ngs.wiki zone (#496) [14:16:45] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 187.95 ms [14:20:27] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=greatphermesiawiki /home/macfan/images --search-recursively (START) [14:20:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/e1ba917169ac...e82d1c017bba [14:23:08] [02miraheze/ssl] 07MirahezeSSLBot 03e82d1c0 - Bot: Add SSL cert for pso2ngs.wiki [14:24:58] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 186.32 ms [14:27:36] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=greatphermesiawiki /home/macfan/images --search-recursively (END - exit=0) [14:27:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:07] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 186.86 ms [14:33:08] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 184.51 ms [14:34:01] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw162 [14:47:19] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 194.13 ms [14:59:34] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 184.77 ms [15:03:44] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 189.32 ms [15:05:45] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 188.07 ms [15:12:01] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1014738083 secs [15:19:32] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.79 ms [15:21:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.40 ms [15:29:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.16 ms [15:31:17] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 193.81 ms [15:31:57] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.07 ms [15:33:19] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 186.78 ms [15:45:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09983605146 secs [15:51:28] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1009265482 secs [15:52:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.40 ms [15:57:22] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09913644195 secs [15:58:04] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.51 ms [16:00:04] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 181.23 ms [16:00:11] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/eval.php --wiki=metawikibeta (END - exit=2) [16:00:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:53] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 162.91 ms [16:10:09] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1066974998 secs [16:11:31] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 184.50 ms [16:13:32] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 191.21 ms [16:17:30] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.90 ms [16:17:43] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 182.35 ms [16:21:47] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 184.60 ms [16:25:48] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.80 ms [16:45:25] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 181.60 ms [16:49:28] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.95 ms [17:00:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.34 ms [17:01:27] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09732231498 secs [17:04:54] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.54 ms [17:13:48] [02ImportDump] 07Universal-Omega created branch 03exception - 13https://github.com/miraheze/ImportDump [17:13:50] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03exception [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/3455e5cf0747 [17:13:51] [02miraheze/ImportDump] 07Universal-Omega 033455e5c - ImportDumpJob: include exception code and class in failure message [17:13:54] [02ImportDump] 07Universal-Omega opened pull request 03#86: ImportDumpJob: include exception code and class in failure message - 13https://github.com/miraheze/ImportDump/pull/86 [17:19:28] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.36 ms [17:21:31] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.76 ms [17:21:46] miraheze/ImportDump - Universal-Omega the build passed. [17:27:01] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1009908915 secs [17:34:03] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 165.63 ms [17:36:06] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.02 ms [17:39:15] [02ImportDump] 07Universal-Omega synchronize pull request 03#86: ImportDumpJob: include exception code and class in failure message - 13https://github.com/miraheze/ImportDump/pull/86 [17:39:17] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03exception [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/3455e5cf0747...04f5c367ef0d [17:39:20] [02miraheze/ImportDump] 07Universal-Omega 0304f5c36 - Use MWExceptionHandler::getLogMessage [17:39:44] [02ImportDump] 07Universal-Omega edited pull request 03#86: ImportDumpJob: include more exception details in failure message - 13https://github.com/miraheze/ImportDump/pull/86 [17:47:08] miraheze/ImportDump - Universal-Omega the build passed. [17:52:37] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09991008043 secs [17:53:56] [02ImportDump] 07Universal-Omega synchronize pull request 03#86: ImportDumpJob: include more exception details in failure message - 13https://github.com/miraheze/ImportDump/pull/86 [17:53:57] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03exception [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/04f5c367ef0d...e1f40f8732df [17:54:00] [02miraheze/ImportDump] 07Universal-Omega 03e1f40f8 - logException also [18:01:46] miraheze/ImportDump - Universal-Omega the build passed. [18:11:06] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 183.04 ms [18:15:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.38 ms [18:28:58] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [18:36:03] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.39 ms [18:38:06] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.15 ms [18:58:57] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 170.72 ms [19:01:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.52 ms [19:28:00] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [19:34:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 183.84 ms [19:36:30] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.32 ms [19:45:25] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40 [19:46:50] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40 [19:48:28] miraheze/RequestSSL - redbluegreenhat the build has errored. [19:50:58] miraheze/RequestSSL - redbluegreenhat the build has errored. [19:53:01] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40 [19:56:12] miraheze/RequestSSL - redbluegreenhat the build has errored. [20:09:32] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.27 ms [20:11:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.37 ms [20:27:29] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1030763686 secs [20:43:14] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09824970365 secs [21:15:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [21:15:43] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.73 ms [21:19:51] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.41 ms [21:32:55] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:45:18] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:01:08] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:20:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [22:43:31] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9707MiB (10% inode=98%); [22:46:33] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9703MiB (10% inode=98%); [22:47:29] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1016180217 secs [23:00:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [23:03:14] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09959578514 secs [23:24:14] [02ImportDump] 07Universal-Omega closed pull request 03#86: ImportDumpJob: include more exception details in failure message - 13https://github.com/miraheze/ImportDump/pull/86 [23:24:53] [02ImportDump] 07Universal-Omega deleted branch 03exception - 13https://github.com/miraheze/ImportDump [23:25:35] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03exception [23:25:36] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e51bca697b9c...398ab2dd107d [23:25:37] [02miraheze/ImportDump] 07Universal-Omega 03398ab2d - ImportDumpJob: include more exception details in failure message (#86) [23:33:22] miraheze/ImportDump - Universal-Omega the build passed.