[00:05:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.89, 3.67, 2.82 [00:08:18] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.41, 3.23, 1.75 [00:09:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.14, 3.86, 3.09 [00:11:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.31, 3.38, 3.02 [00:13:25] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.68, 7.14, 4.38 [00:15:25] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.22, 6.05, 4.33 [00:16:10] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.30, 3.46, 2.62 [00:18:08] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.59, 3.19, 2.62 [00:29:25] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.73, 6.31, 4.93 [00:29:55] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.15, 3.53, 2.93 [00:31:25] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.29, 6.97, 5.34 [00:35:25] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.14, 7.38, 5.93 [00:35:51] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.62, 3.77, 3.31 [00:37:25] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.09, 6.18, 5.66 [00:41:50] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.98, 3.35, 3.29 [00:42:40] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 32.72, 17.88, 10.30 [00:46:38] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.38, 11.25, 9.41 [00:48:38] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.73, 9.09, 8.84 [00:48:40] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.15, 3.50, 3.38 [00:50:39] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.50, 3.08, 3.24 [00:54:33] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.80, 4.03, 3.59 [00:55:25] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.60, 5.79, 5.29 [00:56:31] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.22, 3.67, 3.52 [00:57:25] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.00, 5.37, 5.19 [01:02:26] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.13, 3.01, 3.29 [01:31:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 6.54, 5.99 [01:33:00] PROBLEM - wiki.dunmoore.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': None, 'CNAME': '77980.bodis.com.'} [01:33:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.14, 6.02, 5.88 [01:47:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 7.02, 6.40 [01:49:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.22, 6.72, 6.37 [01:53:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.58, 6.80 [01:55:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.32, 6.81 [01:57:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.64, 7.78, 6.99 [01:57:46] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1004966497 secs [01:59:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.03, 6.76, 6.70 [02:01:00] [02mw-config] 07thelevking opened pull request 03#5259: Update isvwiki to CC BY-SA 4.0 - 13https://github.com/miraheze/mw-config/pull/5259 [02:01:58] miraheze/mw-config - thelevking the build passed. [02:26:02] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.dunmoore.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:27:46] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08485949039 secs [02:32:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.73, 6.34 [02:34:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.37, 6.24, 6.21 [02:51:25] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.06, 5.12, 2.68 [02:53:25] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.77, 3.82, 2.51 [03:10:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 6.27, 5.93 [03:12:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.64, 6.00, 5.87 [03:20:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.54, 7.11, 6.51 [03:22:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.68, 6.55, 6.37 [04:26:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.37, 6.62 [04:32:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.41, 6.81 [04:34:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.39, 7.51, 6.93 [04:44:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.60, 7.97, 7.29 [04:46:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.79, 7.79, 7.30 [04:56:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.69, 6.24, 6.79 [05:03:04] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.009 second response time [05:04:39] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 9.56, 10.68, 6.79 [05:06:38] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.93, 8.51, 6.46 [05:08:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.82, 3.59, 2.68 [05:09:56] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:11:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 6.62, 6.66 [05:12:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.77, 2.80, 2.59 [05:14:15] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13304 MB (10% inode=99%); [05:15:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.75, 6.02, 6.43 [05:22:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.83, 7.41, 6.92 [05:28:15] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7878 MB (5% inode=99%); [05:32:47] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.56, 6.51, 6.71 [05:56:27] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:57] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:11:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.16, 3.28, 2.38 [06:24:34] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:26:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 6.79, 6.36 [06:27:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.17, 3.15, 3.94 [06:28:55] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.50, 6.39, 6.27 [06:32:30] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1606714129 secs [06:33:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.80, 1.99, 3.19 [06:43:29] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.79, 3.54, 3.37 [06:45:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.85, 2.94, 3.17 [06:54:30] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.003072679043 secs [06:56:23] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:12:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 6.39, 6.13 [07:15:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.23, 4.19, 2.99 [07:18:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.01, 6.43, 6.28 [07:21:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.95, 3.02, 3.10 [07:21:40] PROBLEM - db112 MariaDB Connections on db112 is WARNING: Warning connection usage: 83.8%Current connections: 419 [07:29:40] PROBLEM - db112 MariaDB Connections on db112 is CRITICAL: Critical connection usage: 90.4%Current connections: 452 [07:38:20] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:42:33] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.074 second response time [07:44:02] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [07:45:32] PROBLEM - db112 MariaDB on db112 is CRITICAL: Too many connections [07:56:30] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1174439788 secs [08:08:14] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [08:08:30] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.08990043402 secs [08:13:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 3.78, 2.74 [08:15:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.50, 2.92, 2.55 [08:54:00] PROBLEM - scum-wiki.com - reverse DNS on sslhost is UNKNOWN: [08:54:00] PROBLEM - wiki.ravynos.com - reverse DNS on sslhost is UNKNOWN: [08:54:00] PROBLEM - mercy.theking1543.eu.org - reverse DNS on sslhost is UNKNOWN: [08:54:00] PROBLEM - www.ambientguitar.org - reverse DNS on sslhost is UNKNOWN: [10:24:07] PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is UNKNOWN: [10:24:09] PROBLEM - ambient.wiki - reverse DNS on sslhost is UNKNOWN: [10:24:10] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is UNKNOWN: [10:54:29] PROBLEM - ambientguitar.org - reverse DNS on sslhost is UNKNOWN: [10:54:30] PROBLEM - wiki.maxus.space - reverse DNS on sslhost is UNKNOWN: [10:54:30] PROBLEM - knowledgebase.clientmanager.co.za - reverse DNS on sslhost is UNKNOWN: [10:54:48] PROBLEM - nonbinary.wiki - reverse DNS on sslhost is UNKNOWN: [12:24:38] PROBLEM - electrowiki.org - reverse DNS on sslhost is UNKNOWN: [12:24:38] PROBLEM - td10.org - reverse DNS on sslhost is UNKNOWN: [12:24:38] PROBLEM - mercy.theking1543.eu.org - reverse DNS on sslhost is UNKNOWN: [12:24:38] PROBLEM - knowledgebase.clientmanager.co.za - reverse DNS on sslhost is UNKNOWN: [12:24:38] PROBLEM - beaconspace.unrestrictedlorefare.com - reverse DNS on sslhost is UNKNOWN: [12:25:54] PROBLEM - www.ambientguitar.org - reverse DNS on sslhost is UNKNOWN: [12:25:55] PROBLEM - studyabroad.wiki - reverse DNS on sslhost is UNKNOWN: [12:25:55] PROBLEM - mcwiki.kirbygang.tk - reverse DNS on sslhost is UNKNOWN: [12:25:55] PROBLEM - wiki.fossbots.org - reverse DNS on sslhost is UNKNOWN: [12:25:55] PROBLEM - wiki.litek.top - reverse DNS on sslhost is UNKNOWN: [12:38:31] Paladox phab is down [12:40:40] Looks like database again [13:27:28] I'm not sure which dbs are on it but db112 is at 100% disk usage [13:27:58] if I had to throw a guess out I'd say that has something to do with Phab not working [13:29:12] between 1 AM and 1:45 AM the server quickly dropped from ~19GB free space to 0 [13:40:33] @nibblypup that is why yes, db112 has all non-MediaWiki databases [13:47:43] We keep deleting binlogs and such but we really need to find a better soloutuon [13:53:37] PROBLEM - knowledgebase.clientmanager.co.za - reverse DNS on sslhost is UNKNOWN: [13:53:39] PROBLEM - nexus.wiki - reverse DNS on sslhost is UNKNOWN: [13:53:41] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is UNKNOWN: [14:18:37] i'm trying to delete bin log files but seems to be slow? [14:18:51] i initially deleted a bin log file so i could run purge bin logs [14:19:14] and ran service mysql restart [14:19:30] but it seems to be taking a long while so i ctrl-c'd and decided to do service mysql stop [14:19:36] and that's taking for ages [14:20:02] i feel like rebooting the server [14:22:39] !log reboot db112 [14:22:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:13] PROBLEM - wiki.dunmoore.net - reverse DNS on sslhost is UNKNOWN: [14:24:13] PROBLEM - wiki.maxus.space - reverse DNS on sslhost is UNKNOWN: [14:35:04] paladox: do we have innodb_fast_shutdown set? [14:35:31] no [14:42:05] hmm it keeps crashing now [14:45:21] https://www.irccloud.com/pastebin/7ZnD7fQY/ [14:46:14] paladox: believe Wikimedia set that to 1 [14:46:28] Also looking at error [14:47:02] Segfault [14:47:03] Really [14:47:07] That shouldn't happen [14:47:15] paladox: is that after a reboot? [14:47:19] Can we reboot again? [14:47:20] yeh [14:47:22] no [14:47:27] rebooting again won't fix it [14:47:33] might be corrupt data. [14:47:47] Segfault from corrupt data would be bad [14:48:11] paladox: does it start with innodb_force_recovery maybe? [14:48:24] Im seeing that its possible that error is related to disk space [14:49:31] nope [14:51:41] works now [14:54:20] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is UNKNOWN: [14:54:21] PROBLEM - nexus.wiki - reverse DNS on sslhost is UNKNOWN: [14:54:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is UNKNOWN: [14:54:21] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is UNKNOWN: [15:01:34] we've got no disk space on db112 tbh even after deleting all the bin logs [15:01:38] 1g left [15:02:43] uh who keeps trying to upload data to it? [15:02:56] it's full now and there's nothing i can do lol [15:03:05] matomo/piwik will have to be reset [15:03:56] https://www.irccloud.com/pastebin/yVuJU2uk/ [15:03:59] Could it be backups? [15:04:11] oh let me check [15:05:29] 27g piwik backup [15:11:02] we've deleted that before [15:17:40] https://github.com/miraheze/puppet/blob/0d2e94ca9c4976db46191f06dbc9d4f0bebb28d8/modules/base/templates/backups/miraheze-backup.py.erb#L114 [15:17:48] the backup script is supposed to delete when done [15:18:19] I wonder if it could be exiting earlier due to a lack of disk space [15:23:12] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 29597 MB (22% inode=99%); [15:24:13] PROBLEM - wiki.litek.top - reverse DNS on sslhost is UNKNOWN: [15:24:15] PROBLEM - wiki.hsdfz.org.cn - reverse DNS on sslhost is UNKNOWN: [15:24:15] PROBLEM - knowledgebase.clientmanager.co.za - reverse DNS on sslhost is UNKNOWN: [15:24:16] PROBLEM - ambientguitar.org - reverse DNS on sslhost is UNKNOWN: [15:24:17] PROBLEM - mercy.theking1543.eu.org - reverse DNS on sslhost is UNKNOWN: [15:24:18] PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is UNKNOWN: [15:24:18] PROBLEM - nexus.wiki - reverse DNS on sslhost is UNKNOWN: [15:42:52] paladox: looks like icinga is stopped [15:44:02] fixed [15:45:27] not icingabot, Backend icinga is not running [15:45:55] backends working for me [15:46:22] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.205 second response time [15:46:25] oh [15:46:36] now its good [15:47:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0d2e94ca9c49...84cbfe57cb59 [15:47:26] [02miraheze/puppet] 07paladox 0384cbfe5 - Revert "mon141: increase MaxConcurrentChecks to 80" [15:47:32] PROBLEM - redkrayolawiki.org - reverse DNS on sslhost is UNKNOWN: [15:47:33] PROBLEM - wiki.litek.top - reverse DNS on sslhost is UNKNOWN: PROBLEM - electrowiki.org - reverse DNS on sslhost is UNKNOWN: PROBLEM - mcwiki.kirbygang.tk - reverse DNS on sslhost is UNKNOWN: rDNS WARNING - reverse DNS entry for mcwiki.kirbygang.tk could not be found PROBLEM - mirawww.glitchcity.wiki - reve [15:47:33] NS on sslhost is UNKNOWN: [15:47:35] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.328 second response time [15:48:39] also the relaybot seems to be restarting pretty regularly [15:48:47] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:48:48] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 29517 MB (22% inode=99%); [15:49:12] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.265 second response time [15:49:28] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.156 second response time [15:49:46] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20261 bytes in 0.271 second response time [15:49:54] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 1.4%Current connections: 7 [15:49:59] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6281 bytes in 5.562 second response time [15:50:10] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 3519 Threads: 8 Questions: 1417134 Slow queries: 16758 Opens: 677 Open tables: 671 Queries per second avg: 402.709 [15:51:49] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:52:03] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 9.96, 23.12, 15.05 [15:55:44] PROBLEM - db112 NTP time on db112 is WARNING: NTP WARNING: Offset -0.1054427326 secs [19:08:13] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.74, 3.51, 1.93 [19:12:13] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.22, 3.00, 2.15 [19:18:00] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address mrlove.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:19:04] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:28:10] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1476386786 secs [19:36:10] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08179920912 secs [19:43:50] PROBLEM - mrlove.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mrlove.wiki could not be found [19:47:59] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [20:12:10] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1104727089 secs [20:16:10] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09199258685 secs [20:17:42] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2076633871 secs [20:19:42] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.001269519329 secs [20:51:26] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.23, 3.22, 2.29 [20:55:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.27, 3.73, 2.76 [20:57:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.20, 2.78, 2.53 [21:07:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.38, 3.97, 2.57 [21:12:10] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1106944084 secs [21:13:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.03, 3.76, 3.00 [21:15:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.77, 3.21, 2.91 [21:18:10] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.08771169186 secs [21:24:28] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/df9f3e0cbea1...4a2d8941d2e9 [21:24:31] [02miraheze/mw-config] 07AgentIsai 034a2d894 - Disable Swift outage sitenotice [21:25:28] miraheze/mw-config - AgentIsai the build passed. [21:27:29] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:57:17] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:10:17] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.33, 4.02, 2.95 [22:11:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 3.91, 2.72 [22:13:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.42, 3.39, 2.67 [22:14:13] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.46, 3.99, 3.20 [22:16:13] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.87, 3.32, 3.06 [22:27:06] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [22:39:44] PROBLEM - chakra.moe - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for chakra.moe could not be found [22:51:53] PROBLEM - chakra.moe - LetsEncrypt on sslhost is CRITICAL: connect to address chakra.moe and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:44:13] PROBLEM - mw142 SSH on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:12] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:46:09] RECOVERY - mw142 SSH on mw142 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:46:11] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 19.06, 21.91, 12.59 [23:46:47] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw142 [23:47:11] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.289 second response time [23:47:31] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw142 [23:47:33] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw142 [23:47:47] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw142 [23:48:47] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 15 backends are healthy [23:49:31] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 15 backends are healthy [23:49:33] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [23:49:42] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1241922677 secs [23:49:47] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 15 backends are healthy [23:50:08] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 3.45, 11.00, 10.17 [23:52:07] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.88, 8.69, 9.41 [23:55:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 3.84, 2.76 [23:57:09] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.07, 3.73, 2.87 [23:59:09] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.44, 3.11, 2.74