[00:00:23] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:01:18] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:23] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:03:18] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:03] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21441MiB (24% inode=98%); [00:05:06] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 19222MiB (20% inode=98%); [00:05:40] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21445MiB (24% inode=98%); [00:41:33] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:38:04] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:03:48] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [03:31:53] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:02:09] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:30:14] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [05:46:13] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.74 ms [05:48:14] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.77 ms [06:26:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/55c70176a0e0...ea55b182bd0e [06:26:08] [02miraheze/mw-config] 07Universal-Omega 03ea55b18 - Set wgGalleryOptions for dccomicswiki [06:26:58] miraheze/mw-config - Universal-Omega the build passed. [06:30:09] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5d959e26aadd...368aec5f10b6 [06:30:12] [02miraheze/dns] 07Universal-Omega 03368aec5 - Add dccomics [06:32:04] !log [@mwtask181] starting deploy of {'config': True} to all [06:32:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:32:13] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [06:32:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:09] !log [@mwtask171] starting deploy of {'config': True} to all [06:37:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:18] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [06:37:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:38] !log [@test151] starting deploy of {'config': True} to test151 [06:53:39] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [06:53:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:53:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:22:00] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:34:16] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 167.29 ms [09:36:17] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.73 ms [10:12:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 173.76 ms [10:14:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.63 ms [11:14:51] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.24 ms [11:16:38] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=fictionwiki (END - exit=0) [11:16:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:16:52] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 177.28 ms [12:28:19] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:45:31] PROBLEM - wiki.geoparkcorumbatai.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.geoparkcorumbatai.com.br' expires in 15 day(s) (Tue 23 Apr 2024 12:25:43 PM GMT +0000). [12:45:49] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5717daffa306...234373899cb2 [12:45:51] [02miraheze/ssl] 07WikiTideSSLBot 032343738 - Bot: Update SSL cert for wiki.geoparkcorumbatai.com.br [12:57:39] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:03:26] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/varnishlog_logs] [13:31:26] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:32] RECOVERY - wiki.geoparkcorumbatai.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.geoparkcorumbatai.com.br' will expire on Sat 06 Jul 2024 11:45:43 AM GMT +0000. [14:06:45] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.88 ms [14:08:46] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.19 ms [14:27:09] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.90 ms [14:29:10] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.37 ms [14:49:21] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.56, 14.90, 9.16 [14:49:42] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.30, 19.59, 11.71 [14:49:56] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.91, 17.16, 10.24 [14:50:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.17, 19.53, 11.83 [14:50:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.63, 19.65, 11.98 [14:50:31] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.27, 20.29, 12.01 [14:51:06] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10213MiB (10% inode=98%); [14:51:21] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.87, 16.72, 10.57 [14:51:56] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.02, 16.50, 10.88 [14:52:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.54, 20.61, 13.21 [14:52:17] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.53, 19.06, 12.78 [14:52:31] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.21, 18.97, 12.60 [14:53:42] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 11.80, 18.71, 13.47 [14:54:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 9.28, 16.21, 12.49 [15:08:17] PROBLEM - ns2 NTP time on ns2 is WARNING: NTP WARNING: Offset -0.4371784925 secs [15:14:22] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:16:20] PROBLEM - ns2 NTP time on ns2 is WARNING: NTP WARNING: Offset -0.1486296952 secs [15:22:15] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.002556145191 secs [15:34:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.05, 18.03, 11.19 [15:34:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.64, 19.14, 11.78 [15:34:18] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.45, 17.82, 10.66 [15:34:29] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 31.00, 21.27, 13.11 [15:34:31] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.98, 19.58, 11.91 [15:35:21] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.95, 19.65, 11.74 [15:35:27] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.76, 18.59, 11.52 [15:35:56] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.29, 21.35, 12.85 [15:36:22] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.55, 20.95, 14.01 [15:36:31] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.47, 18.60, 12.46 [15:37:21] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.17, 20.41, 13.01 [15:37:27] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 20.25, 18.76, 12.43 [15:37:50] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw171 [15:38:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.98, 19.59, 13.64 [15:38:10] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 43% [15:38:16] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 30.10, 24.22, 16.00 [15:38:16] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw171 [15:38:47] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw151 [15:39:21] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.82, 23.54, 15.06 [15:39:50] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [15:40:08] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 35% [15:40:16] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [15:40:31] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 22.95, 15.56 [15:40:47] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [15:41:27] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 23.72, 16.01 [15:41:56] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.86, 23.80, 17.03 [15:42:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.28, 24.51, 17.07 [15:42:18] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.24, 23.63, 16.96 [15:42:31] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.59, 23.36, 16.63 [15:43:21] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 14.18, 20.97, 16.19 [15:43:27] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.80, 21.48, 16.15 [15:43:57] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.66, 22.83, 18.14 [15:44:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.70, 21.45, 16.86 [15:44:17] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.51, 22.93, 18.09 [15:44:31] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.93, 19.86, 16.15 [15:45:21] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 13.91, 18.74, 15.96 [15:45:27] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.02, 18.41, 15.69 [15:45:50] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.82, 20.04, 17.67 [15:45:56] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 10.94, 17.60, 16.14 [15:46:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.46, 19.45, 16.73 [15:46:17] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.45, 18.98, 17.26 [15:46:18] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 10.21, 17.12, 15.87 [15:47:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.46 ms [15:51:28] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.01 ms [16:09:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.77 ms [16:11:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.29 ms [16:17:42] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.68, 18.98, 12.77 [16:18:31] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.20, 17.52, 12.11 [16:19:42] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.16, 21.01, 14.29 [16:20:31] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.31, 18.00, 12.98 [16:21:42] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.46, 19.98, 14.71 [16:28:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.84, 20.41, 15.49 [16:28:31] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.03, 22.57, 16.68 [16:29:27] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.90, 17.99, 13.50 [16:29:42] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.62, 24.47, 18.40 [16:30:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.16, 22.86, 16.53 [16:31:27] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.86, 17.91, 14.00 [16:32:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.46, 21.85, 16.97 [16:32:17] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.41, 22.26, 17.40 [16:32:31] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.69, 23.87, 18.71 [16:34:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.52, 20.33, 17.00 [16:34:17] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.57, 20.06, 17.20 [16:35:42] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.79, 23.24, 20.22 [16:38:31] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.03, 18.32, 18.03 [16:39:42] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 8.88, 17.15, 18.55 [16:56:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03redis [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9d390fd8e2f6 [16:56:03] [02miraheze/puppet] 07Universal-Omega 039d390fd - mwtask181: use redis jobqueue again [16:56:04] [02puppet] 07Universal-Omega created branch 03redis - 13https://github.com/miraheze/puppet [16:56:07] [02puppet] 07Universal-Omega opened pull request 03#3828: mwtask181: use redis jobqueue again - 13https://github.com/miraheze/puppet/pull/3828 [16:56:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03redis [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9d390fd8e2f6...5ac0a0a11a57 [16:56:44] [02miraheze/puppet] 07Universal-Omega 035ac0a0a - Update [16:56:46] [02puppet] 07Universal-Omega synchronize pull request 03#3828: mwtask181: use redis jobqueue again - 13https://github.com/miraheze/puppet/pull/3828 [16:57:29] [02puppet] 07Universal-Omega closed pull request 03#3828: mwtask181: use redis jobqueue again - 13https://github.com/miraheze/puppet/pull/3828 [16:57:31] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/fa8f52a9507d...b824290b0334 [16:57:34] [02miraheze/puppet] 07Universal-Omega 03b824290 - mwtask181: use redis jobqueue again (#3828) [16:57:36] [02puppet] 07Universal-Omega deleted branch 03redis - 13https://github.com/miraheze/puppet [16:57:39] [02miraheze/puppet] 07Universal-Omega deleted branch 03redis [16:58:25] [02mw-config] 07Universal-Omega synchronize pull request 03#5523: Switch some job types back to redis temporarily - 13https://github.com/miraheze/mw-config/pull/5523 [16:58:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobs [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5253cecc17b9...2a7eed0a6c95 [16:58:28] [02miraheze/mw-config] 07Universal-Omega 032a7eed0 - - [16:59:24] miraheze/mw-config - Universal-Omega the build passed. [17:04:03] [02mw-config] 07Universal-Omega synchronize pull request 03#5523: Switch some job types back to redis temporarily - 13https://github.com/miraheze/mw-config/pull/5523 [17:04:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobs [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2a7eed0a6c95...efb97a806c2f [17:04:09] [02miraheze/mw-config] 07Universal-Omega 03efb97a8 - - [17:04:30] [02mw-config] 07Universal-Omega closed pull request 03#5523: Switch some job types back to redis temporarily - 13https://github.com/miraheze/mw-config/pull/5523 [17:04:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ea55b182bd0e...a817b8dc3889 [17:04:34] [02miraheze/mw-config] 07Universal-Omega 03a817b8d - Switch some job types back to redis temporarily (#5523) [17:04:36] [02miraheze/mw-config] 07Universal-Omega deleted branch 03jobs [17:04:37] [02mw-config] 07Universal-Omega deleted branch 03jobs - 13https://github.com/miraheze/mw-config [17:04:42] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:59] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 16s [17:05:02] miraheze/mw-config - Universal-Omega the build passed. [17:05:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:20] miraheze/mw-config - Universal-Omega the build passed. [17:06:56] !log [@mwtask171] starting deploy of {'config': True} to all [17:07:12] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 16s [17:07:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:42] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.52, 18.86, 13.29 [17:08:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.14, 17.50, 11.97 [17:08:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.46, 18.32, 12.46 [17:08:31] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.38, 18.92, 13.01 [17:09:21] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 17.53, 11.46 [17:09:56] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.71, 20.34, 12.97 [17:10:18] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.43, 16.71, 11.17 [17:11:05] [02mediawiki-repos] 07pixDeVl opened pull request 03#14: Add MediaSpoiler (T11868) - 13https://github.com/miraheze/mediawiki-repos/pull/14 [17:11:21] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.44, 18.41, 12.56 [17:11:56] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.10, 19.37, 13.52 [17:12:18] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.90, 16.29, 11.65 [17:15:27] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.76, 21.29, 15.07 [17:15:56] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.23, 23.81, 16.64 [17:18:06] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.66, 24.35, 17.21 [17:19:07] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 21.05, 15.28 [17:20:04] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 23.81, 17.80 [17:21:06] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.26, 19.44, 15.43 [17:21:21] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 14.91, 22.54, 18.57 [17:21:58] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 9.21, 19.22, 16.90 [17:22:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 9.92, 20.21, 17.43 [17:22:28] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 8.60, 21.44, 19.93 [17:22:46] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 7.58, 20.13, 19.36 [17:22:50] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 6.77, 20.07, 19.34 [17:22:58] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [17:23:09] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 8.54, 20.55, 20.52 [17:23:16] !log [@test151] starting deploy of {'config': True} to test151 [17:23:17] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:23:21] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 5.02, 16.22, 16.73 [17:23:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:27] PROBLEM - cp41 Varnish Backends on cp41 is WARNING: No backends detected. If this is an error, see readme.txt [17:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:41] PROBLEM - cp41 Puppet on cp41 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [17:23:43] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [17:24:27] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 4.36, 15.61, 17.98 [17:25:04] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 5.46, 15.61, 18.72 [17:35:50] [02mw-config] 07pixDeVl opened pull request 03#5533: Add MediaSpoiler (T11868) - 13https://github.com/miraheze/mw-config/pull/5533 [17:36:44] miraheze/mw-config - pixDeVl the build passed. [18:29:23] !log [reception@mwtask181] starting deploy of {'l10n': True, 'lang': 'en', 'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [18:29:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:44] !log [reception@mwtask181] finished deploy of {'l10n': True, 'lang': 'en', 'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 21s [18:29:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:05:56] [02mw-config] 07redbluegreenhat closed pull request 03#5532: T12010: Update user rights - 13https://github.com/miraheze/mw-config/pull/5532 [19:05:59] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a817b8dc3889...d1af0a3ea0b7 [19:06:01] [02miraheze/mw-config] 07redbluegreenhat 03d1af0a3 - T12010: Update user rights (#5532) [19:06:50] miraheze/mw-config - redbluegreenhat the build passed. [19:06:55] !log [@mwtask171] starting deploy of {'config': True} to all [19:06:56] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:07:03] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [19:07:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:05] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [19:07:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:38] [02CreateWiki] 07Universal-Omega closed pull request 03#463: Support additional subdomains - 13https://github.com/miraheze/CreateWiki/pull/463 [19:18:40] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03subdomains [19:18:42] [02CreateWiki] 07Universal-Omega deleted branch 03subdomains - 13https://github.com/miraheze/CreateWiki [19:20:12] [02mediawiki-repos] 07Universal-Omega synchronize pull request 03#14: Add MediaSpoiler (T11868) - 13https://github.com/miraheze/mediawiki-repos/pull/14 [19:22:25] [02mediawiki-repos] 07Universal-Omega reviewed pull request 03#14 commit - 13https://github.com/miraheze/mediawiki-repos/pull/14#discussion_r1555046510 [19:23:29] !log [@test151] starting deploy of {'config': True} to test151 [19:23:30] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [19:23:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:40] [02mediawiki-repos] 07pixDeVl synchronize pull request 03#14: Add MediaSpoiler (T11868) - 13https://github.com/miraheze/mediawiki-repos/pull/14 [19:36:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 167.62 ms [19:38:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.47 ms [19:50:26] !log destroyed a user on phorge [19:50:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:55] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:41:50] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [21:12:06] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9701MiB (10% inode=98%); [21:14:58] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9695MiB (10% inode=98%); [21:30:57] [02mediawiki-repos] 07Universal-Omega closed pull request 03#14: Add MediaSpoiler (T11868) - 13https://github.com/miraheze/mediawiki-repos/pull/14 [21:31:00] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/df0067975140...1379e1b10249 [21:31:03] [02miraheze/mediawiki-repos] 07pixDeVl 031379e1b - Add MediaSpoiler (T11868) (#14) [21:38:09] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [21:38:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:38:14] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3613 bytes in 0.830 second response time [21:38:29] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 4% [22:13:20] PROBLEM - wiki.mhe.my.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mhe.my.id' expires in 15 day(s) (Tue 23 Apr 2024 09:58:38 PM GMT +0000). [22:13:30] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/234373899cb2...2576b6d63161 [22:13:32] [02miraheze/ssl] 07WikiTideSSLBot 032576b6d - Bot: Update SSL cert for wiki.mhe.my.id [22:14:06] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 30.65, 15.66, 6.28 [22:16:05] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 5.16, 10.76, 5.61 [22:16:51] man, that looks so strange out of context lol [22:17:36] nope looks strange with [22:17:52] fair [22:18:04] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 1.07, 7.43, 4.99 [22:27:28] PROBLEM - wiki.gamergeeked.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gamergeeked.us' expires in 15 day(s) (Tue 23 Apr 2024 09:59:09 PM GMT +0000). [22:27:38] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2576b6d63161...df5a0f97cba5 [22:27:39] [02miraheze/ssl] 07WikiTideSSLBot 03df5a0f9 - Bot: Update SSL cert for wiki.gamergeeked.us [22:43:09] RECOVERY - wiki.mhe.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mhe.my.id' will expire on Sat 06 Jul 2024 09:13:25 PM GMT +0000. [22:48:19] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 61.42, 24.63, 9.99 [22:51:12] PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:52:04] PROBLEM - db151 Puppet on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:52:56] PROBLEM - db151 SSH on db151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:56] RECOVERY - db151 SSH on db151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:56:57] RECOVERY - wiki.gamergeeked.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gamergeeked.us' will expire on Sat 06 Jul 2024 09:27:33 PM GMT +0000. [22:57:41] RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 5.801 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [22:58:03] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [23:01:54] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 25.41, 13.59, 5.57 [23:02:02] Uh what’s the link for Mirabeta again [23:02:12] Mirabeta.org gives me a no such wiki err [23:02:28] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:04:25] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.213 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [23:07:13] should be https://meta.mirabeta.org/ but it doesn't seem like it wants to exist right now [23:08:24] Oh, eu odeio Beta [23:09:14] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:10:03] PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:11:11] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.206 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [23:12:04] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:13:36] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:15:37] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:24:50] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.28, 4.19, 11.95 [23:28:49] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.18, 2.00, 9.31 [23:40:20] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 0.37, 0.79, 11.38 [23:42:19] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.14, 0.57, 10.01 [23:43:31] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [23:57:50] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK