[00:00:30] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:00:58] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.39, 1.87, 1.75 [00:01:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09848946333 secs [00:02:30] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 50% [00:04:30] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [00:04:56] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.57, 1.75, 1.73 [00:05:15] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21479MiB (24% inode=98%); [00:05:31] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21465MiB (24% inode=98%); [00:05:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1004421413 secs [00:08:55] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.30, 1.96, 1.81 [00:10:55] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.57, 1.99, 1.85 [00:11:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09989258647 secs [00:12:30] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 58% [00:12:54] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.09, 1.89, 1.82 [00:14:30] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [00:14:53] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.72, 1.49, 1.69 [00:15:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1003085375 secs [00:20:29] !log db171: MariaDB [metawiki]> ALTER TABLE import_requests MODIFY request_status ENUM('complete', 'declined', 'failed', 'inprogress', 'pending', 'starting') NOT NULL; [00:20:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:48] !log db161: MariaDB [metawikibeta]> ALTER TABLE import_requests MODIFY request_status ENUM('complete', 'declined', 'failed', 'inprogress', 'pending', 'starting') NOT NULL; [00:20:51] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 3.03, 2.07, 1.81 [00:20:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:20] [02ImportDump] 07Universal-Omega closed pull request 03#88: Add starting status to prevent conflict - 13https://github.com/miraheze/ImportDump/pull/88 [00:21:21] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03starting [00:21:23] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±7] 13https://github.com/miraheze/ImportDump/compare/b611336eb2ab...24079c459a5c [00:21:26] [02miraheze/ImportDump] 07Universal-Omega 0324079c4 - Add starting status to prevent conflict (#88) [00:21:28] [02ImportDump] 07Universal-Omega deleted branch 03starting - 13https://github.com/miraheze/ImportDump [00:21:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09895053506 secs [00:21:58] !log [universalomega@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to all [00:22:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:12] !log [universalomega@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to all - SUCCESS in 14s [00:22:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:24] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 4% [00:24:57] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41'} to all [00:25:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:45] miraheze/ImportDump - Universal-Omega the build passed. [00:31:37] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41'} to all - SUCCESS in 399s [00:31:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:50] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/24079c459a5c...66a1b9585095 [00:33:53] [02miraheze/ImportDump] 07Universal-Omega 0366a1b95 - Disable decline button if status is already declined [00:34:05] !log [universalomega@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to all [00:34:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:16] !log [universalomega@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to all - SUCCESS in 11s [00:34:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:42] [02miraheze/ImportDump] 07Universal-Omega pushed 0310 commits to 03renamespace [+0/-0/±48] 13https://github.com/miraheze/ImportDump/compare/47292fb62f8f...49f41b812e14 [00:39:44] [02miraheze/ImportDump] 07Universal-Omega 0349f41b8 - Merge branch 'master' into renamespace [00:39:46] [02ImportDump] 07Universal-Omega synchronize pull request 03#85: Renamespace under Miraheze\RequestImport - 13https://github.com/miraheze/ImportDump/pull/85 [00:40:45] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 0.72, 1.33, 1.86 [00:41:48] miraheze/ImportDump - Universal-Omega the build passed. [00:44:44] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.01, 1.16, 1.66 [00:47:33] miraheze/ImportDump - Universal-Omega the build passed. [00:56:13] PROBLEM - wiki.manos.pt - reverse DNS on sslhost is WARNING: NXDOMAIN: The DNS query name does not exist: manos.pt. [01:11:37] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.89, 1.90, 1.53 [01:13:36] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.85, 2.15, 1.65 [01:21:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1000181139 secs [01:29:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09928399324 secs [01:31:17] PROBLEM - mem151 Current Load on mem151 is CRITICAL: LOAD CRITICAL - total load average: 8.56, 5.02, 2.23 [01:33:14] PROBLEM - mem151 Current Load on mem151 is WARNING: LOAD WARNING - total load average: 1.37, 3.44, 1.98 [01:35:14] RECOVERY - mem151 Current Load on mem151 is OK: LOAD OK - total load average: 0.23, 2.32, 1.75 [01:36:09] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [01:36:31] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [01:36:34] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [01:38:09] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [01:38:31] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19090 bytes in 0.068 second response time [01:38:34] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17717 bytes in 0.335 second response time [01:41:31] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.15, 1.53, 1.96 [01:47:30] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.44, 0.99, 1.59 [02:18:20] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.60, 1.94, 1.47 [02:23:06] RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: SSL OK - wiki.minkyu.kim reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:28:17] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.48, 1.94, 1.75 [02:30:16] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.09, 1.66, 1.67 [02:40:14] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.85, 1.84, 1.71 [02:42:13] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.25, 1.56, 1.63 [02:46:11] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.72, 1.78, 1.71 [02:48:14] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 3.01, 2.10, 1.82 [02:52:14] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.80, 1.97, 1.85 [02:54:13] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 1.98, 2.02, 1.88 [02:56:12] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.05, 1.71, 1.78 [03:00:11] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.53, 1.90, 1.83 [03:02:11] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.30, 1.66, 1.74 [03:06:09] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.53, 1.22, 1.55 [03:10:07] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.99, 1.76, 1.70 [03:12:07] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.38, 1.61, 1.65 [03:22:04] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.68, 1.75, 1.66 [03:24:04] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.40, 1.64, 1.62 [03:47:57] PROBLEM - mem161 Current Load on mem161 is CRITICAL: LOAD CRITICAL - total load average: 6.44, 3.49, 1.46 [03:49:54] PROBLEM - mem161 Current Load on mem161 is WARNING: LOAD WARNING - total load average: 2.94, 3.55, 1.75 [03:51:13] PROBLEM - mem161 memcached on mem161 is CRITICAL: connect to address 10.0.16.131 and port 11211: Connection refused [03:51:52] RECOVERY - mem161 Current Load on mem161 is OK: LOAD OK - total load average: 0.39, 2.37, 1.53 [03:56:53] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 3.09, 1.82, 1.48 [03:58:53] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.69, 1.63, 1.45 [03:59:13] RECOVERY - mem161 memcached on mem161 is OK: TCP OK - 0.006 second response time on 10.0.16.131 port 11211 [04:02:51] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.58, 1.73, 1.54 [04:06:50] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.02, 1.48, 1.49 [04:17:47] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 1.92, 2.15, 1.81 [04:19:47] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.80, 1.99, 1.79 [04:21:46] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.77, 1.59, 1.67 [04:24:39] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.47 ms [04:26:42] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.98 ms [04:27:46] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.48, 1.92, 1.74 [04:29:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1008407474 secs [04:29:46] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.72, 1.78, 1.71 [04:31:46] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.99, 1.53, 1.63 [04:46:41] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.97, 1.84, 1.67 [04:48:40] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 3.26, 2.27, 1.84 [04:50:39] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.57, 1.96, 1.78 [04:53:29] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.21 ms [04:54:39] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.86, 1.51, 1.64 [04:55:32] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.71 ms [04:59:34] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09934917092 secs [05:03:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.48 ms [05:05:58] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.04 ms [05:19:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.15 ms [05:21:28] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.85 ms [05:31:34] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1035500169 secs [05:40:25] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.85, 1.76, 1.56 [05:42:24] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.94, 1.41, 1.45 [05:49:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.19 ms [05:51:30] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.55 ms [05:54:45] PROBLEM - corru.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'corru.wiki' expires in 15 day(s) (Mon 01 Apr 2024 05:40:53 AM GMT +0000). [05:55:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/00f87107ac09...c0683a40b169 [05:55:08] [02miraheze/ssl] 07MirahezeSSLBot 03c0683a4 - Bot: Update SSL cert for corru.wiki [05:56:24] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [06:08:02] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 187.44 ms [06:10:05] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.81 ms [06:23:30] RECOVERY - corru.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'corru.wiki' will expire on Fri 14 Jun 2024 04:55:00 AM GMT +0000. [06:28:12] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.99, 1.76, 1.47 [06:30:12] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.57, 1.66, 1.46 [06:36:09] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.97, 1.77, 1.55 [06:40:08] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.36, 1.67, 1.56 [07:31:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 208.11 ms [07:33:41] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.58 ms [08:13:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c0683a40b169...d42dd713eb74 [08:13:43] [02miraheze/ssl] 07MirahezeSSLBot 03d42dd71 - Bot: Add SSL cert for wiki.cor.fyi [08:17:21] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.38 ms [08:23:32] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.23 ms [08:30:52] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.07 ms [08:32:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.15 ms [08:44:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.71 ms [08:46:22] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.77 ms [08:58:22] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.00 ms [09:00:20] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.92 ms [09:27:39] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [09:35:32] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09958079457 secs [10:09:05] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 164.11 ms [10:09:27] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1024062932 secs [10:11:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.51 ms [10:34:05] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.36 ms [10:38:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.17 ms [10:41:28] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.06857216358 secs [10:45:27] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.102915436 secs [10:55:27] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09999775887 secs [10:59:03] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 275.47 ms [11:03:05] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.94 ms [11:07:27] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.100815773 secs [11:09:27] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09699514508 secs [11:23:03] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.67 ms [11:27:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.68 ms [11:49:53] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1047668457 secs [12:19:52] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.89 ms [12:21:50] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.37 ms [12:30:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.52 ms [12:34:58] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.88 ms [12:35:42] @Site Reliability Engineers https://meta.miraheze.org/wiki/Special:RequestImportQueue/697 automatic import failed [12:36:08] additionally, I'll add myself to the failed import notifs [12:44:06] [02mw-config] 07redbluegreenhat opened pull request 03#5501: add myself to the ImportDump notifs - 13https://github.com/miraheze/mw-config/pull/5501 [12:45:00] miraheze/mw-config - redbluegreenhat the build passed. [12:46:12] [02mw-config] 07redbluegreenhat closed pull request 03#5501: add myself to the ImportDump notifs - 13https://github.com/miraheze/mw-config/pull/5501 [12:46:15] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7d1c571166aa...a9b9831f8a04 [12:46:17] [02miraheze/mw-config] 07redbluegreenhat 03a9b9831 - add myself to the ImportDump notifs (#5501) [12:47:08] miraheze/mw-config - redbluegreenhat the build passed. [12:53:14] !log [@test151] starting deploy of {'config': True} to test151 [12:53:16] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [12:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:53:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:05] !log [@mwtask181] starting deploy of {'config': True} to all [13:02:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:15] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [13:02:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:07:46] !log [@mwtask171] starting deploy of {'config': True} to all [13:07:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:07:54] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [13:08:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:27:35] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 56% [13:52:45] PROBLEM - wikigenius.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikigenius.org' expires in 15 day(s) (Mon 01 Apr 2024 01:28:06 PM GMT +0000). [13:52:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d42dd713eb74...a4acae38c7b1 [13:53:00] [02miraheze/ssl] 07MirahezeSSLBot 03a4acae3 - Bot: Update SSL cert for wikigenius.org [14:06:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.08 ms [14:08:17] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.46 ms [14:15:34] PROBLEM - bn.wikiyri.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bn.wikiyri.org' expires in 15 day(s) (Mon 01 Apr 2024 01:46:46 PM GMT +0000). [14:15:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a4acae38c7b1...5d5bad73f6b4 [14:15:49] [02miraheze/ssl] 07MirahezeSSLBot 035d5bad7 - Bot: Update SSL cert for bn.wikiyri.org [14:22:43] RECOVERY - wikigenius.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikigenius.org' will expire on Fri 14 Jun 2024 12:52:51 PM GMT +0000. [14:23:35] PROBLEM - wikiyri.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiyri.org' expires in 15 day(s) (Mon 01 Apr 2024 01:53:01 PM GMT +0000). [14:23:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5d5bad73f6b4...429480ee55f5 [14:23:47] [02miraheze/ssl] 07MirahezeSSLBot 03429480e - Bot: Update SSL cert for wikiyri.org [14:32:24] PROBLEM - wiki.theatlas.live - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.theatlas.live' expires in 15 day(s) (Mon 01 Apr 2024 02:20:04 PM GMT +0000). [14:32:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/429480ee55f5...d953ffb897b5 [14:32:37] [02miraheze/ssl] 07MirahezeSSLBot 03d953ffb - Bot: Update SSL cert for wiki.theatlas.live [14:36:13] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [14:37:50] PROBLEM - wiki.esufranchise.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.esufranchise.com' expires in 15 day(s) (Mon 01 Apr 2024 02:25:28 PM GMT +0000). [14:38:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d953ffb897b5...e4efab6edb28 [14:38:03] [02miraheze/ssl] 07MirahezeSSLBot 03e4efab6 - Bot: Update SSL cert for wiki.esufranchise.com [14:45:30] RECOVERY - bn.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bn.wikiyri.org' will expire on Fri 14 Jun 2024 01:15:40 PM GMT +0000. [14:52:40] RECOVERY - wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiyri.org' will expire on Fri 14 Jun 2024 01:23:40 PM GMT +0000. [15:00:41] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:01:18] RECOVERY - wiki.theatlas.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.theatlas.live' will expire on Fri 14 Jun 2024 01:32:30 PM GMT +0000. [15:07:39] RECOVERY - wiki.esufranchise.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.esufranchise.com' will expire on Fri 14 Jun 2024 01:37:56 PM GMT +0000. [15:29:31] [1/5] Looks like that wiki missed an upgrade SQL patch: [15:29:31] [2/5] ``` [15:29:31] [3/5] Reason for failure: [f2fe62a7c8d4ef5a83e9db09] Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'themagnusarchiveswiki.cu_private_event' doesn't exist Function: MediaWiki\CheckUser\Hooks::onLocalUserCreated Query: INSERT INTO `cu_private_event` (cupe_namespace,cupe_title,cupe_log_type,cupe_log_action,cupe_params,cupe_page,cupe_actor,cupe_timestamp,cupe_ip [15:29:32] [4/5] ,cupe_ip_hex,cupe_xff,cupe_xff_hex,cupe_agent,cupe_comment_id) VALUES (2,,'checkuser-private-event','autocreate-account','a:0:{}',0,2019,'20240316123307','127.0.0.1','7F000001',,NULL,,2) [15:29:32] [5/5] ``` [15:44:26] PROBLEM - project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 15 day(s) (Mon 01 Apr 2024 03:18:24 PM GMT +0000). [15:44:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e4efab6edb28...06ab2f85ee47 [15:44:40] [02miraheze/ssl] 07MirahezeSSLBot 0306ab2f8 - Bot: Update SSL cert for project-patterns.com [15:46:18] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 15 day(s) (Mon 01 Apr 2024 03:18:24 PM GMT +0000). [15:53:40] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03notif [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/8f188d667f91 [15:53:41] [02miraheze/ImportDump] 07Universal-Omega 038f188d6 - Always enable import failed and new request notifications [15:53:42] [02ImportDump] 07Universal-Omega created branch 03notif - 13https://github.com/miraheze/ImportDump [15:53:45] [02ImportDump] 07Universal-Omega opened pull request 03#90: Always enable import failed and new request notifications - 13https://github.com/miraheze/ImportDump/pull/90 [15:54:40] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03notif [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/8f188d667f91...a419322f785a [15:54:41] [02miraheze/ImportDump] 07Universal-Omega 03a419322 - Remove [15:54:44] [02ImportDump] 07Universal-Omega synchronize pull request 03#90: Always enable import failed and new request notifications - 13https://github.com/miraheze/ImportDump/pull/90 [15:58:17] PROBLEM - miraheze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.com' expires in 15 day(s) (Mon 01 Apr 2024 03:35:42 PM GMT +0000). [15:58:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/06ab2f85ee47...7d1d55a2674c [15:58:32] [02miraheze/ssl] 07MirahezeSSLBot 037d1d55a - Bot: Update SSL cert for miraheze.com [16:02:18] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03notif [+0/-0/±20] 13https://github.com/miraheze/ImportDump/compare/a419322f785a...d62e834404cd [16:02:20] [02miraheze/ImportDump] 07Universal-Omega 03d62e834 - Remove i18n [16:02:21] [02ImportDump] 07Universal-Omega synchronize pull request 03#90: Always enable import failed and new request notifications - 13https://github.com/miraheze/ImportDump/pull/90 [16:02:59] miraheze/ImportDump - Universal-Omega the build passed. [16:09:54] miraheze/ImportDump - Universal-Omega the build passed. [16:13:23] RECOVERY - project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Fri 14 Jun 2024 02:44:33 PM GMT +0000. [16:15:35] RECOVERY - www.project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Fri 14 Jun 2024 02:44:33 PM GMT +0000. [16:27:34] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Fri 14 Jun 2024 02:58:23 PM GMT +0000. [16:40:07] [02ImportDump] 07Universal-Omega closed pull request 03#90: Always enable import failed and new request notifications - 13https://github.com/miraheze/ImportDump/pull/90 [16:40:10] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±22] 13https://github.com/miraheze/ImportDump/compare/66a1b9585095...4b54b196ca86 [16:40:13] [02miraheze/ImportDump] 07Universal-Omega 034b54b19 - Always enable import failed and new request notifications (#90) [16:40:20] [02ImportDump] 07Universal-Omega deleted branch 03notif - 13https://github.com/miraheze/ImportDump [16:40:22] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03notif [16:46:14] what even happened to CheckUser man, seems like a good chunk of wikis never upgraded [16:47:47] miraheze/ImportDump - Universal-Omega the build passed. [16:52:58] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=--wiki=themagnusarchiveswiki /srv/mediawiki/1.41/extensions/CheckUser/schema/mysql/patch-cu_private_event-def.sql (END - exit=65280) [16:53:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:38] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=themagnusarchiveswiki /srv/mediawiki/1.41/extensions/CheckUser/schema/mysql/patch-cu_private_event-def.sql (END - exit=0) [16:54:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:59:43] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=themagnusarchiveswiki /srv/mediawiki/1.41/extensions/CheckUser/schema/mysql/patch-cu_changes-add-cuc_only_for_read_old.sql (END - exit=0) [16:59:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:46] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/45217a96f62f [17:14:47] [02miraheze/ssl] 07MacFan4000 0345217a9 - Delete certificates/wiki.seamly.net.crt [17:14:49] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:14:50] [02ssl] 07MacFan4000 opened pull request 03#778: Delete certificates/wiki.seamly.net.crt - 13https://github.com/miraheze/ssl/pull/778 [17:15:44] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/45217a96f62f...07e91cdfde9d [17:15:47] [02miraheze/ssl] 07MacFan4000 0307e91cd - Update certs.yaml [17:15:49] PROBLEM - wiki.wildport.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wildport.com' expires in 15 day(s) (Mon 01 Apr 2024 05:12:22 PM GMT +0000). [17:15:50] [02ssl] 07MacFan4000 synchronize pull request 03#778: Delete certificates/wiki.seamly.net.crt - 13https://github.com/miraheze/ssl/pull/778 [17:16:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7d1d55a2674c...139440371d7d [17:16:04] [02miraheze/ssl] 07MirahezeSSLBot 031394403 - Bot: Update SSL cert for wiki.wildport.com [17:17:06] [02ssl] 07MacFan4000 closed pull request 03#778: Delete certificates/wiki.seamly.net.crt - 13https://github.com/miraheze/ssl/pull/778 [17:17:09] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [17:17:11] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/139440371d7d...56ed516e5bed [17:17:14] [02miraheze/ssl] 07MacFan4000 0356ed516 - Delete certificates/wiki.seamly.net.crt (#778) [17:17:15] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:20:33] PROBLEM - www.documentopublico.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.documentopublico.com.br' expires in 15 day(s) (Mon 01 Apr 2024 05:11:30 PM GMT +0000). [17:20:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/56ed516e5bed...c247a4b59261 [17:20:50] [02miraheze/ssl] 07MirahezeSSLBot 03c247a4b - Bot: Update SSL cert for www.documentopublico.com.br [17:22:55] [02dns] 07MacFan4000 opened pull request 03#499: Add osrwiki.org and grayzonewarfare.wiki zones - 13https://github.com/miraheze/dns/pull/499 [17:28:01] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mahdiruiz.line.pm' expires in 15 day(s) (Mon 01 Apr 2024 05:11:49 PM GMT +0000). [17:28:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c247a4b59261...26a15a65036e [17:28:57] [02miraheze/ssl] 07MirahezeSSLBot 0326a15a6 - Bot: Update SSL cert for wiki.mahdiruiz.line.pm [17:33:02] PROBLEM - wiki.tdrweb.top - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tdrweb.top' expires in 15 day(s) (Mon 01 Apr 2024 05:12:08 PM GMT +0000). [17:33:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/26a15a65036e...0c971d70555a [17:33:15] [02miraheze/ssl] 07MirahezeSSLBot 030c971d7 - Bot: Update SSL cert for wiki.tdrweb.top [17:34:17] PROBLEM - wiki.omegabuild.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.omegabuild.uk' expires in 15 day(s) (Mon 01 Apr 2024 05:25:27 PM GMT +0000). [17:34:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0c971d70555a...fab21318e9ae [17:34:31] [02miraheze/ssl] 07MirahezeSSLBot 03fab2131 - Bot: Update SSL cert for wiki.omegabuild.uk [17:34:36] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [17:42:34] PROBLEM - quadball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'quadball.wiki' expires in 15 day(s) (Mon 01 Apr 2024 05:36:36 PM GMT +0000). [17:42:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fab21318e9ae...5c3712b0d7fe [17:42:52] [02miraheze/ssl] 07MirahezeSSLBot 035c3712b - Bot: Update SSL cert for quadball.wiki [17:45:08] PROBLEM - otherkin.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'otherkin.wiki' expires in 15 day(s) (Mon 01 Apr 2024 05:37:31 PM GMT +0000). [17:45:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5c3712b0d7fe...a835e6cc64f0 [17:45:21] [02miraheze/ssl] 07MirahezeSSLBot 03a835e6c - Bot: Update SSL cert for otherkin.wiki [17:45:47] PROBLEM - wiki.jagtek.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jagtek.org' expires in 15 day(s) (Mon 01 Apr 2024 05:19:25 PM GMT +0000). [17:45:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a835e6cc64f0...0e07c7c5aa6d [17:46:03] [02miraheze/ssl] 07MirahezeSSLBot 030e07c7c - Bot: Update SSL cert for wiki.jagtek.org [17:49:01] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rosestulipsandliberty.com' expires in 15 day(s) (Mon 01 Apr 2024 05:40:02 PM GMT +0000). [17:49:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0e07c7c5aa6d...f7df59c78ac8 [17:49:14] [02miraheze/ssl] 07MirahezeSSLBot 03f7df59c - Bot: Update SSL cert for wiki.rosestulipsandliberty.com [17:49:37] RECOVERY - www.documentopublico.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'www.documentopublico.com.br' will expire on Fri 14 Jun 2024 04:20:42 PM GMT +0000. [17:54:04] PROBLEM - beidipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'beidipedia.com' expires in 15 day(s) (Mon 01 Apr 2024 05:35:47 PM GMT +0000). [17:54:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f7df59c78ac8...c232a219cace [17:54:18] [02miraheze/ssl] 07MirahezeSSLBot 03c232a21 - Bot: Update SSL cert for beidipedia.com [17:56:49] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [17:59:28] PROBLEM - lostmediawiki.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lostmediawiki.ru' expires in 15 day(s) (Mon 01 Apr 2024 05:38:22 PM GMT +0000). [17:59:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c232a219cace...35fa36241abf [17:59:43] [02miraheze/ssl] 07MirahezeSSLBot 0335fa362 - Bot: Update SSL cert for lostmediawiki.ru [18:01:40] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:02:55] RECOVERY - wiki.tdrweb.top - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tdrweb.top' will expire on Fri 14 Jun 2024 04:33:07 PM GMT +0000. [18:02:59] RECOVERY - wiki.omegabuild.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.omegabuild.uk' will expire on Fri 14 Jun 2024 04:34:23 PM GMT +0000. [18:15:27] RECOVERY - wiki.wildport.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wildport.com' will expire on Fri 14 Jun 2024 04:15:55 PM GMT +0000. [18:23:26] RECOVERY - beidipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beidipedia.com' will expire on Fri 14 Jun 2024 04:54:10 PM GMT +0000. [18:24:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03smtp-ipv6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6a612c219cce [18:24:38] [02miraheze/mw-config] 07Universal-Omega 036a612c2 - Force IPv6 for smtp connection [18:24:41] [02mw-config] 07Universal-Omega created branch 03smtp-ipv6 - 13https://github.com/miraheze/mw-config [18:25:36] [02mw-config] 07Universal-Omega opened pull request 03#5502: Force IPv6 for smtp connection - 13https://github.com/miraheze/mw-config/pull/5502 [18:26:25] miraheze/mw-config - Universal-Omega the build passed. [18:28:32] RECOVERY - lostmediawiki.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'lostmediawiki.ru' will expire on Fri 14 Jun 2024 04:59:34 PM GMT +0000. [18:32:39] [02mw-config] 07Universal-Omega closed pull request 03#5502: Force IPv6 for smtp connection - 13https://github.com/miraheze/mw-config/pull/5502 [18:32:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a9b9831f8a04...56285c136fea [18:32:44] [02miraheze/mw-config] 07Universal-Omega 0356285c1 - Force IPv6 for smtp connection (#5502) [18:32:46] [02miraheze/mw-config] 07Universal-Omega deleted branch 03smtp-ipv6 [18:32:47] [02mw-config] 07Universal-Omega deleted branch 03smtp-ipv6 - 13https://github.com/miraheze/mw-config [18:32:49] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:32:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:59] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [18:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:33:30] miraheze/mw-config - Universal-Omega the build passed. [18:37:43] !log [@mwtask171] starting deploy of {'config': True} to all [18:37:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f1bfb764f92b...71941196804c [18:37:48] [02miraheze/puppet] 07Universal-Omega 037194119 - postfix: set smtp_address_preference to ipv6 [18:37:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:51] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [18:37:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:41:59] RECOVERY - quadball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'quadball.wiki' will expire on Fri 14 Jun 2024 04:42:44 PM GMT +0000. [18:44:06] RECOVERY - otherkin.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'otherkin.wiki' will expire on Fri 14 Jun 2024 04:45:13 PM GMT +0000. [18:45:23] RECOVERY - wiki.jagtek.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jagtek.org' will expire on Fri 14 Jun 2024 04:45:53 PM GMT +0000. [18:47:04] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Fri 14 Jun 2024 04:49:07 PM GMT +0000. [18:52:59] !log [@test151] starting deploy of {'config': True} to test151 [18:53:00] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:53:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:14] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09928479791 secs [19:28:41] [02dns] 07Universal-Omega commented on pull request 03#499: Add osrwiki.org and grayzonewarfare.wiki zones - 13https://github.com/miraheze/dns/pull/499#issuecomment-2002099256 [19:47:42] [02dns] 07MacFan4000 synchronize pull request 03#499: Add osrwiki.org and grayzonewarfare.wiki zones - 13https://github.com/miraheze/dns/pull/499 [19:48:13] [02dns] 07MacFan4000 synchronize pull request 03#499: Add osrwiki.org and grayzonewarfare.wiki zones - 13https://github.com/miraheze/dns/pull/499 [19:48:46] [02dns] 07MacFan4000 commented on pull request 03#499: Add osrwiki.org and grayzonewarfare.wiki zones - 13https://github.com/miraheze/dns/pull/499#issuecomment-2002105462 [19:50:14] PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1135315597 secs [19:50:45] [02dns] 07Universal-Omega closed pull request 03#499: Add osrwiki.org and grayzonewarfare.wiki zones - 13https://github.com/miraheze/dns/pull/499 [19:50:48] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/dns/compare/1cd5a4b37815...fdc9d74bfe69 [19:50:49] [02miraheze/dns] 07MacFan4000 03fdc9d74 - Add osrwiki.org and grayzonewarfare.wiki zones (#499) [20:26:14] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09542998672 secs [20:26:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/71941196804c...7e4491190a02 [20:26:20] [02miraheze/puppet] 07Universal-Omega 037e44911 - ssl: use ssl for letsencrypt expiry emails [20:58:28] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [21:08:28] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59% [21:10:28] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [21:14:03] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.68 ms [21:16:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.99 ms [22:11:25] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9702MiB (10% inode=98%); [22:15:42] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9693MiB (10% inode=98%); [22:32:53] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03dmarc [+0/-0/±1] 13https://github.com/miraheze/dns/commit/4ca64e87ed6d [22:32:55] [02miraheze/dns] 07Universal-Omega 034ca64e8 - Update dmarc record for wikitide.org [22:32:56] [02dns] 07Universal-Omega created branch 03dmarc - 13https://github.com/miraheze/dns [22:33:00] [02dns] 07Universal-Omega opened pull request 03#500: Update dmarc record for wikitide.org - 13https://github.com/miraheze/dns/pull/500 [22:35:02] [02dns] 07Universal-Omega closed pull request 03#500: Update dmarc record for wikitide.org - 13https://github.com/miraheze/dns/pull/500 [22:35:05] [02miraheze/dns] 07Universal-Omega deleted branch 03dmarc [22:35:06] [02dns] 07Universal-Omega deleted branch 03dmarc - 13https://github.com/miraheze/dns [22:35:07] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/fdc9d74bfe69...7a66920aa8be [22:35:10] [02miraheze/dns] 07Universal-Omega 037a66920 - Update dmarc record for wikitide.org (#500) [22:35:36] [02dns] 07Universal-Omega closed pull request 03#498: Use wikitide.net for nameservers - 13https://github.com/miraheze/dns/pull/498 [22:35:38] [02miraheze/dns] 07Universal-Omega deleted branch 03ns [22:35:41] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±260] 13https://github.com/miraheze/dns/compare/7a66920aa8be...a4b3dcb7786d [22:35:42] [02miraheze/dns] 07Universal-Omega 03a4b3dcb - Use wikitide.net for nameservers (#498) [22:35:45] [02dns] 07Universal-Omega deleted branch 03ns - 13https://github.com/miraheze/dns [22:41:04] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 164.68 ms [22:41:58] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a4b3dcb7786d...61306486d9aa [22:42:01] [02miraheze/dns] 07Universal-Omega 036130648 - Use wikitide.net for nameservers [22:43:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.33 ms [22:54:03] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±264] 13https://github.com/miraheze/dns/compare/61306486d9aa...c395b3c655bf [22:54:05] [02miraheze/dns] 07Universal-Omega 03c395b3c - Fix iodef in CAA for policy violation emails and use caapolicy@ [23:10:08] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±265] 13https://github.com/miraheze/dns/compare/c395b3c655bf...0c3bc1405c6b [23:10:11] [02miraheze/dns] 07Universal-Omega 030c3bc14 - Update CAA for new iodef [23:18:43] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±263] 13https://github.com/miraheze/dns/compare/0c3bc1405c6b...8fed6f8c8627 [23:18:46] [02miraheze/dns] 07Universal-Omega 038fed6f8 - Correct comment [23:33:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/35fa36241abf...49cfe8a35b8e [23:33:58] [02miraheze/ssl] 07MirahezeSSLBot 0349cfe8a - Bot: Add SSL cert for grayzonewarfare.wiki [23:34:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/49cfe8a35b8e...39759799ec7e [23:34:27] [02miraheze/ssl] 07MirahezeSSLBot 033975979 - Bot: Add SSL cert for osrwiki.org [23:47:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7e4491190a02...860e1e6e04ff [23:47:09] [02miraheze/puppet] 07Universal-Omega 03860e1e6 - postfix: use localhost.wikitide.net [23:51:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/860e1e6e04ff...0fdadaf2c488 [23:51:06] [02miraheze/puppet] 07Universal-Omega 030fdadaf - opensearch-security: use wikitide.net