[00:03:02] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.82, 1.97, 1.56 [00:05:06] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.72, 1.83, 1.56 [00:07:07] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.47, 1.70, 1.54 [00:18:07] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.30, 1.78, 1.61 [00:20:06] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.42, 2.00, 1.71 [00:22:05] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.51, 1.79, 1.66 [00:22:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.43, 2.87 [00:24:05] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.08, 1.57, 1.60 [00:24:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.70, 2.88, 2.73 [00:30:54] [02puppet] 07Universal-Omega commented on pull request 03#1789: Update default.vcl - 13https://git.io/JR12C [00:30:56] [02puppet] 07Universal-Omega closed pull request 03#1789: Update default.vcl - 13https://git.io/JnCUK [00:43:44] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: T7441 -Void, last run 10 minutes ago with 0 failures [01:53:06] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.16, 1.97, 1.60 [01:55:08] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.37, 1.68, 1.53 [02:03:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.02, 2.74 [02:05:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 2.91, 2.73 [02:18:28] RECOVERY - ping6 on jobrunner3 is OK: PING OK - Packet loss = 0%, RTA = 3.06 ms [02:21:15] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.86, 1.63, 1.49 [02:25:14] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.14, 1.55, 1.50 [02:27:06] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JR1db [02:27:07] [02miraheze/mediawiki] 07The-Voidwalker 030c2b120 - T7441: force purge requests to use HTTP [02:27:09] [02mediawiki] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://git.io/vbL5b [02:27:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JR1Fe [02:27:18] [02miraheze/puppet] 07paladox 034b15d26 - Update firewall.yaml [02:27:20] [02puppet] 07paladox synchronize pull request 03#1869: Install mw12 - 13https://git.io/JR1LK [02:27:33] [02puppet] 07paladox closed pull request 03#1869: Install mw12 - 13https://git.io/JR1LK [02:27:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JR1FT [02:27:36] [02miraheze/puppet] 07paladox 03b33c3e0 - Install mw12 (#1869) [02:27:37] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [02:27:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [02:29:43] [02mediawiki] 07The-Voidwalker opened pull request 03#2602: T7441: force purge requests to use HTTP - 13https://git.io/JR1FS [02:31:41] I believe that's good ^ any objections before I merge? [02:34:31] Voidwalker: it's fine with me. [02:40:07] Receive this error from CreateWiki after approving wiki: Exception experienced creating the wiki. Error is: Redis server error: socket error on read socket Wiki request: https://meta.miraheze.org/wiki/Special:RequestWikiQueue/19844 [02:40:08] [url] Wiki requests queue - Miraheze Meta | meta.miraheze.org [02:40:18] Wiki was created but is empty and I think has no admins [02:41:27] [02mediawiki] 07The-Voidwalker closed pull request 03#2602: T7441: force purge requests to use HTTP - 13https://git.io/JR1FS [02:41:29] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JR1xZ [02:41:30] [02miraheze/mediawiki] 07The-Voidwalker 03e6a32af - T7441: force purge requests to use HTTP (#2602) [02:48:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.59, 3.11 [02:48:25] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:50:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.75, 3.22 [02:51:53] PROBLEM - mw12 JobRunner Service on mw12 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [02:52:07] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.081 second response time [02:52:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.61, 3.24 [02:53:02] PROBLEM - mw10 JobRunner Service on mw10 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [02:53:07] PROBLEM - mw9 JobRunner Service on mw9 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [02:53:51] RECOVERY - mw12 JobRunner Service on mw12 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [02:53:51] PROBLEM - mwtask1 JobRunner Service on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [02:54:02] PROBLEM - mw11 JobRunner Service on mw11 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [02:56:02] RECOVERY - mw11 JobRunner Service on mw11 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [02:56:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.73, 3.35 [02:56:59] RECOVERY - mw9 JobRunner Service on mw9 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [02:57:02] RECOVERY - mw10 JobRunner Service on mw10 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:00:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.75, 3.44 [03:00:19] https://github.com/miraheze/mw-config/blob/d9a65eaa0951cd7e43468b97be8d0409d980222b/PhpAutoPrepend.php#L26-L27 should probably be updated [03:00:20] [url] mw-config/PhpAutoPrepend.php at d9a65eaa0951cd7e43468b97be8d0409d980222b · miraheze/mw-config · GitHub | github.com [03:00:48] Voidwalker: https://github.com/miraheze/mw-config/pull/4047 [03:00:49] [url] jobrunner* -> mwtask1 by Universal-Omega · Pull Request #4047 · miraheze/mw-config · GitHub | github.com [03:02:01] ah, one step ahead :) [03:03:23] <^A> I think my message got burried in the pile of icinga messages but I received this error after approving wiki: "Exception experienced creating the wiki. Error is: Redis server error: socket error on read socket Wiki request": https://meta.miraheze.org/wiki/Special:RequestWikiQueue/19844 [03:03:24] [url] Wiki requests queue - Miraheze Meta | meta.miraheze.org [03:03:47] The wiki was created by the main page is missing [03:04:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 4.16, 3.65 [03:05:28] I can confirm per CentralAuth that the requestor was attached to the wiki but not granted rights [03:06:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.86, 3.61 [03:06:19] i see this https://graylog.miraheze.org/messages/graylog_270/0090f630-fb14-11eb-aca7-0200001a24a4 [03:06:27] but doesn't reveal much only the same error as you [03:07:11] s/by/but/ [03:07:11] Agent meant to say: The wiki was created but the main page is missing [03:07:19] I see, how strange [03:07:36] almost definitely related to redis being funky [03:07:59] RECOVERY - mwtask1 JobRunner Service on mwtask1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:10:38] Agent, will fix in a moment [03:10:50] Thanks [03:11:47] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:12:04] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:12:11] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20012 bytes in 3.755 second response time [03:12:39] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:12:48] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:13:40] almost ready to deploy mw12 [03:13:57] :O [03:14:16] :o [03:14:48] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:14:50] also fixed issue for that wiki [03:15:09] looks like the puppet issue was just a timeout, and it will be fixed on rerun [03:15:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JRMTu [03:15:36] [02miraheze/puppet] 07paladox 0386ff31e - Add mw12 to varnish [03:15:38] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [03:15:40] [02puppet] 07paladox opened pull request 03#1870: Add mw12 to varnish - 13https://git.io/JRMTg [03:16:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JRMTi [03:16:20] [02miraheze/puppet] 07paladox 03cee7b1b - Update stunnel.conf [03:16:21] [02puppet] 07paladox synchronize pull request 03#1870: Add mw12 to varnish - 13https://git.io/JRMTg [03:16:57] Voidwalker: Thank you, I've informed the the requester of this. [03:18:16] Lets deploy! [03:18:22] [02puppet] 07paladox closed pull request 03#1870: Add mw12 to varnish - 13https://git.io/JRMTg [03:18:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRMkk [03:18:25] [02miraheze/puppet] 07paladox 036b237d8 - Add mw12 to varnish (#1870) [03:18:26] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [03:18:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [03:18:44] 👍 [03:18:57] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JRMkn [03:18:58] [02miraheze/MirahezeDebug] 07Universal-Omega 0313038f3 - Add mw12 and mw13 [03:19:00] [02MirahezeDebug] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fpk7C [03:19:01] [02MirahezeDebug] 07Universal-Omega opened pull request 03#4: Add mw12 and mw13 - 13https://git.io/JRMkc [03:19:36] Voidwalker: would you like to do mw13 to see how to install servers/setup? [03:20:28] that'd be a good idea [03:21:19] PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:21:19] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:21:21] PROBLEM - cp13 Stunnel Http for mw10 on cp13 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:21:28] PROBLEM - cp13 Stunnel Http for mw9 on cp13 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:21:33] PROBLEM - cp13 Stunnel Http for mw11 on cp13 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:21:38] huh [03:21:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [03:21:44] PROBLEM - cp3 Stunnel Http for mon2 on cp3 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:21:48] 503 paladox [03:21:56] bugger reverting [03:22:05] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 59% [03:22:08] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [03:22:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 0.95, 2.38, 3.15 [03:22:20] PROBLEM - cp13 Stunnel Http for mon2 on cp13 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:22:20] PROBLEM - cp12 Stunnel Http for mw9 on cp12 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:22:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRMI4 [03:22:23] [02miraheze/puppet] 07paladox 03a5d9b63 - Revert "Add mw12 to varnish (#1870)" [03:22:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [03:22:28] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:22:29] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[stunnel4] [03:22:30] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 5 backends are down. mw8 mw9 mw10 mw11 mw12 [03:22:31] PROBLEM - cp12 Stunnel Http for mw11 on cp12 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:22:31] PROBLEM - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is WARNING: WARNING - NGINX Error Rate is 58% [03:22:33] PROBLEM - cp12 Stunnel Http for mon2 on cp12 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:22:47] PROBLEM - cp3 Stunnel Http for mw9 on cp3 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:22:49] PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:22:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [03:22:56] PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:23:13] PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:23:18] RECOVERY - cp13 Stunnel Http for mw10 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 0.009 second response time [03:23:19] RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.011 second response time [03:23:28] RECOVERY - cp13 Stunnel Http for mw9 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.009 second response time [03:23:33] RECOVERY - cp13 Stunnel Http for mw11 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 0.010 second response time [03:24:04] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [03:24:04] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [03:24:16] RECOVERY - cp13 Stunnel Http for mon2 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 32975 bytes in 0.023 second response time [03:24:20] RECOVERY - cp12 Stunnel Http for mw9 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.318 second response time [03:24:28] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:24:29] RECOVERY - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is OK: OK - NGINX Error Rate is 1% [03:24:31] RECOVERY - cp12 Stunnel Http for mon2 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 32975 bytes in 0.337 second response time [03:24:32] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[stunnel4] [03:24:33] RECOVERY - cp12 Stunnel Http for mw11 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 0.368 second response time [03:24:37] found the issue! [03:24:43] something is already binned to 8090 [03:24:50] RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.553 second response time [03:25:13] RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15412 bytes in 0.318 second response time [03:25:28] ah, that'd do it, yeah [03:25:56] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 90.92 ms [03:26:02] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[stunnel4] [03:26:04] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 57% [03:26:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRMLr [03:26:12] [02miraheze/puppet] 07paladox 03a6b5173 - Add mw12 to varnish [03:26:29] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:26:31] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [03:26:54] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 1.030 second response time [03:27:07] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 1.030 second response time [03:27:37] RECOVERY - cp3 Stunnel Http for mon2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 33012 bytes in 1.045 second response time [03:27:58] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 90.91 ms [03:28:03] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 4% [03:28:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:28:34] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 1.001 second response time [03:28:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:28:43] RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 1.005 second response time [03:31:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRMqe [03:31:32] [02miraheze/puppet] 07paladox 03de71d2c - Add mw12 to round robin [03:31:44] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 8 backends are healthy [03:32:04] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:34:34] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:34:48] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.72, 1.44, 1.16 [03:35:09] [02mw-config] 07Universal-Omega synchronize pull request 03#4038: Add wgReCaptchaSiteKey - 13https://git.io/JRsBK [03:35:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JRMmv [03:35:12] [02miraheze/mw-config] 07Universal-Omega 03bb54709 - Update LocalSettings.php [03:35:33] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:36:03] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:36:11] miraheze/mw-config - Universal-Omega the build passed. [03:36:43] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.13, 1.32, 1.15 [03:42:55] [02miraheze/dns] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JRMYb [03:42:56] [02miraheze/dns] 07The-Voidwalker 039b7cc9d - jobrunner4 -> mw13 [03:42:58] [02dns] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://git.io/vbQXl [03:43:03] [02dns] 07The-Voidwalker opened pull request 03#213: jobrunner4 -> mw13 - 13https://git.io/JRMYp [03:43:39] were imports impacted by the 503? [03:44:25] Voidwalker: looks good to me [03:45:00] [02dns] 07The-Voidwalker closed pull request 03#213: jobrunner4 -> mw13 - 13https://git.io/JRMYp [03:45:01] [02miraheze/dns] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRMOa [03:45:03] [02miraheze/dns] 07The-Voidwalker 0393662bb - jobrunner4 -> mw13 (#213) [03:55:07] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 21 minutes ago with 0 failures [03:57:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JRMGJ [03:57:39] [02miraheze/mw-config] 07Universal-Omega 031dc8566 - Update LocalWiki.php [03:57:41] [02mw-config] 07Universal-Omega synchronize pull request 03#4046: Remove default for wgExemptFromUserRobotsControl - 13https://git.io/JRomY [03:58:36] miraheze/mw-config - Universal-Omega the build passed. [04:04:56] !log [mw12]: cd /srv/mediawiki/w && sudo -u www-data wget -O composer.phar https://getcomposer.org/composer-1.phar | php && sudo -u www-data php composer.phar update --no-dev [04:04:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:05:33] paladox: ^ was fataling on some wikis when connecting through mw12 due to it not having wikibase stuff. [04:05:52] oh [04:05:54] thanks! [04:06:18] no problem. [04:07:21] hmm, that probably ought to be a thing in puppet [04:07:40] It is [04:07:45] at least the last time i checked [04:07:49] i guess it must have failed? [04:07:53] ohhh [04:07:56] nvm [04:08:28] [02mw-config] 07Universal-Omega edited pull request 03#4046: Fix wgExemptFromUserRobotsControl - 13https://git.io/JRomY [04:08:36] [02mw-config] 07Universal-Omega edited pull request 03#4046: Fix wgExemptFromUserRobotsControl - 13https://git.io/JRomY [04:09:35] !log [mwtask1]: cd /srv/mediawiki/w && sudo -u www-data wget -O composer.phar https://getcomposer.org/composer-1.phar | php && sudo -u www-data php composer.phar update --no-dev [04:09:37] Universal_Omega: ^ [04:09:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:43] [02mw-config] 07Universal-Omega edited pull request 03#4046: T7770: Fix wgExemptFromUserRobotsControl - 13https://git.io/JRomY [04:10:12] paladox: Oh, yeah, I only realized mw12 checking logs so thanks for doing that one also! [04:10:29] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:10:40] [02mw-config] 07Universal-Omega closed pull request 03#4046: T7770: Fix wgExemptFromUserRobotsControl - 13https://git.io/JRomY [04:10:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRMc5 [04:10:43] [02miraheze/mw-config] 07Universal-Omega 03792e746 - T7770: Fix wgExemptFromUserRobotsControl (#4046) [04:10:45] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [04:10:46] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [04:12:44] miraheze/mw-config - Universal-Omega the build passed. [04:21:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.05, 8.59, 6.13 [04:22:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 15.77, 10.30, 7.11 [04:23:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 19.10, 11.14, 7.20 [04:23:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRM8C [04:23:19] [02miraheze/puppet] 07paladox 037d031bc - Increase to 32 [04:24:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.09, 10.80, 6.43 [04:24:33] paladox, voidwalker: 503s again [04:24:38] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 26.61, 14.47, 7.70 [04:24:43] that wasn't me [04:24:44] but [04:24:51] PROBLEM - cp3 Stunnel Http for mw9 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:24:55] high load seems to have hit some of the mw* [04:24:57] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:14] PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:25:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [04:25:27] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:25:35] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:25:54] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:26:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [04:26:47] PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:27:12] PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:27:22] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 3 backends are down. mw9 mw11 mw12 [04:27:35] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 4 backends are down. mw8 mw9 mw11 mw12 [04:27:42] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:27:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw8 mw9 mw11 mw12 [04:27:51] CPU maxed out on cloud4, not really sure where to look first [04:28:11] ohh [04:28:14] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15404 bytes in 1.035 second response time [04:28:16] mw12 would be the reason [04:28:19] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [04:28:36] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [04:28:51] should we shut that off and migrate it to, say, cloud5? Or something else? [04:29:09] yeh [04:29:20] though the IP was purchased in a batch [04:29:24] so we cannot migrate it [04:29:32] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15411 bytes in 1.009 second response time [04:29:33] (we didn't purchase it individually) [04:29:47] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20013 bytes in 0.377 second response time [04:29:50] RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 1.039 second response time [04:29:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:29:51] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 0.239 second response time [04:30:00] !log stop nginx on mw12 to help relieve preasure [04:30:01] RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.006 second response time [04:30:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:30:14] well for now, let's shut it down, finish with mw13, and figure out a solution for later [04:30:40] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:30:42] it might be that we could swap it out for a low-resource vm on cloud5, but I don't really have the brain power to figure out a good solution at the moment [04:31:09] RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15403 bytes in 0.333 second response time [04:31:30] Voidwalker: sure [04:31:43] PROBLEM - mw12 HTTPS on mw12 is CRITICAL: connect to address 51.195.236.220 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [04:32:02] !log shutdown mw12 [04:32:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:32:53] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:19] !log [@test3] starting deploy of {'config': True} to skip [04:33:20] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [04:33:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:33:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:34:36] downtimed mw12 in icinga fyi Voidwalker [04:35:06] ty [04:35:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.09, 7.39, 7.94 [04:37:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.36, 6.05, 7.71 [04:38:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.53, 5.75, 7.62 [04:42:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.96, 5.35, 7.52 [04:42:14] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.22, 5.28, 6.78 [04:42:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.45, 4.35, 6.61 [04:43:17] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.54, 4.45, 6.41 [04:46:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.50, 4.40, 6.63 [05:06:11] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+1/-0/±0] 13https://git.io/JRMVa [05:06:12] [02miraheze/puppet] 07The-Voidwalker 0354ad5c5 - install mw13 [05:06:14] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://git.io/vbiAS [05:06:17] [02puppet] 07The-Voidwalker opened pull request 03#1871: install mw13 - 13https://git.io/JRMVw [05:07:01] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JRMV9 [05:07:02] [02miraheze/puppet] 07The-Voidwalker 034660bf1 - Update site.pp [05:07:04] [02puppet] 07The-Voidwalker synchronize pull request 03#1871: install mw13 - 13https://git.io/JRMVw [05:09:05] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JRMwC [05:09:06] [02miraheze/puppet] 07The-Voidwalker 03d2268c9 - Update firewall.yaml [05:09:08] [02puppet] 07The-Voidwalker synchronize pull request 03#1871: install mw13 - 13https://git.io/JRMVw [05:10:13] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.43, 1.72, 1.45 [05:11:51] [02puppet] 07Universal-Omega opened pull request 03#1872: Add Stunnel HTTP monitoring for mw12 and mw13 - 13https://git.io/JRMrf [05:12:25] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.42, 1.57, 1.42 [05:12:45] [02puppet] 07Universal-Omega synchronize pull request 03#1872: Add Stunnel HTTP monitoring for mw12 and mw13 - 13https://git.io/JRMrf [05:13:58] paladox, Voidwalker: ^ probably shouldn't merge till mw13 is fully done but it looks like that was missed for mw12, so did that and mw13 there. [05:14:46] thanks [05:15:04] No problem. [05:16:42] [02puppet] 07Universal-Omega edited pull request 03#1872: Add Stunnel HTTP monitoring for mw12 and mw13 - 13https://git.io/JRMrf [05:23:48] PROBLEM - mw13 Puppet on mw13 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [05:28:52] [02puppet] 07The-Voidwalker closed pull request 03#1871: install mw13 - 13https://git.io/JRMVw [05:28:54] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JRMiB [05:28:55] [02miraheze/puppet] 07The-Voidwalker 0351327e3 - install mw13 (#1871) [05:29:09] [02dns] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbQXl [05:29:10] [02miraheze/dns] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [05:29:13] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbiAS [05:29:15] [02miraheze/puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [05:33:14] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JRMPn [05:33:16] [02miraheze/puppet] 07The-Voidwalker 03f2f0fb6 - add mw13 to varnish [05:33:17] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://git.io/vbiAS [05:33:20] [02puppet] 07The-Voidwalker opened pull request 03#1873: add mw13 to varnish - 13https://git.io/JRMPW [05:33:55] [02puppet] 07Universal-Omega opened pull request 03#1874: Update groups.conf - 13https://git.io/JRMPE [05:35:01] [02puppet] 07Universal-Omega edited pull request 03#1874: Update groups.conf - 13https://git.io/JRMPE [05:35:06] Whats up with the 502 Bad Gateway? [05:35:30] PROBLEM - cp13 Varnish Backends on cp13 is WARNING: No backends detected. If this is an error, see readme.txt [05:35:47] paladox: Voidwalker errors again. [05:35:59] hmm [05:36:17] that's odd [05:36:36] !log restart varnish on cp13 [05:36:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:36:59] I haven't faced any errors [05:37:53] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw12 [05:38:39] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JRMXE [05:38:40] [02miraheze/puppet] 07The-Voidwalker 0392150f8 - Update default.vcl [05:38:42] [02puppet] 07The-Voidwalker synchronize pull request 03#1873: add mw13 to varnish - 13https://git.io/JRMPW [05:38:48] that looks better [05:38:52] nvm meta finally went down [05:38:54] Check FAMEPedia [05:39:06] yeah, now I can't access any wiki [05:39:06] oh let me remove mw12 [05:39:07] Oops [05:39:13] PROBLEM - cp12 Varnish Backends on cp12 is WARNING: No backends detected. If this is an error, see readme.txt [05:39:42] hmm [05:39:44] uh, now cp12 is doing it, hmm [05:39:45] cp12? [05:40:09] https://www.irccloud.com/pastebin/6A7CiYsk/ [05:40:10] - cp12 Varnish Backends on cp12 is WARNING: No backends detected [05:40:25] !log restart varnish on cp12 [05:40:30] meta is back [05:40:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:42] so are most wikis [05:40:48] oh [05:40:50] it OOM'd [05:40:52] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=cp12.miraheze.org&var-port=9100&viewPanel=78 [05:40:53] [url] Grafana | grafana.miraheze.org [05:41:04] same for cp13 [05:41:26] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw12 [05:41:37] Yup [05:43:47] bug: https://phabricator.miraheze.org/T7319 [05:43:48] [url] ⚓ T7319 Varnish should clean up ramdisk after failure | phabricator.miraheze.org [05:45:12] thanks! [05:52:17] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JRMDw [05:52:18] [02miraheze/puppet] 07The-Voidwalker 0384b4277 - Update default.vcl [05:52:20] [02puppet] 07The-Voidwalker synchronize pull request 03#1873: add mw13 to varnish - 13https://git.io/JRMPW [05:53:56] [02puppet] 07The-Voidwalker reviewed pull request 03#1872 commit - 13https://git.io/JRMDF [05:53:57] [02puppet] 07The-Voidwalker reviewed pull request 03#1872 commit - 13https://git.io/JRMDb [05:54:54] [02puppet] 07The-Voidwalker reviewed pull request 03#1873 commit - 13https://git.io/JRMyY [05:54:59] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JRMyO [05:55:00] [02miraheze/puppet] 07The-Voidwalker 03fa69b49 - Update modules/varnish/templates/default.vcl [05:55:02] [02puppet] 07The-Voidwalker synchronize pull request 03#1873: add mw13 to varnish - 13https://git.io/JRMPW [05:56:00] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/JRMyl [05:56:01] [02miraheze/puppet] 07The-Voidwalker 03a738be6 - Update default.vcl [05:56:03] [02puppet] 07The-Voidwalker synchronize pull request 03#1873: add mw13 to varnish - 13https://git.io/JRMPW [05:58:02] [02puppet] 07The-Voidwalker closed pull request 03#1873: add mw13 to varnish - 13https://git.io/JRMPW [05:58:03] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRMyA [05:58:05] [02miraheze/puppet] 07The-Voidwalker 037bb350a - add mw13 to varnish (#1873) [05:58:06] [02miraheze/puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [05:58:08] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbiAS [05:58:39] [02puppet] 07Universal-Omega synchronize pull request 03#1872: Add Stunnel HTTP monitoring for mw12 and mw13 - 13https://git.io/JRMrf [05:58:52] [02puppet] 07Universal-Omega reviewed pull request 03#1872 commit - 13https://git.io/JRMSI [05:59:44] Voidwalker: ^ fixed that. Thanks! I missed that. [06:01:23] [02MirahezeDebug] 07Universal-Omega closed pull request 03#4: Add mw12 and mw13 - 13https://git.io/JRMkc [06:01:25] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRMSr [06:01:26] [02miraheze/MirahezeDebug] 07Universal-Omega 037ef9866 - Add mw12 and mw13 (#4) [06:01:28] [02MirahezeDebug] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fpk7C [06:01:29] [02miraheze/MirahezeDebug] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [06:02:49] [02mw-config] 07The-Voidwalker closed pull request 03#4048: Add mw12 and mw13 to wgLinterSubmitterWhitelist - 13https://git.io/JR1Yg [06:02:51] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRMSb [06:02:52] [02miraheze/mw-config] 07Universal-Omega 0388b4e2a - Add mw12 and mw13 to wgLinterSubmitterWhitelist (#4048) [06:03:09] thanks Voidwalker [06:03:23] !log [@test3] starting deploy of {'config': True} to skip [06:03:24] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [06:03:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:03:33] np [06:03:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:03:53] miraheze/mw-config - The-Voidwalker the build passed. [06:04:35] !log [mw13]: cd /srv/mediawiki/w && sudo -u www-data wget -O composer.phar https://getcomposer.org/composer-1.phar | php && sudo -u www-data php composer.phar update --no-dev [06:04:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:48] a ty [06:05:27] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 [06:05:29] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3 [06:05:43] I see mw13 serving content :D [06:06:06] [02mw-config] 07Universal-Omega closed pull request 03#4047: jobrunner* -> mwtask1 - 13https://git.io/JR1ma [06:06:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRM99 [06:06:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.86, 7.36, 5.56 [06:06:09] [02miraheze/mw-config] 07Universal-Omega 03e8a6f0f - jobrunner* -> mwtask1 (#4047) [06:06:10] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [06:06:12] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [06:06:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.93, 6.30, 5.55 [06:07:13] miraheze/mw-config - Universal-Omega the build passed. [06:09:16] [02puppet] 07paladox closed pull request 03#1872: Add Stunnel HTTP monitoring for mw12 and mw13 - 13https://git.io/JRMrf [06:09:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRMHB [06:09:19] [02miraheze/puppet] 07Universal-Omega 0335756d3 - Add Stunnel HTTP monitoring for mw12 and mw13 (#1872) [06:10:08] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [06:10:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.05, 7.18, 6.00 [06:10:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.73, 6.06, 5.67 [06:10:38] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [06:10:47] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [06:12:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.07, 5.72, 5.61 [06:15:08] is https://phabricator.miraheze.org/T7139 actually completely done then since https://phabricator.miraheze.org/T7676 is I think? [06:15:10] [url] ⚓ T7139 MediaWiki Capacity Proposal | phabricator.miraheze.org [06:15:10] [url] ⚓ T7676 [New] Server Request for mw12 and mw13 | phabricator.miraheze.org [06:15:28] I'm gunna keep it open for a bit [06:15:37] though i guess the 2nd one can be closed as resolve [06:21:35] [02mw-config] 07Universal-Omega closed pull request 03#4038: Add wgReCaptchaSiteKey - 13https://git.io/JRsBK [06:21:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRM5L [06:21:38] [02miraheze/mw-config] 07Universal-Omega 032fe14b3 - Add wgReCaptchaSiteKey (#4038) [06:21:39] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [06:21:41] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [06:22:30] miraheze/mw-config - Universal-Omega the build passed. [06:22:36] [02puppet] 07Reception123 commented on pull request 03#1854: Remove $wgReCaptchaSiteKey from PrivateSettings - 13https://git.io/JRM58 [06:22:40] [02puppet] 07Reception123 closed pull request 03#1854: Remove $wgReCaptchaSiteKey from PrivateSettings - 13https://git.io/JRs4u [06:22:41] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRM5B [06:22:43] [02miraheze/puppet] 07Universal-Omega 03fe7bdcc - Remove $wgReCaptchaSiteKey from PrivateSettings (#1854) [06:23:48] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:26:42] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:27:16] [02mw-config] 07Universal-Omega closed pull request 03#4031: Remove ENV for GDFONTPATH - 13https://git.io/JBPSm [06:27:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRMda [06:27:19] [02miraheze/mw-config] 07Universal-Omega 03f5d9d18 - Remove ENV for GDFONTPATH (#4031) [06:27:20] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [06:27:22] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [06:28:03] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:28:09] miraheze/mw-config - Universal-Omega the build passed. [06:28:13] RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:28:22] paladox: so are mw12 & mw13 pooled now? [06:28:34] I see 13 moaning about puppet summary file missing [06:28:35] yup [06:28:41] 13 is fixed [06:28:45] needed a puppet upgrade [06:28:49] Ah [06:29:09] But also \o/ [06:29:51] I saw ramdisk still caused an outage from 2 cps OOM'ing at once [06:30:06] yeh [06:33:11] I mean we were very unlucky 2 went at once if it was 12/13 [06:33:32] !log [@test3] starting deploy of {'config': True} to skip [06:33:33] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [06:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:49] RhinosF1: yeah it was cp12/13 [06:34:05] i guess 12 went because 13 went [06:34:12] Universal_Omega: I've read over 80 emails [06:34:33] paladox: I said the other day there was significant risk of it if it happened at the wrong time [06:34:54] When 13 went a few days ago we were struggling on 12 [06:35:05] John has a task open for 14 [06:35:09] yeh, it'll be like dinamo [06:35:17] the first goes then the rest follows [06:35:25] because the dns will depool [06:35:26] RhinosF1: Yeah been a busy day. But 80 emails? that is alot.... [06:35:33] https://phabricator.miraheze.org/T7783 [06:35:33] [url] ⚓ T7783 [New] Server Resource Request for UK Cache Proxy | phabricator.miraheze.org [06:35:49] that server would also fail like cp12 [06:36:01] we need to fix the OOM and why it happens since it shouldn't... [06:36:02] Universal_Omega: yeah when I looked at the my phone I thought oh god [06:36:29] paladox: well if there's less load on cp13 there's less of risk of it OOM'ing in theory [06:36:42] But yes when one goes it shifts traffic and the next goes and the next [06:37:16] I'm surprised we didn't loose 3 [06:38:02] i caught cp12 in time [06:38:07] Ack [06:38:24] literally restarted cp13 just before cp12 notified us lol [06:38:33] If it would have been much longer I doubt 3 would have taken load of everywhere at once [06:41:00] RECOVERY - mwtask1 APT on mwtask1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:42:43] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:48:53] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:52:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRMxx [06:52:09] [02miraheze/mw-config] 07Universal-Omega 039d2d8b7 - add two more VectorDefaultSkinVersion configs [06:53:04] miraheze/mw-config - Universal-Omega the build passed. [06:55:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JRMhI [06:55:26] [02miraheze/mw-config] 07Universal-Omega 032220187 - add two more VectorDefaultSkinVersion configs [06:55:27] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [06:55:29] [02mw-config] 07Universal-Omega opened pull request 03#4049: add two more VectorDefaultSkinVersion configs - 13https://git.io/JRMht [06:56:25] miraheze/mw-config - Universal-Omega the build passed. [06:57:38] [02mw-config] 07Universal-Omega closed pull request 03#4049: add two more VectorDefaultSkinVersion configs - 13https://git.io/JRMht [06:57:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRMho [06:57:41] [02miraheze/mw-config] 07Universal-Omega 0322501d4 - add two more VectorDefaultSkinVersion configs (#4049) [06:57:42] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [06:57:44] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [06:58:36] miraheze/mw-config - Universal-Omega the build passed. [07:03:23] !log [@test3] starting deploy of {'config': True} to skip [07:03:24] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:03:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:11:55] [02RemovePII] 07Universal-Omega closed pull request 03#17: Allow underscores to be used with the RemovePII action - 13https://git.io/JRGdx [07:11:56] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRDfH [07:11:58] [02miraheze/RemovePII] 07Universal-Omega 031e50332 - Allow underscores to be used with the RemovePII action (#17) [07:11:59] [02RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/Jsfpj [07:12:01] [02miraheze/RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [07:12:25] [02mediawiki] 07Universal-Omega commented on pull request 03#2593: Bump extensions/RemovePII from `7ac312d` to `5c90a7b` - 13https://git.io/JRDJU [07:12:27] [02mediawiki] 07dependabot[bot] edited pull request 03#2593: Bump extensions/RemovePII from `7ac312d` to `5c90a7b` - 13https://git.io/JRRwF [07:12:52] miraheze/RemovePII - Universal-Omega the build passed. [07:13:15] [02mediawiki] 07dependabot[bot] edited pull request 03#2593: Bump extensions/RemovePII from `7ac312d` to `5c90a7b` - 13https://git.io/JRRwF [07:13:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/RemovePII-1e50332 [+0/-0/±1] 13https://git.io/JRDJl [07:13:25] [02miraheze/mediawiki] 07dependabot[bot] 03c8c0b73 - Bump extensions/RemovePII from `7ac312d` to `1e50332` [07:13:27] [02mediawiki] 07dependabot[bot] opened pull request 03#2603: Bump extensions/RemovePII from `7ac312d` to `1e50332` - 13https://git.io/JRDJ8 [07:13:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/RemovePII-1e50332 - 13https://git.io/vbL5b [07:13:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#2603: Bump extensions/RemovePII from `7ac312d` to `1e50332` - 13https://git.io/JRDJ8 [07:13:31] [02mediawiki] 07dependabot[bot] labeled pull request 03#2603: Bump extensions/RemovePII from `7ac312d` to `1e50332` - 13https://git.io/JRDJ8 [07:13:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#2593: Bump extensions/RemovePII from `7ac312d` to `5c90a7b` - 13https://git.io/JRDJB [07:13:34] [02mediawiki] 07dependabot[bot] closed pull request 03#2593: Bump extensions/RemovePII from `7ac312d` to `5c90a7b` - 13https://git.io/JRRwF [07:13:36] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/RemovePII-5c90a7b - 13https://git.io/vbL5b [07:13:37] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/RemovePII-5c90a7b [07:14:38] [02mediawiki] 07Universal-Omega closed pull request 03#2603: Bump extensions/RemovePII from `7ac312d` to `1e50332` - 13https://git.io/JRDJ8 [07:14:39] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JRDJb [07:14:41] [02miraheze/mediawiki] 07dependabot[bot] 0315ad48e - Bump extensions/RemovePII from `7ac312d` to `1e50332` (#2603) [07:14:42] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/RemovePII-1e50332 - 13https://git.io/vbL5b [07:14:44] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/RemovePII-1e50332 [07:16:26] !log upgrade icingaweb2 on mon2 [07:16:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:39] [02puppet] 07Universal-Omega opened pull request 03#1875: Add mw12 and mw13 to trafficserver - 13https://git.io/JRDmw [07:39:10] [02puppet] 07Universal-Omega synchronize pull request 03#1875: Add mw12 and mw13 to trafficserver - 13https://git.io/JRDmw [07:40:36] [02puppet] 07paladox closed pull request 03#1875: Add mw12 and mw13 to trafficserver - 13https://git.io/JRDmw [07:40:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRDY0 [07:40:39] [02miraheze/puppet] 07Universal-Omega 03b27fbd1 - Add mw12 and mw13 to trafficserver (#1875) [07:41:33] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:41:58] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:42:46] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:42:46] [02puppet] 07Universal-Omega edited pull request 03#1874: Update groups.conf - 13https://git.io/JRMPE [07:46:33] [02puppet] 07paladox closed pull request 03#1874: Update groups.conf - 13https://git.io/JRMPE [07:46:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRD3m [07:46:36] [02miraheze/puppet] 07Universal-Omega 0366269b0 - Update groups.conf (#1874) [07:48:47] Thanks for merging those paladox! [07:48:57] yw [08:05:16] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:05:36] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [08:06:11] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:34:03] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=chimaiwiki /home/reception/chimaiwiki --search-recursively --summary=Attribution: https://www.chimai.com [09:34:07] [url] www.chimai.com | www.chimai.com [09:34:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:30:41] !log DELETED and DROPPED succovwiki [10:30:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:33:31] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=succovwiki /home/reception/succovwiki_xml_0021ebaeeee48c3c639e.xml --username-prefix=succov --report 1 [10:33:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:47:00] !log INSERT INTO content_models (model_id, model_name) VALUES ( 4, 'Scribunto'); on succovwiki [10:47:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:07:51] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±8] 13https://git.io/JRyI1 [11:07:52] [02miraheze/CreateWiki] 07translatewiki 03fe8e03c - Localisation updates from https://translatewiki.net. [11:07:53] [url] Main page - translatewiki.net | translatewiki.net [11:07:54] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRyIM [11:07:55] [02miraheze/IncidentReporting] 07translatewiki 0387d398e - Localisation updates from https://translatewiki.net. [11:07:56] [url] Main page - translatewiki.net | translatewiki.net [11:07:57] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JRyID [11:07:58] [02miraheze/DataDump] 07translatewiki 0315a4fcd - Localisation updates from https://translatewiki.net. [11:07:59] [url] Main page - translatewiki.net | translatewiki.net [11:08:00] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+2/-0/±0] 13https://git.io/JRyIy [11:08:01] [02miraheze/landing] 07translatewiki 0353ea1d7 - Localisation updates from https://translatewiki.net. [11:08:02] [url] Main page - translatewiki.net | translatewiki.net [11:08:03] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JRyIS [11:08:04] [02miraheze/ManageWiki] 07translatewiki 0311a751f - Localisation updates from https://translatewiki.net. [11:08:05] [url] Main page - translatewiki.net | translatewiki.net [11:08:06] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRyIH [11:08:07] [02miraheze/WikiDiscover] 07translatewiki 03833b511 - Localisation updates from https://translatewiki.net. [11:08:08] ... [11:08:09] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JRyIQ [11:08:10] [02miraheze/MirahezeMagic] 07translatewiki 03428b461 - Localisation updates from https://translatewiki.net. [11:08:11] ... [11:08:32] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JRyLI [11:08:33] [02miraheze/ssl] 07Reception123 03f951edb - add staratlaswiki.org cert [11:08:45] miraheze/IncidentReporting - translatewiki the build passed. [11:08:46] miraheze/WikiDiscover - translatewiki the build passed. [11:08:54] miraheze/DataDump - translatewiki the build passed. [11:08:57] miraheze/ManageWiki - translatewiki the build passed. [11:09:04] miraheze/CreateWiki - translatewiki the build passed. [11:09:05] miraheze/MirahezeMagic - translatewiki the build passed. [11:09:11] miraheze/landing - translatewiki the build passed. [11:25:00] [02puppet] 07RhinosF1 opened pull request 03#1876: [POC] mwscript: add some logging for 'long' scripts - 13https://git.io/JRyOD [11:25:24] Reception123: not mergable yet but ^ [11:26:42] [02puppet] 07RhinosF1 synchronize pull request 03#1876: [POC] mwscript: add some logging for 'long' scripts - 13https://git.io/JRyOD [11:27:45] looking good [11:29:28] Reception123: Universal_Omega suggested a --long argument but don't want to bloat the untested mess too much so I should probably move it to argparse before we do that [11:29:39] That might also make it less breakable [11:30:02] Yeah that should be fine [11:30:28] There is a task for adding tests [11:34:45] [02puppet] 07RhinosF1 opened pull request 03#1877: Create workflow.yml - 13https://git.io/JRyGp [11:35:34] [02puppet] 07RhinosF1 edited pull request 03#1877: Add tests for python - 13https://git.io/JRyGp [11:37:09] Reception123: they are far from passing [11:37:24] [02puppet] 07RhinosF1 commented on pull request 03#1877: Add tests for python - 13https://git.io/JRyZS [11:43:29] !log [reception@mw12] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0) [11:43:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:44] !log [reception@mw13] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0) [11:44:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:16] [02puppet] 07RhinosF1 opened pull request 03#1878: checkvarnishbackends: fix flake8 vios - 13https://git.io/JRyCa [11:47:44] [02puppet] 07RhinosF1 opened pull request 03#1879: generatevmtouch: remove whitespace - 13https://git.io/JRyWL [11:49:03] Reception123: there's 2 clean ups [11:49:18] looking [11:49:32] [02puppet] 07Reception123 closed pull request 03#1878: checkvarnishbackends: fix flake8 vios - 13https://git.io/JRyCa [11:49:34] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRyW6 [11:49:35] [02miraheze/puppet] 07RhinosF1 038d42913 - checkvarnishbackends: fix flake8 vios (#1878) [11:49:40] [02puppet] 07Reception123 closed pull request 03#1879: generatevmtouch: remove whitespace - 13https://git.io/JRyWL [11:49:42] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRyWP [11:49:43] [02miraheze/puppet] 07RhinosF1 030818c2d - generatevmtouch: remove whitespace (#1879) [11:51:41] Reception123: there's over a 100 flake8 violations in puppet [11:51:52] Some seem false positive [11:51:57] But it's still shit [12:14:52] wow [12:45:04] Reception123: resetWikiCaches only affects version number [12:45:12] running it twice, resets the cache twice globally [12:57:33] JohnLewis: thought it had to be ran per server [12:57:45] That was my bad advice if not [12:58:04] Nope, never been the case :) [12:58:05] We were having the non English not show bug [12:58:15] On new servers [12:58:30] Happens when cache is deleted [12:59:46] oh, really? Good to know in the future then [13:13:41] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=countryhumanswiki /home/reception/countryhumans_pages_full.xml --username-prefix=wikia:countryhumans --report 1 [13:13:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:35:28] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polcompballwiki /home/reception/polcompballs_pages_full.xml --username-prefix=wikia:polcompball --report 1 [13:35:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:55:08] A lot of the Polandball wikis have been getting requested, so of course there's going to be more backtracking, and lots of importation that arises. [14:08:33] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRyAs [14:08:35] [02miraheze/dns] 07JohnFLewis 0359560b1 - add cp14 to dns [14:15:54] backtracking? [14:21:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.54, 7.19, 5.84 [14:25:50] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.67, 6.24, 5.80 [14:31:29] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JRSvY [14:31:30] [02miraheze/puppet] 07JohnFLewis 03bb54c09 - add cp14 to puppet [14:31:32] [02miraheze/puppet] 07JohnFLewis 03422e900 - add cp14 to puppet [14:34:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.50, 7.10, 6.04 [14:35:20] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.48, 7.72, 6.68 [14:35:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.48, 6.88, 5.96 [14:35:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.73, 7.49, 6.38 [14:35:49] High load :( [14:35:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.56, 3.28 [14:36:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.79, 6.27, 5.88 [14:37:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.27, 7.15, 6.60 [14:37:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.16, 6.61, 6.19 [14:37:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.17, 3.17 [14:39:14] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.73, 7.67, 6.83 [14:41:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.55, 7.42, 6.86 [14:41:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.42, 3.32 [14:43:08] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.47, 6.59, 6.61 [14:43:21] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.57, 6.39, 6.18 [14:43:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.04, 3.18 [14:50:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 6.96, 6.46 [14:51:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.55, 3.27 [14:52:48] PROBLEM - cp14 Stunnel Http for mw8 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:48] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.14, 6.58, 6.38 [14:52:51] PROBLEM - db11 Disk Space on db11 is WARNING: DISK WARNING - free space: / 41962 MB (10% inode=97%); [14:52:53] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 6 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 [14:52:56] PROBLEM - cp14 HTTP 4xx/5xx ERROR Rate on cp14 is CRITICAL: CRITICAL - NGINX Error Rate is 92% [14:53:02] PROBLEM - cp14 Stunnel Http for mw12 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:04] Err db11 disk [14:53:13] PROBLEM - cp14 Puppet on cp14 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [14:53:13] PROBLEM - cp14 Stunnel Http for mw13 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:20] PROBLEM - cp14 HTTPS on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4800 bytes in 0.010 second response time [14:53:22] PROBLEM - cp14 Stunnel Http for mw10 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:28] PROBLEM - cp14 Stunnel Http for mw9 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:30] PROBLEM - cp14 Stunnel Http for mw11 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:48] JohnLewis, paladox: do we need to action db11 [14:54:24] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.25, 1.62, 1.17 [14:54:32] 41GB left, so definitely not needing action immediately [14:54:48] Looking at grafana it doesn't seem to be massively rising [14:54:58] RECOVERY - db11 Disk Space on db11 is OK: DISK OK - free space: / 42492 MB (11% inode=97%); [14:55:08] * Reception123 will run the deletion script soon [14:56:27] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.11, 1.42, 1.16 [14:56:38] Reception123: you probably got weeks before it's critical [14:56:57] As in actual critical not icinga [14:57:26] 541 to be deleted so it should do something [14:58:04] Ack [14:59:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.80, 3.47 [15:01:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.66, 3.44 [15:02:04] !log running delbackups ( bash ./delbackups.sh /home/reception/delwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php) [15:02:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:05:10] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.13, 7.40, 6.68 [15:05:40] RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15488 bytes in 0.019 second response time [15:05:41] RECOVERY - cp14 HTTPS on cp14 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2256 bytes in 0.011 second response time [15:05:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.95, 3.23, 3.31 [15:05:54] RECOVERY - cp14 Stunnel Http for mw12 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15489 bytes in 0.005 second response time [15:06:02] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.22, 7.33, 6.74 [15:06:07] RECOVERY - cp14 Stunnel Http for mw11 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15503 bytes in 0.054 second response time [15:06:09] RECOVERY - cp14 Stunnel Http for mw10 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15489 bytes in 0.031 second response time [15:06:18] RECOVERY - cp14 Stunnel Http for mw13 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15489 bytes in 0.033 second response time [15:06:28] RECOVERY - cp14 Stunnel Http for mw9 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15488 bytes in 0.021 second response time [15:06:48] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRSqX [15:06:49] [02miraheze/dns] 07JohnFLewis 03ae097fa - add cp14 to gb pool [15:07:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.27, 7.38, 6.76 [15:07:14] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [15:08:02] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.41, 7.92, 7.04 [15:08:53] JohnLewis: can we fix puppet https://icinga.miraheze.org/monitoring/service/show?host=cp14&service=cp14%20Puppet [15:08:55] [url] Icinga Web 2 Login | icinga.miraheze.org [15:09:13] RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:22] It fixes itself with time [15:09:23] as above [15:09:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.33, 4.29, 3.70 [15:09:59] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.15, 7.33, 6.93 [15:10:41] Error rates are dropping naturally [15:11:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.81, 6.24, 6.46 [15:11:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.76, 3.58 [15:11:57] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.96, 6.55, 6.69 [15:16:47] PROBLEM - cp14 HTTP 4xx/5xx ERROR Rate on cp14 is WARNING: WARNING - NGINX Error Rate is 59% [15:18:42] RECOVERY - cp14 HTTP 4xx/5xx ERROR Rate on cp14 is OK: OK - NGINX Error Rate is 29% [15:19:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.05, 3.17, 3.35 [15:20:11] Yey [15:21:00] Puppet ran on ns2 so 14 should get traffic [15:21:46] Yeah I can see it flow on grafana! [15:21:53] \o/ and thanks JohnLewis [15:24:51] Welcome, I was always puzzled at the decision to take 2 and turn it into one [15:25:18] It's certainly shown recently that it didn't help [15:25:53] Not bad to see 3 new servers in 2 days and 2 reimaged [15:34:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.30, 3.25 [15:34:50] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.25, 7.31, 6.37 [15:36:26] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.29, 20.82, 19.05 [15:36:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.30, 3.26 [15:36:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.51, 6.64, 6.24 [15:38:20] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.20, 19.14, 18.64 [15:40:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.46, 3.32 [15:44:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.20, 3.26 [16:04:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.16, 3.13 [16:08:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.20, 3.26, 3.19 [17:02:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.68, 3.37 [17:04:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.71, 3.41 [17:05:38] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.69, 6.83, 5.69 [17:06:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.48, 3.36 [17:07:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.89, 5.76, 5.44 [17:11:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 3.07, 3.29 [17:25:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.50, 7.54, 5.54 [17:26:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.41, 6.62, 5.52 [17:27:52] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.90, 7.77, 5.89 [17:30:10] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.04, 5.94, 5.49 [17:33:50] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.64, 7.74, 6.39 [17:35:52] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.33, 7.61, 6.51 [17:36:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.20, 6.92, 5.72 [17:37:51] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.98, 6.61, 6.27 [17:37:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.61, 5.88, 5.49 [17:45:41] Hey so, I'm a bureaucrat from the Polandball Wiki [17:45:52] A few days ago requested an import, which went through [17:46:04] but it doesn't seem like the pages or info actually made it to the new wiki [17:46:11] https://phabricator.miraheze.org/T7774 [17:46:12] [url] ⚓ T7774 Import XML File to Polandball Wiki | phabricator.miraheze.org [17:46:17] https://polandball.miraheze.org/wiki/Polandball_Wiki [17:46:18] [url] Polandball Wiki | polandball.miraheze.org [17:46:31] does anyone know what went wrong amidst this whole ordeal [17:46:41] Reception123: ^^ [17:56:25] Where are we getting that from [17:56:45] 3/3 pages I have checked have imported revisions on [18:00:39] I think he's referring to the fact that Special:Statistics shows only 100 pages instead of over 10k+ [18:01:45] Special:Statistics doesnt auto update [18:02:07] Reception123: can you double check it finished successfully then run initSiteStats pls [18:02:36] No access atm but will look in ~1 hour [18:05:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 18.16, 9.39, 6.46 [18:05:17] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.68, 1.76, 1.39 [18:06:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.84, 6.97, 5.79 [18:07:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.64, 7.85, 6.23 [18:07:15] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.47, 1.59, 1.37 [18:08:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.17, 6.00, 5.58 [18:11:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.67, 6.19, 5.93 [18:12:00] that's what I was thinking too, that Statistics was probably outdated but checking out some of their pages, I see many redlinks [18:51:51] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JR9JS [18:51:53] [02miraheze/RottenLinks] 07Universal-Omega 037b77a5e - Use MultiHttpClient [18:51:54] [02RottenLinks] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fxBrr [18:52:08] [02RottenLinks] 07Universal-Omega opened pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [18:53:18] miraheze/RottenLinks - Universal-Omega the build passed. [19:08:54] [02RottenLinks] 07Universal-Omega edited pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [19:09:17] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.71, 1.40, 1.10 [19:10:22] [02RottenLinks] 07Universal-Omega edited pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [19:11:14] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.08, 1.40, 1.15 [19:27:41] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JR9qJ [19:27:42] [02miraheze/RottenLinks] 07Universal-Omega 03a076585 - Lower minimum required MW version to 1.35.3 and bumb version [19:27:44] [02RottenLinks] 07Universal-Omega synchronize pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [19:28:45] miraheze/RottenLinks - Universal-Omega the build passed. [19:29:27] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JR9q0 [19:29:28] [02miraheze/RottenLinks] 07Universal-Omega 030da36c9 - Update CHANGELOG [19:29:30] [02RottenLinks] 07Universal-Omega synchronize pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [19:30:28] miraheze/RottenLinks - Universal-Omega the build passed. [19:30:42] [02RottenLinks] 07Universal-Omega edited pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [19:31:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.14, 2.57 [19:32:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 2.83, 2.52 [19:39:15] !log DELETED and DROPPED https://phabricator.miraheze.org/P441 [19:39:16] [url] ✎ P441 Deleted wikis 12/08/2021 | phabricator.miraheze.org [19:39:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:39:55] available space: [19:40:09] db11: 42G->46G [19:40:24] db12: 61G->64G [19:40:37] db13: 55G->64G [19:40:47] RhinosF1: ^ FYI. Not too much but not terrible either [19:58:10] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 15.80, 21.44, 17.57 [20:00:05] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 12.82, 18.47, 16.93 [20:18:27] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.20, 1.99, 1.49 [20:18:31] Reception|away: I want to consider an optimise run soon [20:18:52] See if we can reclaim space from that [20:19:14] After we drop flagged revs stuff with .37 [20:19:29] And image migration [20:20:27] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.42, 1.76, 1.46 [20:22:24] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.10, 2.05, 1.62 [20:24:25] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.74, 1.98, 1.64 [20:26:24] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.06, 1.54, 1.52 [20:52:52] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.58, 1.82, 1.53 [20:56:45] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.39, 1.56, 1.50 [21:05:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.02, 6.65, 5.48 [21:07:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.21, 5.69, 5.27 [21:43:13] [02RottenLinks] 07Universal-Omega edited pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [21:43:44] [02RottenLinks] 07Universal-Omega edited pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [21:44:13] [02RottenLinks] 07Universal-Omega edited pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [21:44:39] [02RottenLinks] 07Universal-Omega edited pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [22:01:49] [02RottenLinks] 07Universal-Omega edited pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [22:21:42] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JR9XG [22:21:43] [02miraheze/RottenLinks] 07Universal-Omega 03c15554a - Formatting [22:21:45] [02RottenLinks] 07Universal-Omega synchronize pull request 03#38: Use MultiHttpClient - 13https://git.io/JR9JH [22:22:43] miraheze/RottenLinks - Universal-Omega the build passed. [22:46:54] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JR9yv [22:46:56] [02miraheze/RottenLinks] 07Universal-Omega 0374d2ce8 - Update RottenLinks.php [22:46:57] [02RottenLinks] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/fxBrr [22:47:03] [02RottenLinks] 07Universal-Omega opened pull request 03#39: pull status message directly from response not HttpStatus - 13https://git.io/JR9yJ [22:48:00] miraheze/RottenLinks - Universal-Omega the build passed. [22:48:07] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JR9ym [22:48:08] [02miraheze/RottenLinks] 07Universal-Omega 033ba37b2 - Update updateExternalLinks.php [22:48:10] [02RottenLinks] 07Universal-Omega synchronize pull request 03#39: pull status message directly from response not HttpStatus - 13https://git.io/JR9yJ [22:49:03] miraheze/RottenLinks - Universal-Omega the build passed. [22:56:23] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JR9SS [22:56:24] [02miraheze/RottenLinks] 07Universal-Omega 03e71ea13 - Update RottenLinksPager.php [22:56:26] [02RottenLinks] 07Universal-Omega synchronize pull request 03#39: pull status message directly from response not HttpStatus - 13https://git.io/JR9yJ [22:57:22] miraheze/RottenLinks - Universal-Omega the build passed. [23:06:27] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:03] How long would restarting the import take? [23:07:52] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 4.37, 3.06, 1.29 [23:08:23] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:08:26] [02RottenLinks] 07Universal-Omega edited pull request 03#39: pull status message directly from response not HttpStatus - 13https://git.io/JR9yJ [23:09:50] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.76, 2.10, 1.15 [23:18:16] Around 24-48 hours, I believe. A sysadmins might know more than me. [23:18:42] ah [23:18:49] that's good enough confirmation for me tbh [23:23:28] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.91, 1.50, 1.26 [23:25:24] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.24, 1.41, 1.26 [23:25:42] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [23:37:23] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.63, 2.29, 1.72 [23:39:22] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.28, 1.88, 1.63 [23:41:19] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 1.76, 2.04, 1.73 [23:43:15] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.21, 1.87, 1.71 [23:45:15] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.93, 1.57, 1.62 [23:48:33] [02mediawiki] 07dependabot[bot] opened pull request 03#2604: Bump path-parse from 1.0.6 to 1.0.7 - 13https://git.io/JR9AL [23:48:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/path-parse-1.0.7 [+0/-0/±1] 13https://git.io/JR9At [23:48:36] [02miraheze/mediawiki] 07dependabot[bot] 03698cc34 - Bump path-parse from 1.0.6 to 1.0.7 [23:48:38] [02mediawiki] 07dependabot[bot] labeled pull request 03#2604: Bump path-parse from 1.0.6 to 1.0.7 - 13https://git.io/JR9AL [23:48:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#2604: Bump path-parse from 1.0.6 to 1.0.7 - 13https://git.io/JR9AL [23:48:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/path-parse-1.0.7 - 13https://git.io/vbL5b [23:55:13] PROBLEM - towarzystwo.prawa.kredytowego.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'towarzystwo.prawa.kredytowego.pl' expires in 7 day(s) (Fri 20 Aug 2021 23:51:57 GMT +0000).