[00:01:55] I think we should move it behind deploy tool, it can then run on mw11 only, rebuild cdb cache and then push everywhere [00:02:04] then we gain logging of it [00:02:12] so we know at least when it's running [00:02:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.82, 21.93, 23.20 [00:06:40] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 13.79, 7.88, 6.00 [00:12:48] https://phabricator.miraheze.org/T8251 now exists [00:12:48] [url] ⚓ T8251 LocalisationUpdate Disk I/O caused mw11 php-fpm workers to lock up | phabricator.miraheze.org [00:20:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:22:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.24, 6.55, 5.22 [00:23:01] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.02, 5.57, 4.88 [00:23:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.67, 3.22 [00:23:41] I sleep now [00:24:58] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JPNkf [00:24:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.91, 6.29, 5.29 [00:25:00] [02miraheze/CreateWiki] 07Universal-Omega 03613f3cf - Fix HTML in email [00:25:01] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [00:25:03] [02CreateWiki] 07Universal-Omega opened pull request 03#257: Fix HTML in email - 13https://git.io/JPNkJ [00:25:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:26:43] hmm, got a 503 [00:27:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.74, 6.24, 5.32 [00:29:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.75, 2.87, 3.05 [00:30:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:32:29] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JPNkF [00:32:30] [02miraheze/CreateWiki] 07Universal-Omega 031b3b5c4 - Update WikiManager.php [00:32:32] [02CreateWiki] 07Universal-Omega synchronize pull request 03#257: Fix HTML in email - 13https://git.io/JPNkJ [00:32:43] oh cool, wiki request comments got a new Echo notif icon [00:34:37] Yep. [00:34:43] That's the other thing I did. [00:35:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.50, 3.24 [00:35:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:37:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.37, 3.21 [00:37:54] Agent: all Echo notification types got new icons, not just comments also. declined, and creation did as well. [00:38:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.52, 3.29 [00:39:45] The declined one might be changed again though. It's a bit small... [00:41:03] miraheze/CreateWiki - Universal-Omega the build passed. [00:41:34] [02CreateWiki] 07Universal-Omega closed pull request 03#257: Fix HTML in email - 13https://git.io/JPNkJ [00:41:36] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JPNIh [00:41:38] [02miraheze/CreateWiki] 07Universal-Omega 03a38aef4 - Fix HTML in email (#257) [00:41:39] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:41:41] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [00:42:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 3.31, 3.28 [00:43:12] CosmicAlpha: Yeah, the decline icon is small, at least on emails [00:43:19] looks tiny on emails [00:44:09] Agent: Well at least emails are working now... also I will probably change it again tomorrow but thats less of a priority then at least getting them working. [00:45:45] But yeah the current decline icon is tiny on emails... [00:46:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.65, 3.42 [00:48:08] [02CreateWiki] 07Universal-Omega labeled pull request 03#255: Use settings log if there is more than one change - 13https://git.io/JPPVM [00:48:33] [02ManageWiki] 07Universal-Omega labeled pull request 03#314: T8234: allow (un)locking and (un)deleting actions in combination with others - 13https://git.io/JP6BY [00:49:18] miraheze/CreateWiki - Universal-Omega the build passed. [00:49:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.30, 4.16, 5.98 [00:50:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.34, 4.37, 3.74 [00:51:39] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.43, 4.77, 5.96 [00:57:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.90, 5.22, 5.92 [01:02:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.81, 3.81 [01:03:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.43, 3.81, 5.06 [01:04:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 13.13, 17.13, 23.06 [01:06:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.67, 3.73 [01:08:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.59, 3.69 [01:10:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.74, 3.73 [01:12:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.61, 3.68 [01:12:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.45, 15.30, 20.01 [01:14:19] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 211.30 ms [01:14:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.08, 4.19, 3.89 [01:16:20] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 216.04 ms [01:16:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.88, 3.81 [01:22:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 3.81, 3.71 [01:24:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.77, 3.70 [01:28:30] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 245.36 ms [01:28:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.72, 3.81, 3.67 [01:34:37] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 212.59 ms [01:34:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.69, 3.66 [01:40:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.31, 3.05, 3.40 [02:15:09] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 232.35 ms [02:17:11] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 230.96 ms [02:29:20] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 224.85 ms [02:31:22] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 234.64 ms [02:35:25] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 259.67 ms [02:37:27] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 233.45 ms [02:55:41] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 184.60 ms [02:57:43] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 189.78 ms [02:59:45] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 199.12 ms [03:05:51] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 214.13 ms [03:42:19] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 264.85 ms [03:44:20] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 261.67 ms [05:03:42] PROBLEM - vault.agentisai.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vault.agentisai.com' expires in 15 day(s) (Sun 21 Nov 2021 04:59:53 GMT +0000). [05:07:42] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pt.polandballwiki.com' expires in 15 day(s) (Sun 21 Nov 2021 05:03:56 GMT +0000). [05:08:43] PROBLEM - wiki.eversd.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.eversd.com' expires in 15 day(s) (Sun 21 Nov 2021 05:02:13 GMT +0000). [05:11:59] PROBLEM - www.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.polandballwiki.com' expires in 15 day(s) (Sun 21 Nov 2021 05:07:16 GMT +0000). [05:12:09] PROBLEM - polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.polandballwiki.com' expires in 15 day(s) (Sun 21 Nov 2021 05:07:16 GMT +0000). [05:34:27] PROBLEM - wiki.yapsavun.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yapsavun.xyz' expires in 15 day(s) (Sun 21 Nov 2021 05:29:51 GMT +0000). [06:06:27] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.88, 6.12, 3.99 [06:10:28] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.98, 7.30, 4.98 [06:18:27] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.26, 7.94, 6.19 [06:20:27] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.17, 7.64, 6.29 [06:28:27] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.74, 6.60, 6.41 [06:50:27] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.73, 7.14, 6.49 [06:52:28] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.81, 7.76, 6.79 [06:54:28] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.45, 7.54, 6.83 [06:58:28] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.39, 6.52, 6.59 [07:16:58] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.34, 6.82, 6.62 [07:24:46] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.16, 6.77, 6.73 [07:48:28] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.67, 6.24, 6.23 [07:50:28] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.06, 6.75, 6.41 [07:52:28] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.10, 6.60, 6.40 [08:24:37] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.39, 7.05, 6.51 [08:26:34] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.82, 6.44, 6.34 [08:28:01] PROBLEM - reviews.thejoshmeister.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'reviews.thejoshmeister.com' expires in 15 day(s) (Sun 21 Nov 2021 08:23:11 GMT +0000). [08:55:40] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.28, 6.93, 6.61 [08:59:34] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.20, 7.71, 6.95 [09:01:31] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.88, 7.67, 7.03 [09:03:28] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.13, 7.65, 7.09 [09:09:19] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.23, 7.64, 7.29 [09:26:53] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.09, 6.12, 6.70 [14:09:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 2.83, 2.57 [14:11:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 2.79, 2.58 [14:26:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.47, 2.90 [14:28:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.43, 2.95 [14:30:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.85, 3.16 [14:32:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.62, 3.16 [14:39:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.79, 3.42 [14:41:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.53, 3.37 [14:45:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.76, 3.50 [14:47:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.54, 3.46 [14:49:30] PROBLEM - wiki.hausos.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.hausos.co' expires in 7 day(s) (Sat 13 Nov 2021 14:43:36 GMT +0000). [14:49:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.26, 3.35, 3.39 [14:50:12] PROBLEM - id.altilunium.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'id.altilunium.xyz' expires in 7 day(s) (Sat 13 Nov 2021 14:46:14 GMT +0000). [14:53:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.54, 3.47 [14:55:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.98, 3.02, 3.29 [14:58:17] PROBLEM - wiki.alathramc.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.alathramc.com' expires in 7 day(s) (Sat 13 Nov 2021 14:51:51 GMT +0000). [15:09:04] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Sun 21 Nov 2021 15:00:32 GMT +0000). [15:14:55] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.fbpml.org All nameservers failed to answer the query. [15:29:01] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.hostnetbv.nl.', 'ns2.hostnetbv.com.', 'ns1.hostnet.nl.'], 'CNAME': 'fbpml.miraheze.org.'} [15:47:36] !log messing with l10nupdate on mw11 [15:47:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:49:23] JohnLewis: did you see my task filed last night? [15:50:28] Briefly, but unsure what is required of me from it [15:52:10] JohnLewis: more if I was misunderstanding anything [15:52:28] to actually effect it's changes requires cdb rebuild which it doesn't do [15:52:38] and we always do that from mw11 [15:52:46] so it only needs to be there [15:52:53] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.65, 6.59, 4.81 [15:53:06] and i am going to move it so it's logged and runs with less disk priority [15:53:17] assuming it doesn't impact time too bad [15:53:34] JohnLewis: my ssh just gave rhinos@mw11:~$ ioclient_loop: send disconnect: Broken pipe [15:54:39] * RhinosF1 tries again [15:54:53] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.46, 6.03, 4.82 [15:54:59] !log mw11 try again because broken pipe, load is me [15:55:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:53] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.75, 10.48, 7.12 [16:01:03] If it only needs to ran on the one, then it’ll be fine to adjust it as it’ll still work as intended [16:05:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.42, 3.19 [16:07:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.99, 3.16, 3.12 [16:09:20] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:13:29] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:13:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.47, 3.26 [16:15:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.05, 3.12 [16:16:37] [02puppet] 07RhinosF1 opened pull request 03#2105: l10nupdate: move to mwdeploy and avoid I/O starvation - 13https://git.io/JPxxK [16:17:12] [02puppet] 07RhinosF1 synchronize pull request 03#2105: l10nupdate: move to mwdeploy and avoid I/O starvation - 13https://git.io/JPxxK [16:20:37] [02puppet] 07supertassu reviewed pull request 03#2105 commit - 13https://git.io/JPxpX [16:21:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.54, 3.28 [16:21:44] !log started 2 screens on mw11 with 2723.pts-3.mw11 & 3479.pts-1.mw11 to test l10n update timings with ionice and w/o [16:21:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:17] [02puppet] 07RhinosF1 reviewed pull request 03#2105 commit - 13https://git.io/JPxhm [16:23:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.11, 3.07, 3.14 [16:25:56] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:22] mw11 alerts are me [16:27:15] JohnLewis: my command got killed [16:27:55] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:30:14] i've been killed again [16:33:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.74, 3.36 [16:34:53] !log [rhinos@mw11] t2 done [16:34:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.45, 3.30 [16:35:56] !log [rhinos@mw11] t1 done [16:36:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 3.55, 5.22, 7.64 [16:37:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.12, 3.20 [16:37:57] !log [rhinos@mw11] t2 done [16:38:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:57] !log [rhinos@mw11] t1 done [16:39:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:41:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 4.02, 3.56 [16:42:24] [02puppet] 07RhinosF1 synchronize pull request 03#2105: l10nupdate: move to mwdeploy and avoid I/O starvation - 13https://git.io/JPxxK [16:42:53] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.48, 5.02, 6.80 [16:46:06] [02puppet] 07RhinosF1 synchronize pull request 03#2105: l10nupdate: move to mwdeploy and avoid I/O starvation - 13https://git.io/JPxxK [16:47:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.75, 3.65 [16:48:33] JohnLewis: ^ [16:52:25] !log tests on mw11 done - any issues now are probably actual ones and not me breaking it [16:52:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:55:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 4.14, 3.83 [16:56:53] [02puppet] 07paladox reviewed pull request 03#2105 commit - 13https://git.io/JPpU5 [16:57:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.72, 3.53, 3.64 [16:57:29] paladox: the line is going once puppet has ran once [16:57:50] the line i commented on doesn't look like is going to be removed? [16:57:53] Oh that's my stuff too [16:58:14] What issues? [17:00:00] coding style [17:03:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.71, 2.64, 3.23 [17:07:49] paladox: which style rules? [17:08:37] it should be: [17:08:42] https://www.irccloud.com/pastebin/VJ3j2SDw/ [17:10:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.78, 3.58, 3.38 [17:12:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.49, 3.36 [17:15:40] Ok [17:18:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.31, 3.33 [17:22:36] [02puppet] 07Universal-Omega opened pull request 03#2106: deploy-mediawiki: add gitinfo to rsyncpaths - 13https://git.io/JPptu [17:22:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.53, 3.44 [17:22:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.01, 7.07, 5.55 [17:23:33] RhinosF1: ^ I've realised gitinfo isn't actually being synced. Only mw11 shows correct. [17:24:02] CosmicAlpha: ack [17:24:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.04, 6.73, 5.61 [17:28:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.16, 3.31, 3.38 [17:42:02] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-1/±0] 13https://git.io/JPpYd [17:42:03] [02miraheze/CreateWiki] 07Universal-Omega 03ce6f2de - Delete decline.svg [17:42:05] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [17:42:13] [02CreateWiki] 07Universal-Omega opened pull request 03#258: New decline icon - 13https://git.io/JPpYb [17:43:56] CosmicAlpha: is there nothing good in OOUI [17:44:18] Because ooui icons would be more consistent with style & design guidelines [17:45:47] miraheze/CreateWiki - Universal-Omega the build has errored. [17:54:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.65, 3.45 [17:55:40] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-0/±0] 13https://git.io/JPpsz [17:55:41] [02miraheze/CreateWiki] 07Universal-Omega 037526186 - Add new icon [17:55:43] [02CreateWiki] 07Universal-Omega synchronize pull request 03#258: New decline icon - 13https://git.io/JPpYb [17:56:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.62, 3.46 [17:58:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.28, 3.31, 3.36 [18:04:31] miraheze/CreateWiki - Universal-Omega the build passed. [18:05:47] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JPpZk [18:05:49] [02miraheze/CreateWiki] 07Universal-Omega 036b441df - Format [18:05:50] [02CreateWiki] 07Universal-Omega synchronize pull request 03#258: New decline icon - 13https://git.io/JPpYb [18:08:46] RhinosF1: that's a good point. I've never really worked much with OOUI icons but will look into them, but I might deploy the one for now, just because the current one is far to small. But if you don't think I should, I won't either. [18:09:10] CosmicAlpha: no issues as an immediate fix [18:09:31] But using the standard if it's appropiate for design would be good [18:11:13] [02CreateWiki] 07Universal-Omega commented on pull request 03#258: New decline icon - 13https://git.io/JPpnf [18:12:11] RhinosF1: yeah will look into it. [18:12:19] Cool [18:14:42] miraheze/CreateWiki - Universal-Omega the build passed. [18:30:53] [02CreateWiki] 07Universal-Omega closed pull request 03#258: New decline icon - 13https://git.io/JPpYb [18:30:55] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPpCF [18:30:56] [02miraheze/CreateWiki] 07Universal-Omega 0360e59ac - New decline icon (#258) [18:30:58] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [18:30:59] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [18:40:05] miraheze/CreateWiki - Universal-Omega the build passed. [19:19:27] [02puppet] 07AgentIsai opened pull request 03#2107: Disallow indexing /w/ / /w/index.php?* - 13https://git.io/JPpEF [19:21:36] [02puppet] 07AgentIsai synchronize pull request 03#2107: Disallow indexing /w/ / /w/index.php?* - 13https://git.io/JPpEF [19:22:44] [02puppet] 07AgentIsai edited pull request 03#2107: Disallow indexing /w/ / /w/index.php?* - 13https://git.io/JPpEF [19:26:38] [02puppet] 07AgentIsai edited pull request 03#2107: Disallow indexing /w/ / /w/index.php?* - 13https://git.io/JPpEF [19:27:29] Agent: ty [19:30:41] [02puppet] 07AgentIsai synchronize pull request 03#2107: Disallow indexing /w/ / /w/index.php?* - 13https://git.io/JPpEF [19:30:47] RhinosF1: np [19:33:02] Agent: see my review. [19:46:21] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:59:00] CosmicAlpha: Thanks. So just for clarification, the robots.txt approach should be replaced by the one you suggested via Varnish? [20:10:20] Yes [20:15:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.47, 3.27 [20:17:11] Agent: Yeah that's what I meant. I replaced it before because it caused search console errors on Bing and Google at least. [20:17:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.26, 3.40, 3.27 [20:17:48] [02puppet] 07AgentIsai commented on pull request 03#2107: Disallow indexing /w/ / /w/index.php?* - 13https://git.io/JPpoR [20:17:50] [02puppet] 07AgentIsai closed pull request 03#2107: Disallow indexing /w/ / /w/index.php?* - 13https://git.io/JPpEF [20:18:04] Agent: I can do if you want? [20:19:58] CosmicAlpha: If you want, what I was going to do is replace line 450 with "if (req.url ~ "^(/w/api\.php*|/w/index\.php\?*|/wiki/Special\:|/w/index\.php\?title\=Special%3A|/wiki/Special%3A).+$")" where the original "/w/index\.php\?title\=Special\:" was replaced with "/w/index\.php\?*" [20:20:26] can't think of any legitimate need for index.php to be ever indexed [20:22:31] Agent: could also remove the other ?title= stuff from regex. [20:25:42] [02puppet] 07AgentIsai opened pull request 03#2108: Noindex for /w/index.php?* - 13https://git.io/JPpKl [20:25:51] CosmicAlpha: ^ [20:30:51] [02puppet] 07Universal-Omega reviewed pull request 03#2108 commit - 13https://git.io/JPp6f [20:31:29] [02puppet] 07RhinosF1 reviewed pull request 03#2108 commit - 13https://git.io/JPp6L [20:31:34] A [20:31:53] Agent: I would ban api|rest|index too [20:32:00] Not just index [20:32:38] [02puppet] 07Universal-Omega reviewed pull request 03#2108 commit - 13https://git.io/JPp6C [20:33:32] [02puppet] 07Universal-Omega reviewed pull request 03#2108 commit - 13https://git.io/JPp6C [20:33:41] RhinosF1: api already is. [20:34:25] [02puppet] 07RhinosF1 reviewed pull request 03#2108 commit - 13https://git.io/JPp62 [20:34:57] CosmicAlpha: see ^ [20:35:12] Wondering if we can simplify the statement a bit [20:38:29] Ack, I keep getting a few 503s [20:41:23] [02puppet] 07Universal-Omega reviewed pull request 03#2108 commit - 13https://git.io/JPpiW [20:41:53] RhinosF1: ^ should work for simplifying [20:43:58] Lgtm CosmicAlpha [20:44:18] [02puppet] 07AgentIsai synchronize pull request 03#2108: Noindex for /w/index.php?* - 13https://git.io/JPpKl [20:45:40] I see a spike in req [20:45:43] Dura [20:45:48] Duration [20:45:53] But nothing serious [20:46:55] paladox, JohnLewis: can 2106 & 2018 be merged [21:25:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.01, 4.09, 3.43 [21:27:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.66, 3.35 [21:29:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.52, 2.98, 3.15 [21:37:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.47, 3.34 [21:39:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.27, 3.37, 3.32 [21:43:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.66, 3.49 [21:45:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.31, 3.38 [21:46:25] [02puppet] 07paladox closed pull request 03#2108: Noindex for /w/index.php?* - 13https://git.io/JPpKl [21:46:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPp9Z [21:46:28] [02miraheze/puppet] 07AgentIsai 038a3a90c - Noindex for /w/index.php?* (#2108) [21:47:06] [02puppet] 07paladox closed pull request 03#2106: deploy-mediawiki: add gitinfo to rsyncpaths - 13https://git.io/JPptu [21:47:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JPp9l [21:47:09] [02miraheze/puppet] 07Universal-Omega 039a58883 - deploy-mediawiki: add gitinfo to rsyncpaths (#2106) [21:47:24] Is it by any chance possible to change an NS (namespace) number from whatever to whatever? [21:47:40] Possibly [21:47:41] Why [21:48:20] Yes it is [21:48:24] Technically [21:48:31] Assuming it's available [21:48:36] On famepedia the current draft namespace is 3002 and draft talk 3003 instead of 816 and 817 iirc, it should be the same number as meta's number [21:49:26] Meta doesn't have a draft namespace [21:49:32] Ouch [21:49:38] Okay [21:49:39] And why should it be? [21:49:51] Why should what be? [21:50:01] The same as another wiki [21:50:10] o [21:50:18] Custom namespaces are 3000+ [21:50:39] 816 is the default Draft namespace number [21:51:02] unless it was deleted from the wiki and then recreated [21:51:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.57, 3.48 [21:52:21] Ugochimobi: no it's not [21:52:39] 816 is in the extension range [21:52:43] It's unused [21:52:52] Where are you getting that [21:53:24] Because that's what it was before the namespace was messed with [21:53:39] By who [21:53:44] And what defined it [21:54:08] MintyDocs' draft namespace is 620/1 [21:54:44] I am saying famepedia's Draft namespace number was 816 before it was messed up [21:54:58] Messed up by who/what [21:55:36] me of course [21:55:59] And what actually created it [21:56:01] :b [21:56:12] Because no one should be using 816 [21:56:19] I have no damn Idea [21:56:35] I can renumber it if it's clear why [21:56:38] But it isn't [21:56:42] [02puppet] 07Universal-Omega opened pull request 03#2109: T8230: don't set noindex on Special:WikiForum - 13https://git.io/JPpHU [21:56:48] 816 is marked as free [21:57:11] No one should be using 816? [21:57:42] No [21:57:55] If an extension is using it, I need to know which [21:58:07] So I can double check [21:58:19] And mark it on mediawiki.org [21:58:44] https://www.mediawiki.org/wiki/Extension_default_namespaces#810%E2%80%93819 [21:58:44] [url] Extension default namespaces - MediaWiki | www.mediawiki.org [21:59:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.27, 3.26, 3.33 [21:59:15] You can not set custom namespaces to anything lower then 3000. Those are reserved for core and extensions, and can't change extension namespace numbers as it will mess them up I think. [22:00:03] CosmicAlpha: I don't even know where 816 has come from [22:00:25] We can look at fixing Ugochimobi's mess if we knew what it should be [22:00:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.25, 21.17, 17.40 [22:00:57] And I assume you'd disable the extension, drop the namespace and re-enable the extension providing [22:01:12] But I can't see any evidence it should be 816 [22:03:01] RhinosF1: try var_dump($wgExtraNamespaces) on test3 since if it exists it likely would there. [22:03:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.54, 5.17, 3.81 [22:05:39] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.38, 5.33, 4.02 [22:06:13] PROBLEM - cp13 Current Load on cp13 is WARNING: WARNING - load average: 1.47, 1.72, 1.00 [22:07:12] CosmicAlpha: will look later [22:07:19] Need to be up early tommorow [22:07:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.50, 5.17, 4.13 [22:08:13] RECOVERY - cp13 Current Load on cp13 is OK: OK - load average: 0.52, 1.31, 0.94 [22:08:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.07, 23.19, 20.25 [22:09:23] And I'm more bothered about what's registering it not just whether it's there [22:10:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.58, 23.65, 20.75 [22:12:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.74, 23.49, 21.03 [22:13:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.95, 4.69, 4.35 [22:14:21] \O/ and I have no idea, but it was so, I just met it so, I just realized I wasn't even the one that messed with it, I still think so though.' [22:16:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.87, 22.46, 20.94 [22:19:39] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.13, 7.51, 5.63 [22:31:01] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.75, 7.06, 5.63 [22:33:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.71, 6.05, 5.45 [22:37:57] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.43, 3.17, 2.52 [22:39:53] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.67, 2.60, 2.38 [22:45:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.28, 4.70, 5.85 [22:51:40] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.28, 5.27, 5.60 [22:52:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.60, 6.39, 5.47 [22:53:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.96, 5.63, 5.70 [22:54:32] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.30, 1.76, 1.33 [22:54:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.93, 5.79, 5.36 [22:56:29] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.70, 1.41, 1.25 [22:56:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 13.61, 19.22, 23.54 [22:57:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.14, 2.69 [22:57:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.49, 3.23, 4.69 [22:58:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.75, 21.04, 23.59 [22:59:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 2.92, 2.66 [23:00:31] https://meta.miraheze.org/wiki/Requests_for_Stewardship#Nomination_of_John_for_Stewardship I think we may have a winner here. There's more support votes than oppose votes for John to become a Steward again. [23:00:36] [url] Requests for Stewardship - Miraheze Meta | meta.miraheze.org [23:04:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.94, 7.28, 6.03 [23:05:06] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.52, 7.78, 5.99 [23:05:24] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:18] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 7.14, 8.20, 6.71 [23:06:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.82, 23.27, 23.97 [23:08:13] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.17, 7.17, 6.51 [23:09:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.66, 6.57, 5.94 [23:09:38] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:07] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.95, 6.80, 6.45 [23:10:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.50, 7.66, 6.78 [23:12:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.83, 8.49, 7.20 [23:13:01] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.08, 9.06, 7.13 [23:14:59]