[00:05:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 3.76, 3.11 [00:07:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.73, 3.17 [00:11:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.90, 3.33 [00:13:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.78, 3.36 [00:15:31] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49149 MB (11% inode=98%); [00:17:09] [02puppet] 07Universal-Omega reviewed pull request 03#2121 commit - 13https://git.io/JXNSY [00:19:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.78, 3.45 [00:23:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:23:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb [00:25:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.88, 3.59 [00:25:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:26:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.72, 6.64, 4.99 [00:28:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.16, 6.00, 4.95 [00:31:24] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 5.38, 3.46, 2.56 [00:33:23] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.51, 3.01, 2.51 [00:33:32] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48924 MB (10% inode=98%); [00:34:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.02, 3.76 [00:36:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.52, 3.52, 3.61 [00:37:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:42:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:44:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.18, 2.91, 3.32 [00:45:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.38.69.175/cpweb, 2001:41d0:800:170b::5/cpweb [00:45:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::1d3/cpweb [00:47:07] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:47:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:50:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 54.38.211.199/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:51:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:52:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:56:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:28:39] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mgqwiki /home/universalomega/monstergirlquest_pages_full.xml --username-prefix wikia:monstergirlquest --report 1 --no-updates (END - exit=0) [01:28:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:20] PROBLEM - ping6 on jobchron1 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 212.29 ms [01:34:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.64, 3.34 [01:36:18] RECOVERY - ping6 on jobchron1 is OK: PING OK - Packet loss = 0%, RTA = 4.23 ms [01:36:52] [02puppet] 07Universal-Omega reviewed pull request 03#2121 commit - 13https://git.io/JXNSY [01:40:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.21, 20.86, 23.87 [01:42:18] [02puppet] 07Universal-Omega reviewed pull request 03#2121 commit - 13https://git.io/JXNSY [01:42:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.82, 23.80, 24.50 [01:44:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.28, 21.44, 23.59 [01:45:04] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mgqwiki (START) [01:45:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:45:31] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49080 MB (11% inode=98%); [01:48:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.63, 3.49 [01:50:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.49, 3.45 [01:52:10] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mgqwiki (END - exit=256) [01:52:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.11, 3.23, 3.40 [01:56:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.25, 22.93, 22.73 [01:59:33] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48935 MB (10% inode=98%); [01:59:47] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.22, 5.89, 4.27 [02:01:46] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.68, 5.09, 4.18 [02:02:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.20, 23.69, 23.71 [02:18:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.90, 3.84 [02:18:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.47, 18.22, 20.10 [02:22:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 4.00, 3.90 [02:36:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.04, 3.38 [02:37:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.25, 5.11, 5.86 [02:39:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.24, 5.83, 6.05 [02:41:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.28, 5.13, 5.76 [02:43:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.91, 6.05, 6.04 [02:45:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.68, 5.19, 5.73 [02:47:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.17, 5.52, 5.75 [02:50:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.49, 3.32 [02:50:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.76, 21.11, 19.68 [02:54:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.71, 3.46 [02:56:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.80, 18.74, 19.28 [02:57:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.22, 5.35, 5.90 [02:58:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.84, 3.95, 3.58 [03:00:17] SRE, [2f64d93be9caa8023a2e4353] 2021-11-13 02:59:59: Fatal exception of type "Error" on `nonciclopediawiki` [03:00:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.30, 3.37, 3.42 [03:00:35] CosmicAlpha: ^ [03:00:41] might be worth checking graylog [03:01:21] Problem persists when trying to nuke pages [03:02:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.06, 3.30 [03:03:27] dmehus: I'm mobile now, for the night. [03:07:45] I can look first thing tomorrow though, or Reception or Rhinos might be able to when they get online. [03:11:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.16, 5.60, 5.47 [03:11:54] CosmicAlpha, ack, thanks. Agent, if I'm not up when Reception123 or RhinosF1 are up, can you let them know and link them to this timestamp in the IRC logs at https://wm-bot.wmcloud.org/browser/index.php?display=%23miraheze-sre? [03:11:54] Hey dmehus, you are welcome! [03:11:55] [url] Wikimedia IRC logs browser | wm-bot.wmcloud.org [03:12:18] * dmehus also wonders if Agent ever sleeps :P [03:12:47] dmehus: sure [03:12:51] Also, I never sleep ;) [03:12:51] Agent: ty [03:12:55] heh [03:13:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.87, 5.67, 5.52 [03:13:50] I was up till 6:30AM last night, so I think I'm gonna head off entirely for tonight now. [03:14:45] dmehus: if it's not fixed by tomorrow ping me and I'll look first thing. [03:18:33] heh, just as I was replying to you, you quit CosmicAlpha [03:21:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.36, 3.21 [03:21:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.45, 5.85, 5.66 [03:23:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 6.00, 5.79, 5.65 [03:25:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.64, 3.34 [03:25:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.91, 6.17, 5.78 [03:27:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.60, 3.37 [03:28:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 3.39, 3.32 [03:31:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.41, 5.56, 5.68 [03:31:42] CosmicAlpha, okay, SGTM. Thanks :) [03:31:53] .tell CosmicAlpha, okay, SGTM. Thanks :) [03:31:54] dmehus: I'll pass that on when CosmicAlpha is around. [03:32:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.41, 3.37 [03:34:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.60, 3.44 [03:36:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.75, 3.59, 3.45 [03:38:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.90, 3.58 [03:40:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.76, 3.56 [03:42:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.85, 3.61 [03:43:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.02, 5.47, 5.39 [03:44:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.81, 3.62 [03:44:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.10, 19.29, 18.25 [03:45:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.19, 4.69, 5.12 [03:46:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.44, 17.80, 17.84 [03:48:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.88, 3.68 [03:50:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 4.00, 3.75 [03:52:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.39, 4.36, 3.91 [03:55:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.29, 4.76, 5.04 [03:56:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.75, 3.77 [04:06:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.02, 3.39 [04:09:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.14, 5.75, 5.13 [04:13:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.17, 5.09, 5.02 [04:20:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.19, 5.34, 5.15 [04:22:13] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.03, 4.81, 4.97 [04:26:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.68, 5.24, 5.10 [04:26:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.17, 19.22, 18.06 [04:28:00] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.29, 5.68, 5.28 [04:28:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.00, 18.69, 18.01 [04:31:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.56, 5.46, 5.27 [04:35:46] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.13, 6.09, 5.56 [04:39:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.32, 5.79, 5.57 [04:41:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.26, 20.39, 19.13 [04:43:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.30, 19.76, 19.06 [04:45:29] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.35, 4.11, 4.95 [04:52:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.45, 3.15 [04:54:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.27, 3.13 [05:00:39] PROBLEM - vault.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'vault.agentisai.com' expires in 7 day(s) (Sun 21 Nov 2021 04:59:53 GMT +0000). [05:02:43] PROBLEM - wiki.eversd.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.eversd.com' expires in 7 day(s) (Sun 21 Nov 2021 05:02:13 GMT +0000). [05:04:28] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pt.polandballwiki.com' expires in 7 day(s) (Sun 21 Nov 2021 05:03:56 GMT +0000). [05:08:59] PROBLEM - polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.polandballwiki.com' expires in 7 day(s) (Sun 21 Nov 2021 05:07:16 GMT +0000). [05:09:45] PROBLEM - www.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.polandballwiki.com' expires in 7 day(s) (Sun 21 Nov 2021 05:07:16 GMT +0000). [05:31:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.46, 3.26 [05:33:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.23, 3.20 [05:34:38] PROBLEM - wiki.yapsavun.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.yapsavun.xyz' expires in 7 day(s) (Sun 21 Nov 2021 05:29:51 GMT +0000). [05:42:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.39, 3.26 [05:46:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.32, 3.24 [05:50:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.66, 4.03, 3.50 [05:54:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.74, 3.51 [05:56:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.04, 3.11, 3.31 [06:06:36] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.52, 6.80, 3.99 [06:08:36] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.67, 6.90, 4.36 [06:10:37] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.19, 7.84, 5.03 [06:12:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.52, 3.80, 3.25 [06:12:38] PROBLEM - db13 APT on db13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:22] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 33.21, 13.07, 6.37 [06:13:23] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=monstergirlencyclopediawiki /home/reception/monstergirlencyclopedia_pages_full.xml --username-prefix wikia:monstergirlencyclopedia --report 1 --no-updates (END - exit=256) [06:13:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.48, 3.19 [06:14:33] RECOVERY - db13 APT on db13 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:17:19] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 2.02, 6.85, 5.32 [06:19:19] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 2.15, 5.32, 4.95 [06:19:31] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 51206 MB (11% inode=98%); [06:20:36] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.20, 7.50, 6.44 [06:22:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.34, 3.30 [06:25:31] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 47111 MB (10% inode=98%); [06:26:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.45, 3.37 [06:26:36] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.40, 6.52, 6.36 [06:28:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.54, 2.79, 3.14 [06:48:36] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.23, 6.51, 5.88 [06:50:36] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.63, 6.53, 5.97 [07:11:25] PROBLEM - staff.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'staff.polandballwiki.com' expires in 15 day(s) (Mon 29 Nov 2021 07:06:05 GMT +0000). [07:16:31] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=monstergirlencyclopediawiki /home/reception/monstergirlencyclopedia_pages_full.xml --username-prefix wikia:monstergirlencyclopedia --report 1 --no-updates (START) [07:16:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:41] PROBLEM - reviews.thejoshmeister.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'reviews.thejoshmeister.com' expires in 7 day(s) (Sun 21 Nov 2021 08:23:11 GMT +0000). [09:45:10] [02puppet] 07RhinosF1 reviewed pull request 03#2121 commit - 13https://git.io/JXxXz [09:45:24] [02puppet] 07RhinosF1 reviewed pull request 03#2121 commit - 13https://git.io/JXxXz [09:45:40] [02puppet] 07RhinosF1 reviewed pull request 03#2121 commit - 13https://git.io/JXxX6 [09:48:46] [02puppet] 07Universal-Omega reviewed pull request 03#2121 commit - 13https://git.io/JXNSY [09:51:49] RhinosF1: how would that first one be of zero benefit? Isn't it nicer code format? Anyways it's why I just commented on it rather than doing as I didn't know for sure. [09:51:50] [tell] CosmicAlpha: 2021-11-13 - 03:31:53UTC tell CosmicAlpha okay, SGTM. Thanks :) [09:52:21] CosmicAlpha: i deleted it 10 seconds later after realising it's same variable [09:52:55] RhinosF1: oh, I just saw in email. So should that be done then or not? [09:53:21] If the variable is used more than once in that format ye [09:55:49] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [09:58:20] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [09:59:19] [02puppet] 07Universal-Omega edited pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [09:59:42] RhinosF1: ^ should be completely done now. [09:59:58] K [10:00:27] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [10:00:29] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [10:01:22] [02puppet] 07Universal-Omega reviewed pull request 03#2121 commit - 13https://git.io/JXNSY [10:01:23] CosmicAlpha: simply if confirm should work [10:02:19] RhinosF1: then it would always be present, if anything is typed, even if the input is not `y` wouldn't it? [10:02:34] CosmicAlpha: ah right [10:03:11] CosmicAlpha: I think the line above is unneeded [10:03:57] Just do if args.confirm or (input().lower() == 'y'): [10:04:08] Or upper [10:06:05] RhinosF1: `if args.confirm or input("Type 'Y' to confirm: ").upper() == 'Y':`? [10:06:27] CosmicAlpha: yes [10:06:47] [02puppet] 07Universal-Omega synchronize pull request 03#2121: mwscript: rewrite to use argparse - 13https://git.io/JXQhx [10:07:37] RhinosF1: ^ done. [10:08:09] Ty [10:08:42] No problem! [10:10:12] RhinosF1: also the PR description I updated today to include detailed information for all additional changes this would make. I'm gonna head off for tonight now though. [10:10:34] CosmicAlpha: night then! [10:10:54] Thanks! [10:29:51] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JXx7Q [10:29:52] [02miraheze/dns] 07Reception123 031df5d6b - add hypotheticalhurricanes.com zone [10:32:56] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hypotheticalhurricaneswiki /home/reception/hypotheticalhurricanes_pages_full.xml --username-prefix wikia:hypotheticalhurricanes --report 1 (START) [10:32:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:33:02] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hypotheticalhurricaneswiki /home/reception/hypotheticalhurricanes_pages_full.xml --username-prefix wikia:hypotheticalhurricanes --report 1 (END - exit=2) [10:33:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:33:07] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hypotheticalhurricaneswiki /home/reception/hypotheticalhurricanes_pages_full.xml --username-prefix wikia:hypotheticalhurricanes --report 1 --no-updates (START) [10:33:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:13] PROBLEM - mwtask1 Disk Space on mwtask1 is WARNING: DISK WARNING - free space: / 3581 MB (7% inode=90%); [10:39:55] Reception123: ^ [10:43:50] RhinosF1: I know, another large dump I'm afraid [10:46:20] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=polandballwiki /home/reception/polandball.txt (START) [10:46:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:52:22] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=polandballwiki /home/reception/polandball.txt (END - exit=0) [10:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.13, 5.87, 4.39 [11:52:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.88, 5.20, 4.33 [12:29:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.34, 2.97 [12:33:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 2.99, 2.93 [12:49:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.39, 3.18 [12:53:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.03, 3.09 [13:36:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.49, 3.11 [13:38:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.04, 2.99 [13:51:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.63, 3.26 [13:57:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.63, 3.37 [13:59:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.96, 3.95, 3.50 [14:02:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.63, 3.49 [14:04:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.70, 3.29, 3.38 [14:08:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.99, 3.65 [14:22:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.77, 3.91 [14:36:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.58, 3.63 [14:38:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.34, 3.03, 3.42 [14:40:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 2.96, 3.35 [14:51:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.06, 3.65 [14:59:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.88, 3.74 [15:02:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.12, 4.22, 3.89 [15:04:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.69, 3.74 [15:06:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.76, 3.75 [15:07:25] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'miraheze.wiki' expires in 7 day(s) (Sun 21 Nov 2021 15:00:32 GMT +0000). [15:08:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.85, 3.79 [15:20:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 2.88, 3.32 [15:24:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 3.97, 3.67 [15:26:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.27, 3.59, 3.56 [15:31:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.34, 7.95, 6.49 [15:34:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.69, 3.61 [15:35:28] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.32, 6.22, 6.10 [15:38:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.76, 3.67 [15:46:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.93, 8.06, 6.32 [15:46:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.01, 3.77 [15:48:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.47, 6.63, 6.01 [15:48:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.58, 3.64 [15:50:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 4.04, 3.80 [15:52:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.94, 6.47, 5.47 [15:52:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.75, 6.96, 6.32 [15:53:59] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.17, 5.57, 5.25 [15:54:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.83, 5.87, 6.00 [15:58:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.88, 3.96 [16:00:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 4.20, 4.07 [16:12:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.73, 3.92 [16:20:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 4.06, 3.93 [16:22:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.59, 3.77 [16:28:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.08, 4.09, 3.88 [16:32:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.22, 3.42, 3.67 [16:38:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.71, 3.68 [16:40:25] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.44, 6.28, 5.09 [16:42:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.74, 6.30, 5.24 [16:42:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 3.52, 3.63 [16:48:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.12, 3.39 [16:52:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 4.07, 3.74 [16:56:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.68, 3.41, 3.54 [17:03:28] " heh, just as I was replying to you, you quit CosmicAlpha" oh hey lol yeah I usually dont even disconnect but decided to last night. [17:04:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.71, 3.60 [17:05:24] CosmicAlpha: yeah, I took too long writing my response either way to your message so it's fine [17:14:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.37, 3.46, 3.63 [17:16:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 4.00, 3.80 [17:18:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.50, 3.64 [17:24:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.38, 3.89, 3.69 [17:30:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.87, 3.81 [17:46:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.92, 3.73 [17:48:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.60, 3.64 [17:50:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.16, 4.06, 3.80 [17:51:45] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.72, 7.81, 5.42 [17:52:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.42, 6.97, 4.90 [17:52:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.77, 3.72 [17:53:45] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.35, 7.63, 5.69 [17:54:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.99, 6.24, 4.90 [17:54:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.35, 4.26, 3.90 [17:55:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.51, 6.58, 5.54 [17:56:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.94, 3.83 [17:58:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.90, 3.81 [18:00:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.84, 3.80 [18:06:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.48, 3.60 [18:12:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.77, 3.44, 3.60 [18:20:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 3.73, 3.61 [18:22:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.85, 3.68 [18:30:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.12, 3.39 [18:57:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.47, 3.26 [18:59:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 3.10, 3.15 [19:03:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.48, 3.31 [19:04:08] [02dns] 07RhinosF1 opened pull request 03#227: mh.org: add mwtask CNAME to active task - 13https://git.io/JXhix [19:04:23] paladox: ^ [19:04:53] [02dns] 07paladox closed pull request 03#227: mh.org: add mwtask CNAME to active task - 13https://git.io/JXhix [19:04:54] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JXhPq [19:04:56] [02miraheze/dns] 07RhinosF1 03fa63cff - mh.org: add mwtask CNAME to active task (#227) [19:05:26] Ty! [19:05:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.19, 3.21 [19:06:16] thanks, paladox [19:09:31] yw [19:09:50] [02puppet] 07Universal-Omega opened pull request 03#2123: Remove services HostGroup - 13https://git.io/JXhXq [19:13:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.73, 3.45 [19:15:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.33, 3.33 [19:17:23] [02puppet] 07Universal-Omega opened pull request 03#2124: Remove Role::Services reference - 13https://git.io/JXh1W [19:21:59] CosmicAlpha, aren't you supposed to batch your commits onto one PR or something? [19:22:08] I think I remember John saying that's preferred [19:22:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 4.19, 3.67 [19:23:16] [02puppet] 07dmehus commented on pull request 03#2124: Remove Role::Services reference - 13https://git.io/JXhM0 [19:23:17] dmehus: it's 2 different things. [19:23:54] CosmicAlpha, both are DNS related, though. Why not batch them together as a DNS cleanup PR? [19:24:47] dmehus: It was easier this way anyways since I wasn't browsing on my fork when I found the second. And they both are services related, but 2 separate places, I don't even know what I'd call a combined PR. [19:25:29] And John meant when merging to squash and merge rather then a commit for every commit in PR. [19:25:55] dmehus: they are 2 very different things and not dns related [19:26:12] One is dropping the icinga hostgrouo [19:26:23] One is dropping the auto firewall rules [19:26:34] Yep. [19:27:11] [02puppet] 07Universal-Omega commented on pull request 03#2124: Remove Role::Services reference - 13https://git.io/JXhDW [19:29:30] [02dns] 07JohnFLewis commented on pull request 03#227: mh.org: add mwtask CNAME to active task - 13https://git.io/JXhDD [19:31:15] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zymonic.com All nameservers failed to answer the query. [19:33:00] ^ doesn't seem to point to us anymore [19:33:04] or point anywhere [19:34:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.05, 3.41, 3.72 [19:36:24] [02dns] 07dmehus commented on pull request 03#227: mh.org: add mwtask CNAME to active task - 13https://git.io/JXhyN [19:36:35] [02dns] 07dmehus edited a comment on pull request 03#227: mh.org: add mwtask CNAME to active task - 13https://git.io/JXhyN [19:40:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.06, 4.05, 3.87 [19:42:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.67, 3.74 [19:45:12] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [19:46:26] [02dns] 07RhinosF1 commented on pull request 03#227: mh.org: add mwtask CNAME to active task - 13https://git.io/JXhHe [19:46:50] JohnLewis: even the WMF only have one maint server per DC to my knowledge [19:47:33] I’ll remove the resource allocation from the plan then to free up more elsewhere [19:54:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.45, 2.80, 3.32 [20:06:10] JohnLewis: did you plan 2? [20:06:31] I w [20:06:39] I would just do 1 [20:08:57] Apart from disk [20:09:04] Task is never strained [20:12:04] Yeah, there were two in the plan [20:19:49] If anything more disk might be useful but I don't see a benefit to 2 [20:19:57] We never get high usage on them [20:20:04] And any usage is Sysadmin controlled [20:20:19] Unless like I say you had multi DC [20:29:52] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.05, 6.68, 5.24 [20:31:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.20, 5.60, 5.00 [20:49:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.45, 3.16 [20:55:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.31, 3.21 [20:58:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.34, 6.62, 5.55 [21:00:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.35, 6.02, 5.45 [21:07:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.20, 3.08 [21:09:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.10, 3.97, 3.39 [21:11:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.81, 3.53, 3.29 [21:13:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 3.06, 3.15 [21:25:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.43, 3.20 [21:27:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.63, 3.30 [21:29:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.94, 3.30, 3.21 [21:30:31] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.06, 7.19, 5.84 [21:32:31] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.41, 5.77, 5.48 [21:35:14] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78555 MB (10% inode=99%); [21:37:14] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 78982 MB (11% inode=99%); [21:41:14] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78632 MB (10% inode=99%); [21:53:14] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 78888 MB (11% inode=99%); [21:57:14] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78354 MB (10% inode=99%); [22:15:07] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JXj3t [22:15:08] [02miraheze/CreateWiki] 07Universal-Omega 03d6fbcc9 - Remove unused i18n [22:15:10] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [22:15:11] [02CreateWiki] 07Universal-Omega opened pull request 03#261: Remove unused i18n - 13https://git.io/JXj3m [22:21:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.21, 2.81 [22:23:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 2.80, 2.70 [22:31:32] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JXjZq [22:31:33] [02miraheze/MirahezeMagic] 07Universal-Omega 038303b12 - Add reCaptcha response scores to debug logging [22:31:35] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [22:31:38] [02MirahezeMagic] 07Universal-Omega opened pull request 03#303: Add reCaptcha response scores to debug logging - 13https://git.io/JXjZm [22:32:38] miraheze/MirahezeMagic - Universal-Omega the build passed. [22:34:19] miraheze/CreateWiki - Universal-Omega the build passed. [22:38:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.71, 3.18 [22:40:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.34, 3.11 [22:49:31] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26683 MB (5% inode=98%); [22:51:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.19, 4.32, 3.62 [22:55:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.79, 3.57 [22:57:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.24, 3.27, 3.40 [23:00:03] !log [@mw11] starting deploy of {'l10nupdate': True} to all [23:00:05] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 38.61, 25.31, 17.72 [23:04:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 15.37, 8.22, 5.50 [23:05:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.44, 7.14, 5.47 [23:05:13]