[00:02:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimHK [00:02:30] [02miraheze/puppet] 07paladox 036f99f3d - services: Fix sorting ferm rules [00:02:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.85, 19.40, 16.26 [00:03:50] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.74, 3.03, 2.23 [00:04:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.00, 22.06, 17.59 [00:05:50] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.35, 3.44, 2.47 [00:06:00] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.91, 20.89, 17.64 [00:06:40] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [00:07:50] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.16, 3.53, 2.64 [00:08:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.00, 19.80, 17.69 [00:08:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.68, 21.23, 18.43 [00:09:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JimQm [00:09:03] [02miraheze/puppet] 07paladox 03fd9c912 - graylog: Fix sorting ferm rules [00:09:05] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:09:06] [02puppet] 07paladox opened pull request 03#2093: graylog: Fix sorting ferm rules - 13https://git.io/JimQY [00:09:40] [02puppet] 07paladox closed pull request 03#2093: graylog: Fix sorting ferm rules - 13https://git.io/JimQY [00:09:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimQZ [00:09:43] [02miraheze/puppet] 07paladox 0332b254e - graylog: Fix sorting ferm rules (#2093) [00:09:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:09:46] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:09:50] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.34, 3.16, 2.61 [00:10:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.46, 23.19, 19.42 [00:11:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JimQ0 [00:11:01] [02miraheze/puppet] 07paladox 033408223 - memcached: Fix sorting ferm rules [00:11:03] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:11:04] [02puppet] 07paladox opened pull request 03#2094: memcached: Fix sorting ferm rules - 13https://git.io/JimQu [00:11:09] [02puppet] 07paladox closed pull request 03#2094: memcached: Fix sorting ferm rules - 13https://git.io/JimQu [00:11:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimQ2 [00:11:12] [02miraheze/puppet] 07paladox 039f31ef6 - memcached: Fix sorting ferm rules (#2094) [00:11:14] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:11:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:13:17] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.42, 5.97, 4.29 [00:14:11] !log reboot graylog2 [00:14:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jim7f [00:15:10] [02miraheze/puppet] 07paladox 03b087f2f - gluster: Fix sorting ferm rules [00:16:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 13.44, 23.22, 21.23 [00:17:09] huh [00:17:09] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.37, 4.28, 4.03 [00:17:11] metas down [00:17:48] It's back up @paladox. [00:17:50] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 9.25, 3.67, 1.38 [00:18:09] PROBLEM - cp12 HTTPS on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 5613 bytes in 0.324 second response time [00:18:10] PROBLEM - cp15 Stunnel Http for mw10 on cp15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:19] Ugh, damn it, it's so slow. [00:18:24] PROBLEM - cp13 Stunnel Http for mw11 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jim7V [00:18:36] [02miraheze/puppet] 07paladox 035dfa562 - Fix [00:18:37] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 6 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 [00:18:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 9.14, 18.76, 19.85 [00:18:53] PROBLEM - cp15 Varnish Backends on cp15 is CRITICAL: 2 backends are down. mw11 mw13 [00:19:02] PROBLEM - cp14 Stunnel Http for mw11 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:19:14] PROBLEM - cp14 Varnish Backends on cp14 is WARNING: No backends detected. If this is an error, see readme.txt [00:19:18] PROBLEM - cp14 HTTPS on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.046 second response time [00:19:26] fixed [00:19:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:19:33] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:19:42] PROBLEM - cp12 Stunnel Http for mw11 on cp12 is CRITICAL: HTTP CRITICAL - No data received from host [00:19:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 54.38.211.199/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:19:50] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.19, 3.20, 1.49 [00:20:08] RECOVERY - cp15 Stunnel Http for mw10 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.305 second response time [00:20:13] RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3051 bytes in 9.646 second response time [00:20:16] PROBLEM - cp14 HTTP 4xx/5xx ERROR Rate on cp14 is CRITICAL: CRITICAL - NGINX Error Rate is 94% [00:20:20] RECOVERY - cp13 Stunnel Http for mw11 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.005 second response time [00:20:53] RECOVERY - cp15 Varnish Backends on cp15 is OK: All 9 backends are healthy [00:21:01] RECOVERY - cp14 Stunnel Http for mw11 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.015 second response time [00:21:31] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 0.162 second response time [00:21:39] PROBLEM - test3 Check Gluster Clients on test3 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [00:21:42] RECOVERY - cp12 Stunnel Http for mw11 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.332 second response time [00:21:53] !log restart varnish on cp14 (crashed) [00:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:16] PROBLEM - cp14 HTTP 4xx/5xx ERROR Rate on cp14 is WARNING: WARNING - NGINX Error Rate is 48% [00:22:37] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [00:22:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.47, 24.63, 21.83 [00:22:57] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.03, 5.98, 4.59 [00:23:14] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [00:24:16] RECOVERY - cp14 HTTP 4xx/5xx ERROR Rate on cp14 is OK: OK - NGINX Error Rate is 11% [00:24:28] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is WARNING: WARNING - NGINX Error Rate is 46% [00:24:29] PROBLEM - cp12 Stunnel Http for mw13 on cp12 is CRITICAL: HTTP CRITICAL - No data received from host [00:24:31] PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: HTTP CRITICAL - No data received from host [00:24:38] Errors globally I think 502/503 [00:25:23] RECOVERY - cp14 HTTPS on cp14 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3061 bytes in 0.009 second response time [00:25:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:25:31] PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:25:32] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.42, 4.47, 3.23 [00:25:33] PROBLEM - cp14 Stunnel Http for mw8 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:25:41] PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:25:55] PROBLEM - cp12 Stunnel Http for mw9 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:24] RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 18% [00:26:37] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8 [00:27:28] RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15860 bytes in 0.315 second response time [00:27:31] !log reboot mw11 [00:27:31] RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15860 bytes in 0.006 second response time [00:27:32] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.90, 4.41, 3.37 [00:27:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:35] RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15868 bytes in 0.005 second response time [00:27:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:27:49] RECOVERY - cp12 Stunnel Http for mw9 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15860 bytes in 0.618 second response time [00:28:23] RECOVERY - cp12 Stunnel Http for mw13 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.586 second response time [00:28:31] !log reboot mw12 [00:28:37] RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.314 second response time [00:28:52] !log reboot mw8 [00:29:10] !log reboot mw9 [00:29:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:29:20] !log reboot mw10 [00:29:36] !log reboot mw13 [00:30:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:30:54] PROBLEM - cp15 Varnish Backends on cp15 is CRITICAL: 2 backends are down. mw9 mw13 [00:31:02] PROBLEM - cp13 Stunnel Http for mw12 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:03] PROBLEM - cp15 Stunnel Http for mw12 on cp15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:31:07] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:16] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 2 backends are down. mw12 mw13 [00:31:40] ok, what is happening [00:31:41] PROBLEM - cp15 Stunnel Http for mw13 on cp15 is CRITICAL: HTTP CRITICAL - No data received from host [00:31:45] PROBLEM - mw13 APT on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [00:31:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:31:48] PROBLEM - cp13 Stunnel Http for mw13 on cp13 is CRITICAL: HTTP CRITICAL - No data received from host [00:31:54] huh [00:31:56] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: connect to address 51.195.236.251 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:31:57] PROBLEM - mw13 NTP time on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [00:32:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:32:36] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw13 [00:32:48] I filed a task [[phab:T8201]] [00:32:50]

https://phabricator.miraheze.org/T8201 [00:32:51] [url] ⚓ T8201 503 Backend fetch failed on Wikis | phabricator.miraheze.org [00:33:06] RECOVERY - cp15 Stunnel Http for mw12 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 6.812 second response time [00:33:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 167.114.2.161/cpweb, 2607:5300:201:3100::1d3/cpweb [00:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:14] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [00:33:36] RECOVERY - cp15 Stunnel Http for mw13 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.329 second response time [00:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:42] RECOVERY - cp13 Stunnel Http for mw13 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.021 second response time [00:33:45] RECOVERY - mw13 APT on mw13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:33:56] RECOVERY - mw13 NTP time on mw13 is OK: NTP OK: Offset -0.002852708101 secs [00:34:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:36] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [00:34:54] RECOVERY - cp15 Varnish Backends on cp15 is OK: All 9 backends are healthy [00:35:13] RECOVERY - cp13 Stunnel Http for mw12 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.897 second response time [00:35:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 7.935 second response time [00:35:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:35:28] things seem to be working now [00:35:32] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.67, 4.81, 3.89 [00:35:39] RECOVERY - test3 Check Gluster Clients on test3 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:35:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:36:00] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.13, 20.99, 17.69 [00:36:12] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 9.028 second response time [00:36:37] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [00:37:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:37:32] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.63, 4.92, 4.06 [00:38:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.24, 20.35, 17.85 [00:38:34] hmm [00:38:50] CosmicAlpha: i see redis errors in https://graylog.miraheze.org/messages/graylog_342/bb479940-3398-11ec-8db1-0200001a24a4 possibly related to your script? [00:38:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:39:57] paladox: I don't see how it would be related. But possible I guess [00:40:14] i wonder how we fix that [00:40:18] and what the actual issue is [00:41:32] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.35, 5.99, 4.71 [00:42:32] paladox: that was because things went down, so it couldn't connect to it, I'd think? [00:42:49] I'm not sure because redis would be unaffected. [00:42:55] as it's on a seperate host [00:43:32] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.68, 5.14, 4.55 [00:45:33] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 5.02, 4.95, 4.54 [00:47:37] PROBLEM - jobchron1 Redis Process on jobchron1 is CRITICAL: PROCS CRITICAL: 2 processes with args 'redis-server' [00:48:52] hmm [00:48:57] wasn't me [00:49:18] https://www.irccloud.com/pastebin/hjErdiuT/ [00:49:37] RECOVERY - jobchron1 Redis Process on jobchron1 is OK: PROCS OK: 1 process with args 'redis-server' [00:50:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimdO [00:50:35] [02miraheze/mw-config] 07paladox 031ea6f37 - Increase redis timeout for jobrunner [00:51:34] !log [paladox@mw11] starting deploy of {'config': True, 'gitinfo': True} to all [00:51:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:42] miraheze/mw-config - paladox the build passed. [00:52:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.41, 19.84, 23.61 [00:52:53] !log [paladox@mw11] finished deploy of {'config': True, 'gitinfo': True} to all - SUCCESS in 78s [00:52:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:14] didn't fix it :/ [00:59:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.09, 2.87 [01:00:38] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.58, 4.50, 5.77 [01:01:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.36, 3.25, 2.96 [01:03:15] !log [@test3] starting deploy of {'config': True} to skip [01:03:16] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:03:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.67, 3.19 [01:08:38] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.49, 3.84, 4.93 [01:08:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.81, 18.13, 20.39 [01:09:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 2.98, 2.99 [01:12:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JimF4 [01:12:27] [02miraheze/mw-config] 07Universal-Omega 0357df2b3 - Disable AutoCreatePage [01:12:28] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [01:12:30] [02mw-config] 07Universal-Omega opened pull request 03#4174: Disable AutoCreatePage - 13https://git.io/JimFB [01:13:32] miraheze/mw-config - Universal-Omega the build passed. [01:22:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.75, 3.35 [01:23:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JimbJ [01:23:33] [02miraheze/mw-config] 07paladox 0355ab784 - Revert "Increase redis timeout for jobrunner" [01:24:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.83, 3.42 [01:24:38] miraheze/mw-config - paladox the build passed. [01:25:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.48, 3.34 [01:27:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.29, 3.29 [01:30:38] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.33, 4.74, 4.21 [01:32:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.85, 4.42, 4.17 [01:33:25] !log [@test3] starting deploy of {'config': True} to skip [01:33:26] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:33:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:50] !log [@mw11] starting deploy of {'config': True} to all [01:34:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:57] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 7s [01:35:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.52, 3.29 [01:53:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.22, 3.20 [02:07:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.40, 3.20 [02:09:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.84, 2.90, 3.05 [02:38:51] !log [universalomega@mwtask1] Finished running checkStorage.php --fix on all wikis [02:39:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:27] ^ seemed that failed to run on 81 wikis... [03:07:43] !log [universalomega@mwtask1] Started running checkStorage.php --fix on all wikis [03:07:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:09] !log universalomega@mwtask1:~$ sudo -u www-data rm -f /home/reception/wikibackups16102021/wikibackups/wikitrashwiki.xml.gz [03:10:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:29] !log [universalomega@mwtask1] Started running checkStorage.php --fix on all wikis [03:10:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.46, 3.06 [03:13:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.98, 3.30, 3.05 [03:13:52] hmm.... https://www.irccloud.com/pastebin/K1WfDbln/ [03:15:01] paladox: ^ any ideas why? That failed on the remaining wikis, though it is probably fine to skip, as it is unlikely they were having the same issues. [03:15:23] nope :/ [03:19:59] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JiYfW [03:20:00] [02miraheze/mediawiki] 07Universal-Omega 0363dba48 - Update Cosmos [03:20:46] !log [universalomega@mw11] starting deploy of {'world': True, 'gitinfo': True} to all [03:20:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JiYfV [03:21:50] [02miraheze/mediawiki] 07Universal-Omega 0364c0f8e - Update Cosmos [03:23:05] !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [03:23:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:20] !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 314s [03:28:23] !log [universalomega@mw11] finished deploy of {'world': True, 'gitinfo': True} to all - SUCCESS in 457s [03:28:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.45, 3.32 [03:42:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.70, 3.42 [03:44:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 3.30, 3.31 [03:49:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.64, 3.45 [03:53:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.26, 3.36 [04:53:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiYml [04:53:04] [02miraheze/mw-config] 07Universal-Omega 039e20021 - Remove wgTabberNeueEnableMD5Hash [04:54:05] miraheze/mw-config - Universal-Omega the build passed. [05:03:48] !log [@test3] starting deploy of {'config': True} to skip [05:03:49] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [05:03:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:38] !log [@mw11] starting deploy of {'config': True} to all [05:04:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:45] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 7s [05:04:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:42:06] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 110.94 ms [05:42:23] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 112.56 ms [06:07:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.12, 6.14, 4.04 [06:09:05] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:13:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.27, 6.58, 4.95 [06:19:05] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:19:59] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JiYzg [07:20:00] [02miraheze/MirahezeMagic] 07Universal-Omega 03fcb37f1 - RebuildVersionCache: remove the `--use-staging` argument [07:20:02] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/fQRGX [07:20:05] [02MirahezeMagic] 07Universal-Omega opened pull request 03#299: RebuildVersionCache: remove the `--use-staging` argument - 13https://git.io/JiYza [07:21:01] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#298: rebuildVersion: drop --use-staging - 13https://git.io/JiYzH [07:21:03] [02MirahezeMagic] 07Universal-Omega closed pull request 03#298: rebuildVersion: drop --use-staging - 13https://git.io/J67fR [07:21:03] miraheze/MirahezeMagic - Universal-Omega the build passed. [07:21:05] [02MirahezeMagic] 07Universal-Omega deleted branch 03RhinosF1-patch-1 - 13https://git.io/fQRGX [07:21:06] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03RhinosF1-patch-1 [08:05:19] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#299: RebuildVersionCache: remove the `--use-staging` argument - 13https://git.io/JiYo5 [08:12:33] [02mw-config] 07Universal-Omega commented on pull request 03#4171: otherProjectsLinks on FAMEPedia and FAMEData - 13https://git.io/JiYKp [08:12:34] [02mw-config] 07Universal-Omega closed pull request 03#4171: otherProjectsLinks on FAMEPedia and FAMEData - 13https://git.io/JiIfW [08:14:21] [02mw-config] 07Universal-Omega closed pull request 03#4174: Disable AutoCreatePage - 13https://git.io/JimFB [08:14:23] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [08:14:24] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [08:17:04] [02mw-config] 07Universal-Omega closed pull request 03#4150: Remove unnecessary configs - 13https://git.io/JKnee [08:17:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JiY6V [08:17:07] [02miraheze/mw-config] 07Universal-Omega 03dc974a8 - Remove unnecessary configs (#4150) [08:17:08] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [08:17:10] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [08:18:07] miraheze/mw-config - Universal-Omega the build passed. [08:27:14] [02mw-config] 07ugochimobi commented on pull request 03#4171: otherProjectsLinks on FAMEPedia and FAMEData - 13https://git.io/JiYPI [08:33:15] !log [@test3] starting deploy of {'config': True} to skip [08:33:16] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:33:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:34:25] !log [@mw11] starting deploy of {'config': True} to all [08:34:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:34:35] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 10s [08:34:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:06] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.73, 7.48, 5.14 [11:44:06] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.19, 5.97, 4.86 [11:51:32] PROBLEM - wiki.yumeka.icu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yumeka.icu' expires in 15 day(s) (Mon 08 Nov 2021 11:46:22 GMT +0000). [12:02:58] PROBLEM - civwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'civwiki.org' expires in 15 day(s) (Mon 08 Nov 2021 11:58:53 GMT +0000). [12:12:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [12:12:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:12:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:201:3100::1d3/cpweb [12:12:52] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 54.23, 31.47, 19.66 [12:12:57] Here [12:13:04] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 15.39, 9.09, 4.80 [12:13:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.17, 8.68, 5.19 [12:13:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.19, 7.80, 5.09 [12:13:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.07, 7.56, 5.01 [12:13:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.78, 6.43, 4.32 [12:14:04] It'll calm down [12:14:06] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.93, 8.18, 5.70 [12:14:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:14:07] Just workers high [12:14:10] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.45, 8.15, 5.36 [12:14:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:15:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.92, 7.07, 5.03 [12:15:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.18, 6.27, 4.85 [12:15:46] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.95, 6.14, 4.80 [12:15:50] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.90, 5.36, 4.18 [12:16:06] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.73, 6.47, 5.37 [12:16:10] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.07, 6.24, 5.00 [12:16:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.82, 5.68, 4.36 [12:17:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:17:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 2.48, 5.52, 4.72 [12:18:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.74, 21.75, 19.23 [12:18:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.72, 4.77, 4.18 [12:20:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.32, 19.35, 18.66 [13:21:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.25, 2.94 [13:23:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.94, 3.07, 2.91 [13:44:51] [465b4bad2668afa47163b213] 2021-10-23 13:42:45: Fatal exception of type "Wikimedia\Assert\ParameterTypeException Error on FAMEPedia... [13:51:01] When viewing Diff https://en.famepedia.org/w/index.php?title=FAMEPedia%20talk:New%20pages%20patrol/Patrollers&diff=prev&oldid=52910 [13:51:04] [url] Internal error - FAMEPedia | en.famepedia.org [14:21:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.62, 3.11 [14:23:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.41, 3.09 [14:24:07] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JiOw5 [14:24:09] [02miraheze/ssl] 07Reception123 03aa790fa - add www.plan.eu.org cert [14:25:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.24, 3.07 [14:52:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.49, 19.89, 17.25 [14:54:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.79, 17.23, 16.57 [15:06:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.37, 23.67, 19.31 [15:07:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.52, 6.97, 5.15 [15:08:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.16, 22.01, 19.24 [15:08:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.49, 6.25, 5.11 [15:10:37] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.62, 19.99, 18.82 [15:18:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.10, 2.86 [15:20:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.50, 3.03 [15:21:59] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.27, 3.00 [15:45:28] PROBLEM - phabdigests.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bots.miraheze.wiki' expires in 15 day(s) (Mon 08 Nov 2021 15:41:49 GMT +0000). [15:45:30] PROBLEM - sopel.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bots.miraheze.wiki' expires in 15 day(s) (Mon 08 Nov 2021 15:41:49 GMT +0000). [15:46:56] PROBLEM - wildcard.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bots.miraheze.wiki' expires in 15 day(s) (Mon 08 Nov 2021 15:41:49 GMT +0000). [15:47:02] PROBLEM - phab-storage.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bots.miraheze.wiki' expires in 15 day(s) (Mon 08 Nov 2021 15:41:49 GMT +0000). [15:47:05] PROBLEM - icinga.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bots.miraheze.wiki' expires in 15 day(s) (Mon 08 Nov 2021 15:41:49 GMT +0000). [15:49:39] PROBLEM - phab.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bots.miraheze.wiki' expires in 15 day(s) (Mon 08 Nov 2021 15:41:49 GMT +0000). [15:50:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.61, 3.31 [15:52:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.36, 3.25 [15:52:55] Reception123: are we bothered about bots legacy redirects? [15:54:02] [02ssl] 07RhinosF1 opened pull request 03#435: remove legacy bots redirects - 13https://git.io/JiO5J [15:54:49] [02ssl] 07RhinosF1 synchronize pull request 03#435: remove legacy bots redirects - 13https://git.io/JiO5J [15:55:16] [02ssl] 07RhinosF1 synchronize pull request 03#435: remove legacy bots redirects - 13https://git.io/JiO5J [15:56:59] [02dns] 07RhinosF1 opened pull request 03#225: rm legacy bots stuff - 13https://git.io/JiO5z [16:05:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.11, 21.58, 18.74 [16:05:42] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.06, 5.28, 3.98 [16:05:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.63, 3.37 [16:07:25] Spookreeeno: oh you mean ones that aren't needed anymore? [16:07:40] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.27, 4.40, 3.81 [16:07:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.27, 18.98, 18.12 [16:07:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.85, 2.97, 3.16 [16:29:24] Reception123: don't see need anymore no [16:29:36] We kept them to make google transfer [16:29:39] ok then we can get rid of them [16:29:41] That is now done [16:29:56] Merge SSL before DNS pls [16:30:00] Reception123: ^ [16:30:29] [02ssl] 07Reception123 closed pull request 03#435: remove legacy bots redirects - 13https://git.io/JiO5J [16:30:30] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±2] 13https://git.io/JiOxb [16:30:32] [02miraheze/ssl] 07RhinosF1 03571dc6a - remove legacy bots redirects (#435) [16:33:15] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiOpB [16:33:16] [02miraheze/dns] 07RhinosF1 039081e0d - rm legacy bots stuff (#225) [16:33:18] [02dns] 07Reception123 closed pull request 03#225: rm legacy bots stuff - 13https://git.io/JiO5z [16:55:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 5.04, 6.05, 4.45 [16:57:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.02, 5.33, 4.40 [16:59:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.09, 4.22, 4.10 [17:00:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.53, 3.15 [17:02:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.16, 3.07 [17:33:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.40, 22.51, 18.42 [17:35:15] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.36, 5.87, 4.18 [17:37:10] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.59, 5.39, 4.21 [17:39:06] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.12, 4.60, 4.06 [17:42:07] [02ManageWiki] 07Universal-Omega closed pull request 03#306: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JimOV [17:42:08] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Ji3t0 [17:42:10] [02miraheze/ManageWiki] 07dependabot[bot] 03261563b - Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 (#306) [17:42:11] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 [17:42:13] [02ManageWiki] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 - 13https://git.io/vpSns [17:44:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.86, 3.36 [17:45:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.85, 21.45, 21.24 [17:46:57] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.50, 23.84, 22.07 [17:48:32]