[00:00:34] !log [universalomega@test3] starting deploy of {'pull': 'landing', 'landing': True} to skip [00:00:35] !log [universalomega@test3] finished deploy of {'pull': 'landing', 'landing': True} to skip - SUCCESS in 1s [00:00:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.67, 3.53 [00:03:26] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.80, 3.60 [00:03:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 7.47, 6.97 [00:03:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [00:03:58] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb [00:04:10] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:10] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:04:39] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:45] paladox: ^ is still happening. [00:05:13] !log reboot mw13 [00:05:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.67, 7.45, 7.02 [00:06:05] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.083 second response time [00:06:07] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.011 second response time [00:06:40] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 1.456 second response time [00:06:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:27] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.805 second response time [00:07:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.15, 7.89, 7.25 [00:07:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.49, 2.56, 0.92 [00:07:51] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.697 second response time [00:08:56] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:01] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:24] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.66, 3.60 [00:09:35] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:43] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 14.12, 6.54, 2.57 [00:11:00] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 9.440 second response time [00:11:05] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 8.247 second response time [00:11:28] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.846 second response time [00:13:06] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.lefrenchmelee.fr_private] [00:13:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.33, 7.89, 7.55 [00:13:55] !log [@mw11] DEPLOY ABORTED: Canary check failed for mw9 [00:15:03] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.30, 18.72, 20.01 [00:15:21] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:15:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:29] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:15:45] paladox: ^ mw9 is down. Has been for awhile. Also issues on mw10 and mw13 again. Also visible 502 errors at least 40% of the time for me on-wiki for awhile today. [00:15:47] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.34, 22.88, 17.22 [00:15:49] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 5.319 second response time [00:16:11] i'm really not sure why this is happening [00:16:16] looking like higher load on mw* [00:16:31] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.72, 5.27, 3.54 [00:16:41] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[Landing Sync] [00:17:21] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 5.334 second response time [00:17:32] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 5.720 second response time [00:17:41] oh [00:17:46] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.88, 21.36, 17.35 [00:17:48] paladox: mw12 is down also. Both mw9 and mw12 have been completely down for a couple hours, leaving only the others. [00:18:31] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.15, 4.66, 3.52 [00:18:32] !log reboot mw9 [00:18:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:58] !log reboot mw12 [00:19:04] Basically mw10 and mw13 were doing all the work with mw8 issues also. So that's why they kept going down. [00:19:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 6.77, 7.04 [00:19:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:19:45] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.04, 19.24, 17.07 [00:19:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:20:10] mw11 was doing pretty good though. [00:20:47] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 1 minute ago with 14 failures. Failed resources (up to 3 shown): Service[ulogd2],Service[prometheus-node-exporter],Service[nagios-nrpe-server],Service[ssh] [00:20:57] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.637 second response time [00:21:09] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.009 second response time [00:21:12] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20007 bytes in 0.234 second response time [00:21:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 1.648 second response time [00:21:31] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.249 second response time [00:21:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.46, 6.87, 7.06 [00:21:33] damn spoke to soon, mw11 is down. [00:21:48] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [00:21:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [00:21:59] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 0.391 second response time [00:22:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:22:05] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 1.015 second response time [00:22:08] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.049 second response time [00:22:22] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:22:39] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.317 second response time [00:22:45] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [00:22:45] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [00:23:03] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:23:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 39.50, 30.55, 22.05 [00:24:31] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 5.48, 6.01, 4.44 [00:24:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.82, 6.16, 2.61 [00:24:52] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.352 second response time [00:24:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.93, 6.35, 2.57 [00:25:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.19, 9.25, 7.86 [00:26:31] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.72, 5.38, 4.41 [00:26:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.59, 6.83, 3.31 [00:27:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:31] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.82, 4.55, 4.22 [00:28:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.22, 7.15, 3.76 [00:28:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.77, 6.74, 3.71 [00:29:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.37, 7.92, 7.73 [00:30:38] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.06, 6.34, 3.86 [00:31:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.22, 23.20, 22.02 [00:31:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.62, 7.40, 7.49 [00:35:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.13, 5.50, 6.71 [00:37:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.79, 5.57, 7.98 [00:37:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.39, 5.33, 7.88 [00:37:43] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.19, 5.32, 6.53 [00:37:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.83, 1.73, 1.20 [00:39:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.46, 18.24, 20.37 [00:39:59] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.68, 1.37, 1.13 [00:42:37] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:45:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.25, 4.99, 6.70 [00:45:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.57, 3.95 [00:47:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.54, 5.26, 6.71 [00:51:54] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49420 MB (11% inode=98%); [00:52:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.33, 6.83, 5.45 [00:53:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 3.98, 3.90 [00:53:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 7.44, 6.66 [00:54:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.41, 6.27, 5.40 [00:57:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.51, 7.52, 6.95 [00:57:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.29, 6.83, 6.51 [00:59:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 7.15, 6.88 [00:59:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.58, 3.77 [00:59:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.09, 6.45, 6.49 [00:59:43] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.99, 6.21, 6.32 [01:01:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.13, 6.35, 6.61 [01:01:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.86, 3.85 [01:07:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.66, 3.68, 3.83 [01:17:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.72, 3.70 [01:19:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.32, 3.22, 3.52 [01:20:18] [02mediawiki] 07Universal-Omega closed pull request 03#4383: Bump extensions/GlobalWatchlist from `ae19951` to `dc6c067` - 13https://git.io/JM4qF [01:20:20] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-dc6c067 [01:20:21] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JM1YN [01:20:23] [02miraheze/mediawiki] 07dependabot[bot] 037f367a5 - Bump extensions/GlobalWatchlist from `ae19951` to `dc6c067` (#4383) [01:20:24] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-dc6c067 - 13https://git.io/vbL5b [01:21:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.06, 4.38, 3.91 [01:27:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.92, 3.94 [01:41:54] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48932 MB (10% inode=98%); [01:47:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.72, 4.02, 3.72 [01:51:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.87, 3.72 [02:03:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.70, 2.88, 3.30 [02:11:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.52, 3.36 [02:13:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.84, 3.50 [02:17:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.71, 3.53 [02:21:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.86, 3.60 [02:23:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.66, 3.55 [02:27:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.25, 3.40 [02:31:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.51, 3.49 [02:33:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 2.98, 3.29 [02:42:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.05, 3.63, 3.39 [02:44:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.59, 3.40 [02:46:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.34, 3.36, 3.32 [02:52:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.87, 4.46, 3.84 [02:53:54] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49352 MB (11% inode=98%); [03:04:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.91, 3.89 [03:19:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.76, 3.72 [03:23:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.92, 3.83 [03:33:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.67, 3.61 [03:35:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.64, 3.62 [03:39:54] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48946 MB (10% inode=98%); [03:41:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.88, 3.70 [03:45:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.16, 3.28, 3.51 [03:49:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.50, 3.53 [03:49:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 6.67, 4.86 [03:51:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.42, 3.49 [03:51:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.70, 5.72, 4.73 [03:55:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.03, 3.91, 3.64 [03:57:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.68, 3.60 [04:03:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.00, 2.92, 3.32 [04:41:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.23, 2.95 [04:43:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.03, 2.81, 2.84 [04:49:54] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49299 MB (11% inode=98%); [05:31:54] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48936 MB (10% inode=98%); [05:52:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.41, 3.22 [05:54:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.01, 3.09 [06:05:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.92, 5.74, 3.59 [06:07:02] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.99, 6.71, 4.20 [06:15:02] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.97, 7.83, 5.88 [06:15:54] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 52515 MB (11% inode=98%); [06:21:02] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.68, 6.73, 5.99 [06:21:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.71, 7.67, 5.64 [06:23:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.36, 7.15, 5.70 [06:23:54] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48578 MB (10% inode=98%); [06:27:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.99, 6.49, 5.77 [06:51:34] PROBLEM - db12 Current Load on db12 is CRITICAL: CRITICAL - load average: 11.93, 9.49, 6.29 [06:51:54] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 50454 MB (11% inode=98%); [06:53:31] RECOVERY - db12 Current Load on db12 is OK: OK - load average: 2.57, 6.64, 5.62 [07:04:55] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±4] 13https://git.io/JMM3B [07:04:57] [02miraheze/ssl] 07Reception123 03fe8fb1b - add nl.xliving.tk cert [07:04:58] [02miraheze/ssl] 07Reception123 039bff46c - add wiki.projectdiablo2.com cert [07:09:21] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JMMsr [07:09:23] [02miraheze/dns] 07Reception123 03cdbbca7 - add hoodwiki.xyz zone [09:16:11] [02puppet] 07RhinosF1 opened pull request 03#2140: mediawiki: remove branch data where no effect - 13https://git.io/JMMiy [09:16:22] Reception123: making pr [09:17:06] [02puppet] 07RhinosF1 synchronize pull request 03#2140: mediawiki: remove branch data where no effect - 13https://git.io/JMMiy [09:17:11] RhinosF1: thanks. feel free to do the mw11 switch PR too while you're at it [09:17:12] so we can be ready to go [09:17:25] [02puppet] 07RhinosF1 synchronize pull request 03#2140: mediawiki: remove branch data where no effect - 13https://git.io/JMMiy [09:17:43] [02puppet] 07RhinosF1 synchronize pull request 03#2140: mediawiki: remove branch data where no effect - 13https://git.io/JMMiy [09:18:24] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JMMP3 [09:18:26] [02miraheze/mw-config] 07Reception123 03ac63157 - set all wikis to readonly (MW 1.37 upgrade) [09:18:27] [02mw-config] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [09:18:29] [02mw-config] 07Reception123 opened pull request 03#4245: set all wikis to readonly (MW 1.37 upgrade) - 13https://git.io/JMMPs [09:18:40] [02puppet] 07RhinosF1 synchronize pull request 03#2140: mediawiki: remove branch data where no effect - 13https://git.io/JMMiy [09:19:29] miraheze/mw-config - Reception123 the build passed. [09:19:41] [02puppet] 07RhinosF1 synchronize pull request 03#2140: mediawiki: remove branch data where no effect - 13https://git.io/JMMiy [09:20:33] [02puppet] 07RhinosF1 opened pull request 03#2141: mediawiki: update to 1.37 - 13https://git.io/JMMP9 [09:20:56] [02puppet] 07RhinosF1 synchronize pull request 03#2141: mediawiki: update to 1.37 - 13https://git.io/JMMP9 [09:20:57] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://git.io/JMMPA [09:20:59] [02miraheze/mw-config] 07Reception123 033bd3683 - Liberty removal sitenotice for wikis with the skin enabled [09:21:00] [02mw-config] 07Reception123 created branch 03Reception123-patch-3 - 13https://git.io/vbvb3 [09:21:02] [02mw-config] 07Reception123 opened pull request 03#4246: Liberty removal sitenotice for wikis with the skin enabled - 13https://git.io/JMMPx [09:21:24] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://git.io/JMMXe [09:21:25] [02miraheze/mw-config] 07github-actions 03f62c746 - CI: lint code to MediaWiki standards [09:21:27] [02mw-config] 07github-actions[bot] synchronize pull request 03#4246: Liberty removal sitenotice for wikis with the skin enabled - 13https://git.io/JMMPx [09:22:11] Reception123: #2140 is fine [09:22:11] miraheze/mw-config - Reception123 the build passed. [09:23:01] [02puppet] 07Reception123 closed pull request 03#2140: mediawiki: remove branch data where no effect - 13https://git.io/JMMiy [09:23:03] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/JMMXC [09:23:04] [02miraheze/puppet] 07RhinosF1 03a6f272b - mediawiki: remove branch data where no effect (#2140) [09:28:45] [02mw-config] 07Universal-Omega reviewed pull request 03#4246 commit - 13https://git.io/JMM1g [09:29:49] Reception123: ^ (IMPORTANT) otherwise every page load on every wiki will give log spam for undefined variable since extension configs aren't extracted to globals at that stage yet. [09:30:04] CosmicAlpha: ah, I've always done it like that though (for example https://github.com/miraheze/mw-config/commit/594d96e44c37023632cdabfd9d0cead34510a896#diff-0ebce748027f36364059daa8eb9847c933aa7562798d2624f259264c2d2014fd ) [09:30:05] [url] approvedrevs and myvariables conflict sitenotice (#3821) · miraheze/mw-config@594d96e · GitHub | github.com [09:30:07] so I guess that caused spam? [09:31:05] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://git.io/JMMMU [09:31:06] [02miraheze/mw-config] 07Reception123 03ab19034 - Update Sitenotice.php [09:31:06] Reception123: no, the automation of the extension loading changed that a bit, which wasn't done at that time. [09:31:11] [02mw-config] 07Reception123 synchronize pull request 03#4246: Liberty removal sitenotice for wikis with the skin enabled - 13https://git.io/JMMPx [09:31:12] CosmicAlpha: ah [09:32:20] miraheze/mw-config - Reception123 the build passed. [09:33:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.18, 2.40, 1.29 [09:35:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.39, 1.93, 1.25 [09:37:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.40, 1.37, 1.12 [09:37:47] [02puppet] 07Universal-Omega reviewed pull request 03#2141 commit - 13https://git.io/JMMDy [09:38:30] RhinosF1: ^ [09:39:01] Not sure if you want to so just a comment, not requesting changes at all. [09:40:19] CosmicAlpha: if we get multiversion out, we won't be doing it all next time [09:40:21] So meh [09:40:32] [02mw-config] 07Universal-Omega assigned pull request 03#4245: set all wikis to readonly (MW 1.37 upgrade) - 13https://git.io/JMMPs [09:40:44] Hopefully I might at least get the puppet work done [09:42:08] [02mw-config] 07Universal-Omega reviewed pull request 03#4244 commit - 13https://git.io/JMMya [09:44:12] [02mw-config] 07Universal-Omega assigned pull request 03#4239: Lingo conflicts with Newsletter in 1.37 (see T8166) - 13https://git.io/JMzOX [09:53:50] !log add agentisai to github mw-config with triage access [09:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:54:56] RhinosF1: I already invited a few minutes ago. Was actually just typing log lol. [09:55:38] Anyways that's all for me tonight. I'm going now. [09:57:26] Ye we conflicted [10:10:01] paladox: hi, would you be available on Tuesday after 17:00 UTC? [10:10:14] (just in case something goes wrong) [11:18:38] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 0.94, 2.50, 1.65 [11:20:38] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.41, 1.77, 1.48 [11:22:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.50, 1.43, 1.40 [11:38:47] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.07, 7.74, 5.61 [11:42:34] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.90, 6.57, 5.68 [11:50:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 16.92, 8.89, 5.88 [11:52:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.76, 7.14, 5.53 [11:53:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.09, 6.89, 5.15 [11:55:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.58, 6.30, 5.14 [11:55:57] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.45, 1.06 [11:56:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.41, 7.73, 6.38 [11:56:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.94, 6.56, 5.18 [11:57:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.44, 1.40, 1.09 [11:58:30] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.97, 6.08, 5.18 [12:00:28] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.34, 6.71, 6.28 [12:00:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.82, 6.19, 5.79 [12:04:02] [02mw-config] 07Reception123 synchronize pull request 03#4246: Liberty removal sitenotice for wikis with the skin enabled - 13https://git.io/JMMPx [12:04:03] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://git.io/JMDOe [12:04:05] [02miraheze/mw-config] 07Reception123 03c5d0d06 - Update Sitenotice.php [12:05:18] miraheze/mw-config - Reception123 the build passed. [12:05:23] [02mw-config] 07Reception123 closed pull request 03#4246: Liberty removal sitenotice for wikis with the skin enabled - 13https://git.io/JMMPx [12:05:25] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMDOs [12:05:27] [02miraheze/mw-config] 07Reception123 03f267e57 - Liberty removal sitenotice for wikis with the skin enabled (#4246) [12:05:28] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://git.io/vbvb3 [12:05:29] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-3 [12:05:43] !log [reception@mw11] starting deploy of {'config': True} to all [12:05:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:05:54] !log [reception@mw11] finished deploy of {'config': True} to all - SUCCESS in 10s [12:05:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:06:30] miraheze/mw-config - Reception123 the build passed. [12:06:48] !log [@test3] starting deploy of {'config': True} to skip [12:06:49] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [12:06:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:06:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:07:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 5.55, 4.87 [12:09:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.56, 8.12, 5.96 [12:11:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.21, 7.20, 5.88 [12:11:41] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMDsg [12:11:43] [02miraheze/mw-config] 07Reception123 0392c5a34 - revert back to regular method for $wmgUseLiberty and increase font size [12:11:45] !log [@mw11] starting deploy of {'config': True} to all [12:11:46] !log [reception@mw11] starting deploy of {'config': True} to all [12:11:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:11:50] !log [reception@mw11] finished deploy of {'config': True} to all - SUCCESS in 5s [12:11:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:11:51] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 5s [12:11:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:11:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:12:46] miraheze/mw-config - Reception123 the build passed. [12:13:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.62, 5.90, 5.56 [12:23:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.06, 5.58 [12:24:06] !log [@test3] starting deploy of {'config': True} to skip [12:24:07] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [12:25:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.05, 5.96, 5.36 [12:32:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.01, 6.66, 5.26 [12:34:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.33, 6.07, 5.19 [12:38:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.53, 7.04, 5.46 [12:38:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.90, 3.48, 2.89 [12:40:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.18, 6.77, 5.54 [12:40:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.51, 2.97 [12:42:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.06, 5.93, 5.40 [12:42:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 3.00, 2.85 [12:52:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.56, 3.12 [12:53:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.79, 6.61, 5.40 [12:54:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:54:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.32, 8.78, 6.35 [12:55:46] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://git.io/JMDuS [12:55:47] [02miraheze/mw-config] 07Reception123 03690bbec - 1.37 upgrade sitenotice [12:55:49] [02mw-config] 07Reception123 created branch 03Reception123-patch-3 - 13https://git.io/vbvb3 [12:55:50] [02mw-config] 07Reception123 opened pull request 03#4247: 1.37 upgrade sitenotice - 13https://git.io/JMDu9 [12:55:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.95, 5.83, 5.26 [12:56:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.35, 3.14 [12:56:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.04, 7.46, 6.15 [12:56:51] miraheze/mw-config - Reception123 the build passed. [12:58:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.46, 6.44, 5.94 [12:59:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:07:14] Reception123: I’ll be around, yeh [13:08:15] paladox: perfect, thanks! In that case Tuesday 5pm should be good [13:51:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.57, 3.21 [13:54:14] paladox: since you have Phab admin, could you disable this advertiser? https://phabricator.miraheze.org/p/Bastservice/ [13:54:15] [url] ♟ Bastservice | phabricator.miraheze.org [13:55:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.27, 3.17 [13:59:31] !log disabled spammer on Phab [13:59:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:01:51] !log removed spam from a Phab profile via DB [14:01:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:23] @Reception123 Will you also lock the spammer account too? [14:03:53] @DarkMatterMan4500 sure [14:04:05] There was also • Aaronsmithrowe which I caught and disabled [14:04:30] Oh. [14:09:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.52, 3.24 [14:09:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.95, 6.75, 5.13 [14:11:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.30, 6.90, 5.39 [14:11:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.69, 6.26, 5.24 [14:13:03] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: SSL OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:13:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.15, 6.15, 5.32 [14:14:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.15, 8.83, 6.59 [14:15:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.00, 6.39, 5.54 [14:17:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.88, 3.63 [14:18:00] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.81, 7.67, 6.66 [14:18:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.36, 7.13, 6.10 [14:19:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 4.03, 3.70 [14:20:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.90, 8.48, 6.74 [14:21:31] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.68, 7.03, 6.14 [14:21:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.72, 20.75, 17.81 [14:22:00] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.34, 6.63, 6.46 [14:22:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.99, 7.83, 6.70 [14:23:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.05, 20.10, 17.94 [14:24:33] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.81, 6.36, 6.30 [14:25:20] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.22, 5.97, 5.93 [14:25:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 4.00, 3.88 [14:35:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.02, 2.76, 3.33 [14:39:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.82, 3.68 [14:42:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.19, 7.16, 5.81 [14:44:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.94, 7.18, 5.99 [14:45:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.31, 4.00, 3.74 [14:47:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.72, 3.67 [14:48:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.61, 6.70, 6.07 [14:48:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.72, 6.58, 5.75 [14:48:28] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.61, 7.11, 6.34 [14:49:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.31, 7.36, 6.33 [14:50:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.05, 6.41, 5.82 [14:50:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.35, 7.15, 6.45 [14:51:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.29, 4.09, 3.81 [14:51:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.62, 6.19, 6.02 [14:52:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.41, 8.01, 6.75 [14:52:16] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.59, 6.57, 6.32 [14:52:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.12, 6.93, 6.01 [14:52:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.75, 6.36, 6.24 [14:53:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.99, 3.81 [14:54:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 7.83, 6.83 [14:54:17] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.07, 6.33, 5.91 [14:58:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.15, 6.62, 6.59 [15:02:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.45, 6.96, 6.74 [15:04:43] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.95, 6.29, 6.53 [15:05:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.63, 3.97, 3.75 [15:07:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.46, 3.33, 3.54 [15:08:38] !log changed an unverified email on Phab to correct email [15:08:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:09:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.87, 3.04, 3.39 [15:14:54] !log RENAMED inryeswiki to inreyswiki [15:14:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.76, 7.54, 6.49 [15:26:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.65, 7.61, 6.68 [15:28:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.18, 7.03, 6.58 [15:30:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.61, 6.34, 6.39 [15:52:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.30, 3.22 [15:56:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.29, 3.25 [16:02:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.75, 3.42 [16:04:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.41, 3.34 [16:06:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 3.98, 3.56 [16:17:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.92, 3.82 [16:19:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.84, 4.17, 3.92 [16:21:10]