[00:12:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.45, 2.91 [00:14:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.15, 2.87 [00:14:38] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.60, 24.98, 19.29 [00:14:40] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.00, 6.32, 4.34 [00:20:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.96, 2.53, 1.54 [00:24:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.56, 1.64, 1.42 [00:28:20] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.09, 5.69, 5.66 [00:28:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.01, 21.81, 22.70 [00:32:14] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.70, 4.19, 5.06 [00:34:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.30, 17.49, 20.37 [00:42:37] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JyZ1D [00:42:39] [02miraheze/mediawiki] 07Universal-Omega 03ab49155 - Remove duplicate `$wgIncludejQueryMigrate` [00:42:40] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [00:42:50] [02mediawiki] 07Universal-Omega opened pull request 03#4709: Remove duplicate `$wgIncludejQueryMigrate` - 13https://git.io/JyZ1y [00:43:07] [02mediawiki] 07Universal-Omega labeled pull request 03#4709: Remove duplicate `$wgIncludejQueryMigrate` - 13https://git.io/JyZ1y [00:43:08] [02mediawiki] 07Universal-Omega labeled pull request 03#4709: Remove duplicate `$wgIncludejQueryMigrate` - 13https://git.io/JyZ1y [00:43:10] [02mediawiki] 07Universal-Omega labeled pull request 03#4709: Remove duplicate `$wgIncludejQueryMigrate` - 13https://git.io/JyZ1y [00:47:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.12, 4.81, 4.41 [00:49:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.51, 4.70, 4.43 [01:01:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.50, 3.21 [01:05:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.40, 3.24 [01:51:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:56:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:59:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48956 MB (10% inode=98%); [02:53:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49725 MB (11% inode=98%); [03:23:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.20, 2.86 [03:25:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 2.97, 2.83 [03:50:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.04, 2.73 [03:52:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.33, 2.89 [05:37:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48942 MB (10% inode=98%); [06:03:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49518 MB (11% inode=98%); [06:06:32] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.35, 5.59, 3.34 [06:12:32] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.48, 6.22, 4.42 [06:23:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48826 MB (10% inode=98%); [06:26:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:31:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:38:53] PROBLEM - wiki.apadotech.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.apadotech.ga could not be found [06:41:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49707 MB (11% inode=98%); [07:04:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.23, 2.30, 1.52 [07:06:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.32, 1.60, 1.35 [08:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:09:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48952 MB (10% inode=98%); [09:31:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49314 MB (11% inode=98%); [09:35:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48813 MB (10% inode=98%); [09:56:11] PROBLEM - bacula2 ferm_active on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:56:19] PROBLEM - bacula2 PowerDNS Recursor on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:56:23] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 100% [09:56:43] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 100% [09:56:56] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:57:15] PROBLEM - bacula2 APT on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:57:22] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [09:57:27] PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:57:29] PROBLEM - Host bacula2 is DOWN: PING CRITICAL - Packet loss = 100% [10:15:27] PROBLEM - mwtask1 SSH on mwtask1 is CRITICAL: Server answer: Exceeded MaxStartups [10:17:27] RECOVERY - mwtask1 SSH on mwtask1 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [10:45:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49165 MB (11% inode=98%); [11:08:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:18:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:27:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48954 MB (10% inode=98%); [14:16:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:21:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.75, 3.49, 3.00 [14:21:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:23:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.21, 2.96 [14:28:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.56, 3.22 [14:30:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.19, 3.13 [14:37:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49138 MB (11% inode=98%); [14:56:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.67, 3.39 [15:00:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 3.04, 3.21 [15:20:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.08, 2.95 [15:22:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 2.83, 2.87 [15:23:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48956 MB (10% inode=98%); [16:04:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.62, 3.15 [16:05:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.49, 3.16 [16:07:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.04, 3.02 [16:39:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.77, 3.31 [16:41:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.47, 3.25 [16:43:35] !log reboot puppet3 [16:43:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.92, 2.95, 3.09 [16:43:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:44:31] !log upgrade kernel on puppet3 [16:44:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:10] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 13.25, 5.92, 2.59 [16:46:15] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:47:24] PROBLEM - puppet3 puppetserver on puppet3 is CRITICAL: connect to address 51.195.236.216 and port 8140: Connection refused [16:47:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.40, 5.72, 4.58 [16:48:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.40, 6.07, 4.99 [16:48:38] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.45, 20.52, 15.91 [16:49:19] RECOVERY - puppet3 puppetserver on puppet3 is OK: TCP OK - 0.002 second response time on 51.195.236.216 port 8140 [16:49:26] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:49:45] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:50:03] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:50:16] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:50:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:51:06] PROBLEM - puppet3 Current Load on puppet3 is CRITICAL: CRITICAL - load average: 6.27, 3.58, 1.50 [16:51:30] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:51:35] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:51:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.80, 6.45, 5.16 [16:52:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.53, 6.18, 5.33 [16:52:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.02, 20.71, 17.20 [16:53:00] RECOVERY - puppet3 Current Load on puppet3 is OK: OK - load average: 2.20, 3.10, 1.58 [16:54:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.05, 19.21, 17.05 [16:55:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:55:35] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 0.71, 3.70, 3.66 [16:57:33] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.41, 2.65, 3.28 [17:06:59] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.23, 1.81, 1.21 [17:08:59] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.75, 1.37, 1.11 [17:11:20] RECOVERY - Host bacula2 is UP: PING OK - Packet loss = 0%, RTA = 75.24 ms [17:11:22] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [17:11:22] RECOVERY - ping6 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 76.79 ms [17:11:22] PROBLEM - bacula2 Puppet on bacula2 is WARNING: WARNING: Puppet last ran 7 hours ago [17:11:22] RECOVERY - bacula2 ferm_active on bacula2 is OK: OK ferm input default policy is set [17:11:22] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [17:11:32] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [17:11:32] RECOVERY - bacula2 Bacula Databases db13 on bacula2 is OK: OK: Full, 77836 files, 29.60GB, 2021-12-26 00:44:00 (16.5 hours ago) [17:11:42] PROBLEM - bacula2 Bacula Daemon on bacula2 is CRITICAL: PROCS CRITICAL: 1 process with UID = 114 (bacula) [17:12:31] RECOVERY - bacula2 Bacula Databases db12 on bacula2 is OK: OK: Full, 77875 files, 22.76GB, 2021-12-26 00:27:00 (16.8 hours ago) [17:12:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:12:39] RECOVERY - bacula2 PowerDNS Recursor on bacula2 is OK: DNS OK: 0.288 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:12:39] RECOVERY - bacula2 Bacula Databases db11 on bacula2 is OK: OK: Full, 83645 files, 16.78GB, 2021-12-26 00:14:00 (17.0 hours ago) [17:12:45] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 6817 files, 57.43MB, 2021-12-26 00:46:00 (16.4 hours ago) [17:12:46] RECOVERY - ping4 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 76.19 ms [17:12:59] RECOVERY - bacula2 Disk Space on bacula2 is OK: DISK OK - free space: / 2074330 MB (82% inode=99%); [17:13:06] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [17:13:17] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:13:42] RECOVERY - bacula2 Bacula Daemon on bacula2 is OK: PROCS OK: 2 processes with UID = 114 (bacula) [17:17:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:23:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.51, 3.18 [17:25:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.35, 3.16 [17:30:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.70, 3.35 [17:32:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.46, 3.31 [17:34:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 4.02, 3.53 [17:38:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.72, 3.54 [17:40:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.85, 3.60 [17:41:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:44:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.53, 3.40, 3.49 [17:45:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.01, 6.56, 5.43 [17:46:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.62, 3.55 [17:49:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.86, 6.67, 6.03 [17:49:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.73, 6.67, 5.80 [17:50:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 7.16, 6.29 [17:52:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 7.10, 6.37 [17:53:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.98, 6.88, 6.10 [17:54:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.17, 6.70, 6.16 [17:54:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.15, 3.49, 3.65 [17:55:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.75, 6.58, 6.09 [17:56:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.88, 6.52, 6.17 [18:01:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:02:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.76, 6.60, 6.54 [18:04:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 2.99, 3.36 [18:09:15] !log upgrade puppet-agent on bacula2 [18:09:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:27] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:12:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:14:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.42, 3.35 [18:16:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.26, 3.24, 3.29 [18:17:52] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.53, 6.62, 5.91 [18:19:47] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.74, 6.41, 5.91 [18:22:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.80, 3.50 [18:24:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.52, 3.43 [18:25:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.43, 6.60, 6.08 [18:29:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.05, 6.13, 6.03 [18:30:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.14, 3.31 [18:32:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:38:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:42:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 3.83, 3.50 [18:44:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.88, 3.57 [18:45:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49097 MB (11% inode=98%); [18:50:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.35, 19.34, 17.63 [18:52:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.99, 18.73, 17.60 [18:54:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.64, 3.57 [18:56:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.24, 3.43 [18:58:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.02, 3.32 [18:58:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:19:03] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48955 MB (10% inode=98%); [19:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:34:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.70, 3.35 [19:38:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.37, 3.29 [19:42:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.56, 3.38 [19:44:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.40, 3.35 [19:44:59] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.80, 1.86, 1.36 [19:46:59] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.53, 1.39, 1.24 [19:54:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.60, 3.34 [19:56:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.36, 3.29 [20:05:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.98, 4.02, 3.53 [20:09:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.27, 3.90, 3.62 [20:14:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.56, 3.55 [20:18:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:18:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.80, 3.68 [20:20:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 4.09, 3.81 [20:20:38] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.81, 19.35, 16.52 [20:21:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.23, 6.84, 5.95 [20:23:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.48, 7.09, 6.15 [20:23:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.22, 6.11, 5.18 [20:25:00] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.50, 7.51, 6.41 [20:25:14] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.30, 6.90, 5.79 [20:25:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.61, 6.42, 5.42 [20:26:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 4.00, 3.91 [20:26:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.51, 18.06, 17.04 [20:26:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.02, 7.02, 6.37 [20:27:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.50, 6.34, 5.73 [20:28:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.88, 4.17, 4.00 [20:28:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.18, 6.34, 6.19 [20:30:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.62, 3.82 [20:32:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.81, 3.86 [20:34:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.23, 3.63 [20:38:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:42:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.09, 3.40 [21:01:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.50, 3.36 [21:02:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.21, 3.27 [21:08:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.51, 3.40 [21:10:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.40, 3.37 [21:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.62, 3.46 [21:16:34] [02CreateWiki] 07RhinosF1 commented on pull request 03#268: Add realm option to setVariables to set SiteConfiguration tags - 13https://git.io/JyCFO [21:16:39] [02CreateWiki] 07RhinosF1 closed pull request 03#268: Add realm option to setVariables to set SiteConfiguration tags - 13https://git.io/JDFSi [21:16:40] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyCF3 [21:16:42] [02miraheze/CreateWiki] 07Universal-Omega 0344bf1fc - Add realm option to setVariables to set SiteConfiguration tags (#268) [21:16:43] [02CreateWiki] 07RhinosF1 deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [21:16:45] [02miraheze/CreateWiki] 07RhinosF1 deleted branch 03Universal-Omega-patch-1 [21:20:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.96, 4.07, 3.67 [21:24:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.60, 3.57 [21:24:55] miraheze/CreateWiki - RhinosF1 the build passed. [21:28:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.96, 2.92, 3.31 [21:48:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.39, 3.28 [21:50:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 3.89, 3.48 [21:52:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.92, 3.54 [21:54:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.78, 4.20, 3.69 [21:56:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.33, 3.60, 3.54 [22:00:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.56, 3.51 [22:02:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.48, 3.49 [22:04:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.38, 3.03, 3.32 [22:11:03] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49266 MB (11% inode=98%); [22:12:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.68, 3.85, 3.51 [22:14:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 3.13, 3.29 [22:22:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:27:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:38:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.42, 3.26 [22:42:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.90, 3.80, 3.42 [22:45:56] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:52:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:57:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:58:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.94, 3.69, 3.72 [23:00:05] !log [@mw11.miraheze.org] starting deploy of {'l10nupdate': True} to ovlon [23:00:06] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.67, 3.67 [23:02:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 1.64, 1.17 [23:04:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.92, 1.31, 1.10 [23:06:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.71, 3.70 [23:07:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:08:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.96, 4.35, 3.94 [23:10:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.72, 3.75 [23:11:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.08, 21.11, 17.93 [23:13:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.28, 21.50, 18.45 [23:17:04] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48950 MB (10% inode=98%); [23:17:14] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.75, 18.56, 18.04 [23:20:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 7.60, 4.49, 3.94 [23:23:04] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.87, 22.86, 19.91 [23:26:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.60, 3.72 [23:28:11] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.06, 5.44, 4.11 [23:30:10] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.58, 3.36, 2.23 [23:32:05] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 6.00, 5.77, 4.55 [23:32:11] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.08, 3.19, 2.30 [23:36:00] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.11, 5.92, 4.87 [23:37:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:38:13] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2287s [23:38:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:43] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:42:35] !log [@mw11.miraheze.org] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2550s [23:42:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.04, 3.34 [23:57:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:59:52] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.19, 4.65, 5.80