[00:01:14] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 90691 MB (12% inode=99%); [00:01:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.82, 4.10, 5.18 [00:02:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.61, 20.21, 23.87 [00:03:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.34, 4.19, 5.07 [00:04:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.68, 21.07, 23.77 [00:05:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.46, 5.18, 5.34 [00:06:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.42, 3.04 [00:07:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.59, 5.41, 5.41 [00:08:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.08, 3.21, 3.00 [00:10:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.41, 24.26, 24.06 [00:11:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.27, 6.56, 5.78 [00:22:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.76, 3.31 [00:24:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.46, 3.25 [00:26:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.12, 3.15 [00:34:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.74, 3.41 [00:42:24] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hypotheticalhurricaneswiki /home/reception/hypotheticalhurricanes_pages_full.xml --username-prefix wikia:hypotheticalhurricanes --report 1 --no-updates (END - exit=0) [00:42:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.65, 3.49, 3.53 [00:42:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb [00:42:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.38.69.175/cpweb, 2001:41d0:800:170b::5/cpweb, 2607:5300:205:200::1c30/cpweb [00:43:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:43:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.19, 3.40 [00:44:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:46:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:48:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:53:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.70, 3.57 [01:00:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 10.78, 15.11, 23.61 [01:01:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.31, 2.79, 5.65 [01:06:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.20, 3.39 [01:06:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.57, 14.01, 20.26 [01:07:27] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.95, 3.40, 4.95 [01:12:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.51, 3.45 [01:20:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.04, 3.68 [01:24:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.52, 3.60, 3.62 [01:40:36] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JXjdV [01:40:38] [02miraheze/MirahezeMagic] 07Universal-Omega 03dcf0039 - Add whitespace [01:40:39] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#303: Add reCaptcha response scores to debug logging - 13https://git.io/JXjZm [01:41:04] [02MirahezeMagic] 07Universal-Omega edited pull request 03#303: Add reCAPTCHA response scores to debug logging - 13https://git.io/JXjZm [01:41:45] miraheze/MirahezeMagic - Universal-Omega the build passed. [01:42:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.07, 3.34 [01:48:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.63, 3.48 [01:50:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.46, 4.34, 3.73 [01:56:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.90, 3.76 [02:02:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 4.14, 3.90 [02:08:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.84, 3.86 [02:12:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 4.24, 4.02 [02:36:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.29, 3.88 [02:46:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 2.87, 3.39 [04:04:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.53, 3.18 [04:06:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.61, 3.23 [04:08:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.65, 3.29 [04:12:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.15, 3.10, 3.17 [04:31:58] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 0.98, 1.89, 1.33 [04:33:58] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.47, 1.36, 1.20 [06:10:37] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.62, 6.75, 4.50 [06:12:37] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.47, 6.69, 4.76 [06:17:31] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 28774 MB (6% inode=98%); [06:21:31] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26393 MB (5% inode=98%); [08:16:25] PROBLEM - sportsanalytics.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'sportsanalytics.wiki' expires in 7 day(s) (Mon 22 Nov 2021 08:11:56 GMT +0000). [08:25:31] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 29122 MB (6% inode=98%); [08:29:37] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.12, 6.78, 6.27 [08:33:26] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.71, 6.64, 6.31 [08:41:31] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26686 MB (5% inode=98%); [08:51:36] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.23, 7.13, 6.70 [08:57:19] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.61, 6.79, 6.71 [09:25:20] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.25, 6.63, 6.30 [09:27:14] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.42, 6.47, 6.28 [09:47:58] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.33, 1.72, 1.04 [09:49:59] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.27, 1.20, 0.93 [09:51:52] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 5.77, 8.51, 5.75 [09:52:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.07, 7.40, 5.25 [09:53:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.47, 6.71, 5.43 [09:54:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 2.99, 5.72, 4.90 [10:13:02] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.04, 7.02, 6.23 [10:14:57] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.73, 6.61, 6.16 [10:19:14] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78652 MB (10% inode=99%); [10:23:14] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 79139 MB (11% inode=99%); [11:00:24] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.69, 8.32, 5.66 [11:01:14] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78370 MB (10% inode=99%); [11:02:23] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.50, 6.64, 5.37 [11:04:05] RhinosF1: I don't think I still have the stack trace for https://phabricator.miraheze.org/T8290 :/ [11:04:06] [url] ⚓ T8290 Internal error when member changes password | phabricator.miraheze.org [11:04:36] majavah: mobile but will post afternoon [11:04:43] thanks! [11:15:14] RECOVERY - graylog2 Disk Space on graylog2 is OK: DISK OK - free space: / 79270 MB (11% inode=99%); [11:31:14] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 78624 MB (10% inode=99%); [12:14:11] Also mobile but yes I have noticed such issues relating to changing passwords in the past [12:14:20] Might be CA related if I had to guess [12:32:26] It is [12:34:27] @raidarr: we take any security issue seriously and take all steps possible to quickly mitigate and respond however small [12:34:44] That includes close monitoring and upstream partnerships [12:39:29] And I appreciate the vigilance as well as reporting. [13:05:25] majavah: https://www.irccloud.com/pastebin/xzBwwAmj/ [13:05:36] Pretty sure the actual exception was account does not exist [13:06:12] 27 Oct was last time I pinged you [13:06:37] line 96 in REL1_36 is "return new Exception( "Global user '{$info['name']}' does not exist." );" [13:11:32] Yes [13:11:33] That [13:11:53] majavah: it's happening though when people have accounts created for them [13:12:03] Which for us has never created the global account [13:12:08] It's normally created on login [13:50:43] [02puppet] 07RhinosF1 reviewed pull request 03#2121 commit - 13https://git.io/J1JIz [13:51:26] CosmicAlpha: ^ [14:14:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.50, 3.26 [14:16:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.38, 3.25 [14:21:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.66, 3.40 [14:27:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.19, 3.94, 3.54 [14:31:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.72, 3.55 [14:33:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.82, 3.59 [14:35:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.48, 3.49 [14:37:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.96, 3.06, 3.35 [14:41:40] [02puppet] 07RhinosF1 opened pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://git.io/J1JlH [14:43:15] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://git.io/J1JlH [14:49:28] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://git.io/J1JlH [14:51:35] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [14:52:17] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://git.io/J1JlH [14:53:08] PROBLEM - bacula2 SSH on bacula2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:17] PROBLEM - bacula2 NTP time on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:41] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [14:53:48] PROBLEM - bacula2 PowerDNS Recursor on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:16] PROBLEM - bacula2 APT on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:28] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:31] PROBLEM - Host bacula2 is DOWN: PING CRITICAL - Packet loss = 100% [15:01:01] !log reinstalling bacula due to new disk size increase not being applied due to custom ISO [15:01:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:03:21] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://git.io/J1JlH [15:06:30] RECOVERY - Host bacula2 is UP: PING OK - Packet loss = 0%, RTA = 78.10 ms [15:06:34] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection refusedconnect to host 168.235.111.29 port 5666: Connection refused [15:06:54] PROBLEM - bacula2 Bacula Daemon on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection refusedconnect to host 168.235.111.29 port 5666: Connection refused [15:06:59] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection refusedconnect to host 168.235.111.29 port 5666: Connection refused [15:07:04] PROBLEM - bacula2 Current Load on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection refusedconnect to host 168.235.111.29 port 5666: Connection refused [15:07:24] PROBLEM - bacula2 conntrack_table_size on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection refusedconnect to host 168.235.111.29 port 5666: Connection refused [15:07:24] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 100% [15:07:24] PROBLEM - bacula2 ferm_active on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection refusedconnect to host 168.235.111.29 port 5666: Connection refused [15:07:29] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 100% [15:09:33] RECOVERY - ping4 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 73.00 ms [15:19:00] RECOVERY - bacula2 SSH on bacula2 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [15:19:52] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://git.io/J1JlH [15:20:17] off for bit from laptop [15:23:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.18, 2.83 [15:25:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.12, 3.13, 2.85 [15:25:17] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://git.io/J1JlH [15:28:01] I know args.time is missing [15:28:04] I'll fix later [15:28:17] args.force even [15:31:31] RECOVERY - bacula2 PowerDNS Recursor on bacula2 is OK: DNS OK: 0.429 seconds response time. miraheze.org returns 167.114.2.161,2607:5300:201:3100::1d3,2607:5300:205:200::1c30,51.222.25.132 [15:31:53] RECOVERY - bacula2 Bacula Databases db12 on bacula2 is OK: OK: Full, 70416 files, 28.15GB, 2021-11-07 00:36:00 (1.1 weeks ago) [15:32:08] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6467 files, 49.72MB, 2021-10-31 00:51:00 (2.1 weeks ago) [15:32:16] RECOVERY - bacula2 Current Load on bacula2 is OK: OK - load average: 0.49, 0.42, 0.17 [15:32:21] RECOVERY - bacula2 Bacula Databases db13 on bacula2 is OK: OK: Full, 70911 files, 26.39GB, 2021-11-07 00:59:00 (1.1 weeks ago) [15:32:36] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:32:46] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [15:33:05] RECOVERY - bacula2 ferm_active on bacula2 is OK: OK ferm input default policy is set [15:33:07] RECOVERY - bacula2 Disk Space on bacula2 is OK: DISK OK - free space: / 2524504 MB (99% inode=99%); [15:33:07] RECOVERY - bacula2 NTP time on bacula2 is OK: NTP OK: Offset 0.003641754389 secs [15:33:12] RECOVERY - bacula2 conntrack_table_size on bacula2 is OK: OK: nf_conntrack is 0 % full [15:33:26] RECOVERY - bacula2 Bacula Daemon on bacula2 is OK: PROCS OK: 2 processes with UID = 114 (bacula) [15:33:31] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 28932 MB (6% inode=98%); [15:36:25] RECOVERY - bacula2 Bacula Databases db11 on bacula2 is OK: OK: Full, 81266 files, 13.66GB, 2021-11-07 00:13:00 (1.1 weeks ago) [15:46:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 3.83, 3.39 [15:48:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.64, 3.37 [15:50:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.20, 4.27, 3.64 [15:55:40] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:50] RECOVERY - bacula2 Bacula Databases db11 on bacula2 is OK: OK: Full, 81266 files, 13.66GB, 2021-11-07 00:13:00 (1.1 weeks ago) [16:00:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 4.00, 3.87 [16:03:14] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:23] RECOVERY - bacula2 Bacula Databases db12 on bacula2 is OK: OK: Full, 70416 files, 28.15GB, 2021-11-07 00:36:00 (1.1 weeks ago) [16:05:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.26, 4.14, 3.92 [16:10:16] PROBLEM - cp14 Current Load on cp14 is CRITICAL: CRITICAL - load average: 2.07, 2.52, 1.43 [16:10:37] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:11:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.84, 3.85 [16:12:12] PROBLEM - cp14 Current Load on cp14 is WARNING: WARNING - load average: 0.80, 1.90, 1.33 [16:12:39] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6467 files, 49.72MB, 2021-10-31 00:51:00 (2.1 weeks ago) [16:13:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.99, 4.50, 4.08 [16:14:08] RECOVERY - cp14 Current Load on cp14 is OK: OK - load average: 0.81, 1.61, 1.30 [16:14:50] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:16:53] RECOVERY - bacula2 Bacula Databases db13 on bacula2 is OK: OK: Full, 70911 files, 26.39GB, 2021-11-07 00:59:00 (1.1 weeks ago) [16:21:14] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:23:04] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.39, 6.53, 5.17 [16:23:23] RECOVERY - bacula2 Bacula Databases db11 on bacula2 is OK: OK: Full, 81266 files, 13.66GB, 2021-11-07 00:13:00 (1.1 weeks ago) [16:25:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.72, 5.82, 5.07 [16:33:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.54, 3.97 [16:37:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.78, 3.97 [16:43:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.44, 3.78 [16:47:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.15, 3.69, 3.76 [16:48:32] PROBLEM - cp15 Current Load on cp15 is WARNING: WARNING - load average: 1.04, 1.83, 1.27 [16:50:30] RECOVERY - cp15 Current Load on cp15 is OK: OK - load average: 0.41, 1.32, 1.15 [16:51:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.63, 3.73 [16:53:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.08, 4.43, 4.00 [16:56:11] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://git.io/J1JlH [16:57:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.87, 3.89 [16:59:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 4.05, 3.96 [17:01:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.83, 3.90 [17:05:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.39, 4.26, 4.03 [17:11:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.71, 3.71, 3.91 [17:17:04] [02MirahezeMagic] 07Universal-Omega closed pull request 03#303: Add reCAPTCHA response scores to debug logging - 13https://git.io/JXjZm [17:17:06] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J1Utq [17:17:07] [02miraheze/MirahezeMagic] 07Universal-Omega 034071a2a - Add reCAPTCHA response scores to debug logging (#303) [17:17:08] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [17:17:10] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [17:17:59] miraheze/MirahezeMagic - Universal-Omega the build passed. [17:18:14] [02puppet] 07Universal-Omega reviewed pull request 03#2121 commit - 13https://git.io/J1Utl [17:21:31] RhinosF1: thanks! ^ still want me to fix that or is it fine? [17:25:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.09, 3.37 [17:51:45] CosmicAlpha: it's fine as it is [17:51:57] If I say it's a bit [17:52:00] Nit [17:52:04] You can ignore [17:53:46] RhinosF1: alright, thanks! [18:00:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 4.11, 3.56 [18:04:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.81, 3.54 [18:10:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.76, 3.58 [18:14:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.76, 3.66 [18:27:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 2.93, 3.33 [18:52:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.85, 3.52 [18:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.85, 3.56 [19:01:28] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.polandballwiki.com could not be found [19:03:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.07, 3.21, 3.37 [19:39:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.84, 3.86, 3.42 [19:43:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.86, 3.53 [19:49:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.19, 3.35 [19:50:09] [02puppet] 07Universal-Omega opened pull request 03#2126: MediaWiki: fix timeouts - 13https://git.io/J1UMm [20:00:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J1Uyl [20:00:45] [02miraheze/puppet] 07Universal-Omega 03884418f - MediaWiki: fix timeouts (#2126) [20:00:46] [02puppet] 07paladox closed pull request 03#2126: MediaWiki: fix timeouts - 13https://git.io/J1UMm [20:02:48] Thanks paladox, I'm hoping it works. As it'd be good to get timeouts fixed. [20:02:58] ok [20:16:41] RECOVERY - www.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - www.polandballwiki.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:22:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 3.83, 3.52 [20:26:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.54, 3.39, 3.43 [20:28:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.23, 3.37 [20:51:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.14, 4.13, 3.58 [20:53:05] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.47, 7.24, 5.61 [20:53:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.97, 3.58 [20:55:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.78, 6.18, 5.42 [20:55:48] [02mw-config] 07ugochimobi opened pull request 03#4219: TE and XCON protection levels on GP - 13https://git.io/J1UND [20:56:55] miraheze/mw-config - ugochimobi the build passed. [20:59:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 3.16, 3.37 [21:04:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb [21:04:17] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 48.92, 33.42, 21.95 [21:05:37] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 12.50, 7.86, 4.55 [21:08:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:10:06] @CosmicAlpha have time to check on latest PR on LocalSettings.php ? 🤗 [21:12:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.39, 3.24 [21:13:16] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:14:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.30, 3.23 [21:15:23] RECOVERY - bacula2 Bacula Databases db13 on bacula2 is OK: OK: Full, 70911 files, 26.39GB, 2021-11-07 00:59:00 (1.1 weeks ago) [21:15:37] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.17, 5.80, 5.33 [21:17:37] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.03, 4.55, 4.93 [21:17:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.56, 3.36 [21:18:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 12.60, 19.56, 23.24 [21:21:37] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.34, 5.58, 5.26 [21:21:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.71, 3.44 [21:22:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 7.08, 5.68 [21:23:37] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.47, 4.70, 4.97 [21:24:30] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.06, 5.99, 5.45 [21:32:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.93, 16.61, 20.09 [21:35:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.83, 3.75 [21:37:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 4.13, 3.87 [21:38:06] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.29, 4.96, 4.70 [21:39:07] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:39:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.84, 3.62, 3.71 [21:40:00] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.72, 3.71, 4.28 [21:41:19] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6467 files, 49.72MB, 2021-10-31 00:51:00 (2.1 weeks ago) [21:44:55] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.27, 3.06, 2.07 [21:45:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.56, 20.44, 19.91 [21:45:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.01, 3.39 [21:45:55] !log importing two wiki dumps [21:45:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:55] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.74, 3.30, 2.28 [21:47:13] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.27, 20.03, 19.86 [21:48:55] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.11, 2.86, 2.25 [21:49:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.49, 3.50 [21:51:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.80, 3.62 [21:54:35] [02mw-config] 07ugochimobi commented on pull request 03#4219: TE and XCON protection levels on GP - 13https://git.io/J1TTH [21:55:37] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.51, 5.60, 4.64 [21:55:57] .task 8289 [21:55:59] https://phabricator.miraheze.org/T8289 - Import XML dump, authored by Ugochimobi, assigned to None, Priority: Normal, Status: Open [21:56:09] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.12, 26.50, 22.65 [21:56:40] It seems this task is abandoned, Lmao😀 [21:56:55] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.22, 3.52, 2.76 [21:58:55] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.18, 3.36, 2.80 [21:59:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.97, 3.26, 3.39 [22:02:55] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 5.91, 4.28, 3.27 [22:03:57] PROBLEM - mwtask1 Disk Space on mwtask1 is CRITICAL: DISK CRITICAL - free space: / 2700 MB (5% inode=90%); [22:08:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.64, 3.44 [22:10:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.64, 3.48 [22:10:55] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 2.17, 3.59, 3.55 [22:12:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.19, 3.98, 3.60 [22:12:37] [02mw-config] 07Universal-Omega commented on pull request 03#4219: TE and XCON protection levels on GP - 13https://git.io/J1TLP [22:14:52] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:55] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.03, 2.59, 3.16 [22:17:04] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6467 files, 49.72MB, 2021-10-31 00:51:00 (2.1 weeks ago) [22:17:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.64, 3.33, 3.54 [22:18:54] [02mw-config] 07dmehus commented on pull request 03#4219: TE and XCON protection levels on GP - 13https://git.io/J1Tty [22:19:56] [02mw-config] 07ugochimobi opened pull request 03#4220: Inline with PR #4219 - 13https://git.io/J1TqT [22:20:57] [02mw-config] 07ugochimobi commented on pull request 03#4219: TE and XCON protection levels on GP - 13https://git.io/J1Tqc [22:21:01] miraheze/mw-config - ugochimobi the build passed. [22:21:13] [02mw-config] 07Universal-Omega commented on pull request 03#4220: Inline with PR #4219 - 13https://git.io/J1TqB [22:21:15] [02mw-config] 07Universal-Omega closed pull request 03#4220: Inline with PR #4219 - 13https://git.io/J1TqT [22:22:59] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --wiki=risingbeatwiki --username=ItsSunday --auto (END - exit=0) [22:23:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:07] [02mw-config] 07ugochimobi synchronize pull request 03#4219: TE and XCON protection levels on GP - 13https://git.io/J1UND [22:25:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 3.06, 3.37 [22:25:37] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.92, 5.22, 5.98 [22:26:05] miraheze/mw-config - ugochimobi the build passed. [22:26:15] [02mw-config] 07ugochimobi commented on pull request 03#4219: TE and XCON protection levels on GP - 13https://git.io/J1TmB [22:26:39] [02mw-config] 07Universal-Omega closed pull request 03#4219: TE and XCON protection levels on GP - 13https://git.io/J1UND [22:26:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J1Tmg [22:26:42] [02miraheze/mw-config] 07ugochimobi 037deba2f - TE and XCON protection levels on GP (#4219) [22:27:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 6.55, 5.38 [22:27:40] miraheze/mw-config - Universal-Omega the build passed. [22:29:18] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.98, 6.02, 5.32 [22:29:18] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 16.71, 10.07, 6.83 [22:29:56] PROBLEM - mwtask1 Disk Space on mwtask1 is WARNING: DISK WARNING - free space: / 3330 MB (7% inode=90%); [22:33:07] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.72, 7.87, 6.74 [22:33:37] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.24, 3.87, 5.00 [22:33:56] RECOVERY - mwtask1 Disk Space on mwtask1 is OK: DISK OK - free space: / 18551 MB (39% inode=90%); [22:35:02] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.80, 6.59, 6.41 [22:35:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.74, 19.72, 23.91 [22:37:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.00, 21.47, 23.96 [22:39:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.73, 19.30, 22.88 [22:40:01] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=hypotheticalhurricaneswiki --update (END - exit=0) [22:40:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:41] !log [universalomega@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=hypotheticalhurricaneswiki --update (END - exit=256) [22:41:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:42:03] !log [@mw11] starting deploy of {'config': True} to all [22:42:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:42:16] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 12s [22:42:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:50] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.22, 17.12, 20.38 [22:51:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.59, 3.38 [22:54:24] !log [@test3] starting deploy of {'config': True} to skip [22:54:25] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:54:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:25] CosmicAlpha, see DM when you get a chance [22:55:58] CosmicAlpha, ty :) [22:57:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.26, 3.30 [22:58:23] CosmicAlpha thanks for the merge. [22:58:32] dmehus thanks for the review [22:59:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.33, 20.70, 20.51 [23:00:02] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:03] !log [@mw11] starting deploy of {'l10nupdate': True} to all [23:00:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:58] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 14.44, 9.77, 7.32 [23:02:24] No problem Ugochimobi [23:03:01] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 13.92, 8.92, 6.73 [23:03:53] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.31, 7.83, 7.33 [23:07:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.47, 3.32 [23:07:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 43.31, 27.68, 23.19 [23:08:03] RECOVERY - mw11 APT on mw11 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:08:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.72, 9.96, 8.17 [23:09:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.87, 3.48 [23:10:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.51, 7.57, 6.28 [23:12:26] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.85, 7.56, 7.57 [23:12:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.86, 6.76, 6.12 [23:13:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.62, 22.72, 22.55 [23:13:44] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:14:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.69, 7.86, 7.66 [23:14:42] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:15:54] RECOVERY - bacula2 Bacula Databases db11 on bacula2 is OK: OK: Full, 81266 files, 13.66GB, 2021-11-07 00:13:00 (1.1 weeks ago) [23:16:00] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.13, 7.50, 6.52 [23:16:55] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 3.74, 6.69, 7.73 [23:18:00] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:19:42] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:20:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.99, 7.18, 7.57 [23:21:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.90, 22.82, 22.47 [23:22:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.06, 4.79, 6.50 [23:23:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.43, 21.42, 22.06 [23:25:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.46, 3.57, 3.80 [23:26:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.50, 5.22, 6.63 [23:26:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.83, 6.63, 7.00 [23:27:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.33, 17.95, 20.40 [23:27:51] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:28:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.53, 5.56, 6.56 [23:29:23] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1761s [23:29:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:54] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6467 files, 49.72MB, 2021-10-31 00:51:00 (2.1 weeks ago) [23:30:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.56, 6.69, 7.00 [23:30:41] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:32:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.35, 5.85, 6.66 [23:32:51] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:35:00] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6467 files, 49.72MB, 2021-10-31 00:51:00 (2.1 weeks ago) [23:35:41] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:37:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 2.86, 3.39 [23:39:30] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.44, 24.19, 20.94 [23:39:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.39, 4.39, 3.78 [23:41:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.95, 22.29, 20.65 [23:41:40] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.61, 3.98, 3.70 [23:42:44] !log [@mw11] finished deploy of {'l10nupdate': True} to all - SUCCESS in 2562s [23:42:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:29] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.14, 20.19, 20.05 [23:44:21] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:44:26] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.14, 22.90, 19.71 [23:46:21] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.16, 19.49, 18.84 [23:49:24]