[00:00:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.40, 10.59, 9.85 [00:00:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.67, 3.54 [00:01:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 10.52, 10.12 [00:02:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.29, 10.18, 9.78 [00:02:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 3.53, 3.50 [00:02:45] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3f160b7e13e6...5bec9d9fe287 [00:02:47] [02miraheze/MirahezeMagic] 07Universal-Omega 035bec9d9 - Update rebuildVersionCache.php [00:03:00] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/MirahezeMagic', 'ignoretime': True} to all [00:03:01] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/MirahezeMagic', 'ignoretime': True} to all - SUCCESS in 1s [00:03:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:56] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/5bec9d9fe287...8c0ff7ef09ae [00:03:57] [02miraheze/MirahezeMagic] 07Universal-Omega 038c0ff7e - Update rebuildVersionCache.php [00:04:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.36, 11.08, 10.73 [00:04:07] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/MirahezeMagic', 'ignoretime': True} to all [00:04:08] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/MirahezeMagic', 'ignoretime': True} to all - SUCCESS in 1s [00:04:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.61, 12.01, 11.71 [00:04:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.25, 3.37 [00:06:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.81, 10.74, 10.66 [00:06:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.54, 9.30, 9.57 [00:06:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.85, 4.43, 3.84 [00:06:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.43, 11.58, 11.60 [00:09:25] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [00:09:34] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.77, 3.14, 2.19 [00:09:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.99, 10.12, 10.19 [00:11:35] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.28, 3.50, 2.43 [00:13:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.65, 5.14, 4.17 [00:13:34] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.35, 3.52, 2.58 [00:13:44] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:14:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.79, 9.57, 10.09 [00:15:21] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/8c0ff7ef09ae...9efceaddafd5 [00:15:23] [02miraheze/MirahezeMagic] 07Universal-Omega 039efcead - bump to 2.0 and support MW_INSTALL_PATH ENV [00:15:46] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.09, 7.35, 7.95 [00:15:56] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/MirahezeMagic', 'ignoretime': True} to all [00:15:58] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/MirahezeMagic', 'ignoretime': True} to all - SUCCESS in 1s [00:16:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:35] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.28, 3.81, 2.90 [00:18:59] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/9efceaddafd5...613d285535f5 [00:19:01] [02miraheze/MirahezeMagic] 07Universal-Omega 03613d285 - Remove IP override [00:19:35] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.47, 3.31, 2.83 [00:19:46] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.31, 7.83, 8.01 [00:19:47] [02puppet] 07Universal-Omega opened pull request 03#2520: Run rebuildVersionCache from within staging - 13https://github.com/miraheze/puppet/pull/2520 [00:20:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.19, 11.06, 10.54 [00:20:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.72, 10.25 [00:21:40] paladox: are you around to merge ^? [00:21:48] sure [00:22:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.19, 10.92, 10.56 [00:22:04] [02puppet] 07paladox closed pull request 03#2520: Run rebuildVersionCache from within staging - 13https://github.com/miraheze/puppet/pull/2520 [00:22:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f40a2cde954c...1c3efb2d5415 [00:22:07] [02miraheze/puppet] 07Universal-Omega 031c3efb2 - Run rebuildVersionCache from within staging (#2520) [00:22:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.34, 10.82, 10.34 [00:23:13] Thanks! [00:23:46] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.98, 7.62, 7.95 [00:24:42] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/3bb7c3aa325a...4c671ec0176c [00:24:44] [02miraheze/mediawiki] 07Universal-Omega 034c671ec - Update MirahezeMagic [00:26:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.71, 9.55, 10.07 [00:26:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/3031a83d0389...30c989a209fd [00:26:57] [02miraheze/mediawiki] 07Universal-Omega 0330c989a - Update MirahezeMagic [00:29:07] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:29:24] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/613d285535f5...23b0122ffdab [00:29:25] [02miraheze/MirahezeMagic] 07Universal-Omega 0323b0122 - Don't override $wgBaseDirectory [00:32:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.91, 11.71, 11.22 [00:33:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4c671ec0176c...fd8aea1ecd90 [00:33:09] [02miraheze/mediawiki] 07Universal-Omega 03fd8aea1 - Update MirahezeMagic [00:35:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/30c989a209fd...8a11cc4f1eba [00:35:29] [02miraheze/mediawiki] 07Universal-Omega 038a11cc4 - Update MirahezeMagic [00:35:46] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.45, 7.04, 7.26 [00:36:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.88, 11.02, 11.12 [00:37:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.73, 1.98 [00:37:46] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.78, 6.83, 7.16 [00:38:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.82, 11.84, 11.41 [00:39:28] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:39:46] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.17, 7.31, 7.29 [00:40:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.47, 11.10, 11.21 [00:41:46] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.42, 7.30, 7.29 [00:42:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.34, 9.53, 9.13 [00:43:46] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.71, 7.84, 7.49 [00:44:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.16, 10.51, 9.86 [00:45:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 1.78, 1.87 [00:45:46] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.35, 7.64, 7.46 [00:46:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.41, 10.78, 10.04 [00:46:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.87, 11.84, 11.47 [00:46:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.43, 10.02, 9.48 [00:47:08] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'ignoretime': True} to all [00:47:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:28] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'ignoretime': True} to all [00:47:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:10] PROBLEM - es121 Disk Space on es121 is WARNING: DISK WARNING - free space: / 28015 MB (10% inode=99%); [00:48:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.06, 11.09, 11.24 [00:50:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.96, 10.45, 9.81 [00:52:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.63, 10.01, 10.03 [00:52:22] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.56, 9.06, 9.38 [00:53:43] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.57, 9.85, 10.20 [00:55:05] PROBLEM - es121 Current Load on es121 is WARNING: WARNING - load average: 3.73, 3.28, 2.75 [00:55:49] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 7.748 second response time [00:56:36] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.761 second response time [00:57:00] RECOVERY - es121 Current Load on es121 is OK: OK - load average: 2.94, 3.11, 2.75 [00:57:46] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.38, 6.07, 6.69 [00:57:49] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 5.25, 4.06, 2.76 [01:00:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.72, 3.42, 2.54 [01:00:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.76, 10.42, 9.80 [01:02:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.15, 9.60, 10.19 [01:04:26] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.41, 9.86, 9.73 [01:05:23] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:37] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:50] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.39, 3.79, 3.29 [01:05:51] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:05] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.16, 10.70, 9.24 [01:07:50] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.12, 3.25, 3.15 [01:07:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.68, 11.17, 10.60 [01:08:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.16, 10.32, 9.99 [01:09:05] !log sudo -u www-data php /srv/mediawiki/w/maintenance/findBadBlobs.php --wiki=trollpastawiki --mark=true --revisions='5141,5142,5143,5144,5145,5146,5147,5148,5149,5150,5151,5152,5153,5154,5155,5156,5157,5158,5159' [01:10:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 3.18, 3.68, 3.18 [01:10:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:10:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.40, 9.94, 9.88 [01:10:31] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.49, 3.99 [01:10:57] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:02] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.95, 11.46, 9.82 [01:11:36] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:37] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:42] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:44] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:02] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.69, 4.46, 3.47 [01:12:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.39, 3.84, 3.29 [01:12:11] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:18] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:26] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.322 second response time [01:12:59] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 2.741 second response time [01:13:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.53, 11.51, 10.05 [01:13:33] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.022 second response time [01:13:36] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.015 second response time [01:14:02] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 4.09, 4.28, 3.52 [01:14:10] RECOVERY - es121 Disk Space on es121 is OK: DISK OK - free space: / 53164 MB (19% inode=99%); [01:14:17] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.570 second response time [01:14:20] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 7.971 second response time [01:14:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.36, 3.79 [01:15:09] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.239 second response time [01:15:44] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.920 second response time [01:15:44] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.887 second response time [01:16:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.10, 3.64 [01:21:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:21:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.54, 9.73, 9.85 [01:22:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 2.90, 3.37 [01:23:29] !log [universalomega@test101] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'ignoretime': True} to all - SUCCESS in 2181s [01:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:39] [02puppet] 07Universal-Omega opened pull request 03#2521: Use femiwiki-deploy - 13https://github.com/miraheze/puppet/pull/2521 [01:26:28] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.823 second response time [01:26:49] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.325 second response time [01:26:56] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 0.073 second response time [01:27:02] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.956 second response time [01:27:51] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.352 second response time [01:28:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.23, 10.86, 9.47 [01:28:47] [02puppet] 07Universal-Omega synchronize pull request 03#2521: Use femiwiki-deploy - 13https://github.com/miraheze/puppet/pull/2521 [01:29:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:29:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.99, 9.36, 10.20 [01:30:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.01, 10.54, 9.53 [01:30:40] [02puppet] 07Universal-Omega synchronize pull request 03#2521: Use femiwiki-deploy - 13https://github.com/miraheze/puppet/pull/2521 [01:30:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.82, 3.61 [01:31:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.90, 10.48, 9.69 [01:31:34] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.27, 3.40, 2.70 [01:32:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 2.72, 3.58, 3.92 [01:32:14] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:16] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:21] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:26] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.45, 10.14, 9.51 [01:33:00] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:15] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 7.039 second response time [01:34:20] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 7.083 second response time [01:34:23] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.274 second response time [01:34:30] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 5.191 second response time [01:35:04] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 5.295 second response time [01:35:34] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 3.08, 3.39, 2.88 [01:35:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.13, 11.16, 10.53 [01:36:33] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:36:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.46, 10.91, 9.96 [01:37:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.37, 9.75, 9.81 [01:37:20] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:23] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:28] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:40] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.11, 11.27, 10.20 [01:38:58] !log reboot mw111 [01:39:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 0.40, 0.08, 0.03 [01:40:28] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.317 second response time [01:40:31] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.022 second response time [01:40:45] !log reboot mw101 [01:41:09] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:15] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:41:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:41:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.335 second response time [01:41:19] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.322 second response time [01:41:19] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.016 second response time [01:41:22] PROBLEM - mw111 Check Gluster Clients on mw111 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [01:41:51] !log reboot mw112 [01:41:53] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: connect to address 2a10:6740::6:207 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [01:41:58] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:58] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [01:42:03] RECOVERY - es111 Current Load on es111 is OK: OK - load average: 2.59, 2.78, 3.36 [01:42:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:09] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:25] !log reboot mw111 [01:42:28] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:30] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [01:42:32] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.96, 2.33, 0.85 [01:42:39] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [01:42:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:59] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw111 mw112 [01:43:07] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.330 second response time [01:43:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.34, 1.27, 2.00 [01:43:15] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:43:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.433 second response time [01:43:41] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [01:43:50] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.374 second response time [01:43:53] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:43:53] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.016 second response time [01:43:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.93, 11.26, 9.38 [01:44:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:44:21] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:30] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [01:44:34] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 8.092 second response time [01:44:34] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 7.452 second response time [01:44:39] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [01:44:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.09, 10.80, 10.16 [01:45:22] RECOVERY - mw111 Check Gluster Clients on mw111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:45:30] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.021 second response time [01:45:41] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [01:45:50] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.362 second response time [01:45:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.97, 10.87, 9.47 [01:46:03] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.018 second response time [01:46:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.592 second response time [01:46:20] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.018 second response time [01:46:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.55, 10.13, 10.00 [01:46:59] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [01:47:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.79, 9.77, 9.09 [01:47:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.32, 9.86, 9.27 [01:48:16] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.07, 7.56, 6.52 [01:49:25] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.91, 9.14, 8.95 [01:50:14] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.96, 7.61, 6.66 [01:50:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.54, 10.95, 10.39 [01:51:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.32, 8.91, 4.33 [01:51:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.77, 1.92 [01:52:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.83, 9.57, 9.96 [01:52:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.76, 9.75, 5.17 [01:53:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.51, 1.65, 1.86 [01:54:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.49, 3.87, 3.53 [01:54:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.08, 9.40, 5.60 [01:55:24] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'ignoretime': True} to all - SUCCESS in 4076s [01:55:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 4.00, 3.88, 3.57 [01:57:12] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.33, 1.36, 1.68 [01:58:06] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.70, 6.75, 6.61 [02:00:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 10.15, 6.81 [02:02:03] RECOVERY - es111 Current Load on es111 is OK: OK - load average: 2.81, 3.19, 3.37 [02:02:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.81, 9.96, 7.13 [02:06:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.29, 3.93, 3.64 [02:06:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 10.67, 8.05 [02:08:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 3.60, 3.77, 3.62 [02:09:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.59, 1.60 [02:10:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.93, 4.16, 3.77 [02:10:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.46, 8.94 [02:11:12] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.61, 1.61, 1.61 [02:12:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.02, 9.26, 9.06 [02:12:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.86, 11.37, 9.19 [02:14:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.93, 9.53, 9.19 [02:16:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.72, 9.95, 9.13 [02:18:35] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.74, 1.40 [02:20:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.82, 11.02, 9.73 [02:21:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.92, 1.73 [02:22:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.67, 11.23, 9.97 [02:23:12] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.70, 1.68 [02:24:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.34, 1.92, 1.63 [02:28:35] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.65, 1.59 [02:30:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 2.54, 3.32, 3.85 [02:30:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.82, 11.87, 10.67 [02:34:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 11.60, 10.84 [02:36:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.93, 11.05 [02:38:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.53, 11.41, 10.97 [02:38:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.13, 3.43, 3.65 [02:40:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 3.93, 3.41, 3.60 [02:47:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.54, 9.55, 10.15 [02:49:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 1.73, 1.52 [02:50:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.35, 3.68, 3.63 [02:50:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.90, 3.04, 3.79 [02:51:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 2.00, 1.83, 1.58 [02:53:12] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.16, 1.61, 1.53 [02:54:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:55:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 10.88, 10.53 [02:57:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.88, 1.67 [02:57:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.16, 10.61, 10.48 [02:58:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.88, 3.81 [02:59:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 6.29, 3.10, 2.11 [03:00:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 3.52, 3.91, 3.83 [03:01:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.25, 11.22, 10.74 [03:02:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:02:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.72, 4.02, 3.87 [03:02:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.72, 3.74 [03:03:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 11.02, 10.72 [03:06:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 4.05, 3.88 [03:07:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:09:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.99, 9.57, 10.20 [03:15:29] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 17756 MB (13% inode=99%); [03:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:18:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 3.68, 3.76, 3.96 [03:20:03] PROBLEM - es111 Current Load on es111 is CRITICAL: CRITICAL - load average: 4.28, 4.01, 4.03 [03:21:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 10.60, 10.33 [03:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:28:54] miraheze/RemovePII - Universal-Omega the build passed. [03:29:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.15, 9.47, 10.04 [03:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:32:33] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.03, 7.22, 6.14 [03:34:31] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.35, 6.86, 6.14 [03:36:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.71, 9.71, 8.88 [03:36:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.31, 6.56, 6.10 [03:38:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.73, 9.68, 8.97 [03:38:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.51, 3.95 [03:42:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.32, 10.08, 8.53 [03:42:52] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [03:44:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.69, 9.69, 8.60 [03:46:03] PROBLEM - es111 Current Load on es111 is WARNING: WARNING - load average: 1.00, 2.66, 3.69 [03:46:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.58, 3.78 [03:48:03] RECOVERY - es111 Current Load on es111 is OK: OK - load average: 1.13, 2.20, 3.40 [03:48:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.49, 3.72 [03:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:57:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.90, 10.77, 10.21 [03:58:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.41, 4.00 [03:58:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 3.03, 3.40 [03:59:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.69, 9.98, 9.99 [04:02:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.52, 3.88 [04:02:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.69, 3.61 [04:03:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.67, 1.73, 1.34 [04:04:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.51, 3.84 [04:04:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c660c8c623a6 [04:04:41] [02miraheze/mw-config] 07Universal-Omega 03c660c8c - Add CategoryExplorer to ManageWikiExtensions on 1.38 [04:04:42] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [04:04:44] [02mw-config] 07Universal-Omega opened pull request 03#4658: Add CategoryExplorer to ManageWikiExtensions on 1.38 - 13https://github.com/miraheze/mw-config/pull/4658 [04:05:48] miraheze/mw-config - Universal-Omega the build has errored. [04:06:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.17, 3.01, 3.35 [04:07:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.80, 1.96, 1.50 [04:08:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.26, 3.88 [04:08:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c660c8c623a6...cef8e1b16625 [04:08:33] [02miraheze/mw-config] 07Universal-Omega 03cef8e1b - Update ExtensionsTest.php [04:08:34] [02mw-config] 07Universal-Omega synchronize pull request 03#4658: Add CategoryExplorer to ManageWikiExtensions on 1.38 - 13https://github.com/miraheze/mw-config/pull/4658 [04:09:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.59, 1.69, 1.45 [04:09:44] miraheze/mw-config - Universal-Omega the build passed. [04:10:06] [02mw-config] 07Universal-Omega closed pull request 03#4658: Add CategoryExplorer to ManageWikiExtensions on 1.38 - 13https://github.com/miraheze/mw-config/pull/4658 [04:10:08] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [04:10:09] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:10:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/cff4768ddf8a...76450f266a70 [04:10:12] [02miraheze/mw-config] 07Universal-Omega 0376450f2 - Add CategoryExplorer to ManageWikiExtensions on 1.38 (#4658) [04:11:12] miraheze/mw-config - Universal-Omega the build passed. [04:11:44] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [04:11:44] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:13:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/76450f266a70...c2ef87dc9faa [04:13:01] [02miraheze/mw-config] 07Universal-Omega 03c2ef87d - ExtensionsTest covers $wgManageWikiExtensions not $wgManageWikiSettings [04:13:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.17, 2.08, 1.64 [04:14:03] miraheze/mw-config - Universal-Omega the build passed. [04:14:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.63, 3.70 [04:15:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.83, 1.60 [04:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:17:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.18, 2.03, 1.68 [04:18:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.06, 4.05, 3.89 [04:19:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.60, 1.56 [04:20:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.63, 3.76 [04:22:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:22:12] !log [@mwtask111] starting deploy of {'config': True} to all [04:22:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:22:22] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [04:22:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:23:23] !log [@test101] starting deploy of {'config': True} to all [04:23:25] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [04:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:24:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.57, 3.74 [04:28:23] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 2.98, 3.38 [04:32:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.60, 2.82, 3.32 [04:38:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:45:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.42, 3.31 [04:46:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.60, 3.38 [04:48:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:48:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 3.14, 3.24 [04:56:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.21, 2.98 [05:00:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.64, 3.24 [05:04:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.74, 3.84, 3.38 [05:06:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 4.25, 3.61 [05:10:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.85, 3.60 [05:10:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 4.04, 3.37 [05:10:52] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [05:12:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.65, 3.99, 3.67 [05:14:23]