[00:00:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.76, 8.83, 9.20 [00:02:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.82, 7.62, 8.29 [00:03:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.32, 8.93, 9.74 [00:03:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.58, 3.16 [00:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:05:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.71, 9.88, 9.41 [00:06:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 3.70, 3.25 [00:06:13] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 447 failures. Last run 5 minutes ago with 447 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_conntrack],File[/usr/lib/nagios/plugins/check_ferm],File[/usr/lib/nagios/plugins/check_smart],File[/usr/lib/nagios/plugins/check_ipmi_sensors] [00:08:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.14, 9.24, 9.29 [00:09:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.38, 3.18 [00:09:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.84, 9.90, 9.42 [00:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:12:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.85, 3.38, 3.24 [00:12:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.45, 9.66, 9.63 [00:12:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 9.42, 9.33 [00:14:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.84, 8.83, 9.35 [00:18:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.72, 3.52 [00:20:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.09, 10.22, 9.72 [00:21:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.95, 3.28, 3.38 [00:29:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.68, 7.70, 8.37 [00:29:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.81, 9.86, 9.74 [00:32:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.56, 8.46, 7.78 [00:32:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.39, 2.24, 1.75 [00:33:06] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:34:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.08, 8.64, 8.74 [00:34:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.91, 7.26, 7.43 [00:35:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.08, 1.82, 1.69 [00:37:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 9.07, 8.92 [00:38:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.09, 9.68, 9.52 [00:38:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.77, 2.27, 1.88 [00:40:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.38, 9.05 [00:41:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.73, 1.73 [00:43:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.44, 8.13, 8.68 [00:44:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.71, 9.31, 9.36 [00:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:44:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.23, 1.50, 1.64 [00:46:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.94, 7.18, 8.21 [00:48:54] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.86, 2.94, 1.82 [00:50:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.69, 9.52, 9.32 [00:53:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.16, 9.22 [00:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:01:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.39, 6.95, 8.25 [01:03:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.60, 9.12, 8.63 [01:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:06:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.56, 2.03, 1.72 [01:09:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.62, 1.62 [01:12:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.45, 3.20 [01:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:14:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.22, 10.19, 9.21 [01:18:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 3.11, 3.16 [01:18:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.36, 9.79, 9.85 [01:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:22:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.31, 11.39, 10.45 [01:26:50] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:27:35] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:59] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:28:55] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:55] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:51] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:20] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:52] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:57] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:33:19] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:54] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:33:58] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:15] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:36:23] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 3.660 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:36:34] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [01:37:06] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [01:37:15] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [01:38:57] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: connect to address 2a10:6740::6:208 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [01:43:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.32, 9.02, 8.47 [01:46:08] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:46:24] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:46:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.67, 8.22, 8.25 [01:47:08] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:16] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:07] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:04] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 57 minutes ago with 0 failures [01:53:22] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:53:31] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [01:53:34] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [01:53:49] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.001726776361 secs [01:53:51] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.531 second response time [01:53:52] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.491 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:54:26] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [01:54:30] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [01:54:32] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [01:54:56] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 50744 MB (76% inode=90%); [02:00:27] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet last ran 1 hour ago [02:06:58] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:26] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet last ran 1 hour ago [02:13:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-0/±0] 13https://github.com/miraheze/mw-config/commit/355cb0622ae9 [02:13:56] [02miraheze/mw-config] 07Universal-Omega 03355cb06 - Create MirahezeFunctions.php [02:13:58] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [02:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:15:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/355cb0622ae9...000a0a553216 [02:15:46] [02miraheze/mw-config] 07Universal-Omega 03000a0a5 - Update LocalSettings.php [02:16:28] [02mw-config] 07Universal-Omega opened pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:17:34] miraheze/mw-config - Universal-Omega the build has errored. [02:17:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/000a0a553216...4106c5e131ac [02:18:00] [02miraheze/mw-config] 07Universal-Omega 034106c5e - Update MirahezeFunctions.php [02:18:01] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:19:03] miraheze/mw-config - Universal-Omega the build has errored. [02:21:40] [02mw-config] 07Universal-Omega commented on pull request 03#4462: Remove UserMerge (T8671) - 13https://github.com/miraheze/mw-config/pull/4462#issuecomment-1024810956 [02:21:54] [02mw-config] 07Universal-Omega closed pull request 03#4462: Remove UserMerge (T8671) - 13https://github.com/miraheze/mw-config/pull/4462 [02:22:06] [02mw-config] 07Universal-Omega commented on pull request 03#4464: Remove UserMerge (T8671) - 13https://github.com/miraheze/mw-config/pull/4464#issuecomment-1024811051 [02:22:07] [02mw-config] 07Universal-Omega closed pull request 03#4464: Remove UserMerge (T8671) - 13https://github.com/miraheze/mw-config/pull/4464 [02:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:22:45] [02mw-config] 07Universal-Omega commented on pull request 03#4463: Remove UserMerge (T8671) - 13https://github.com/miraheze/mw-config/pull/4463#issuecomment-1024811152 [02:22:46] [02mw-config] 07Universal-Omega closed pull request 03#4463: Remove UserMerge (T8671) - 13https://github.com/miraheze/mw-config/pull/4463 [02:23:20] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:23:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.94, 9.18, 8.64 [02:25:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 9.19, 8.84 [02:26:34] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet last ran 1 hour ago [02:27:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.84, 1.62 [02:30:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.22, 8.26, 8.47 [02:30:19] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:31:08] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:31:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4106c5e131ac...6c49e61a3dab [02:31:11] [02miraheze/mw-config] 07Universal-Omega 036c49e61 - Update MirahezeFunctions.php [02:31:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6c49e61a3dab...bf6bf1f0f410 [02:31:30] [02miraheze/mw-config] 07Universal-Omega 03bf6bf1f - Update LocalSettings.php [02:31:32] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:31:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.62, 1.58 [02:32:00] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 0.049 second response time [02:32:17] miraheze/mw-config - Universal-Omega the build has errored. [02:32:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.59, 9.24, 8.88 [02:32:38] miraheze/mw-config - Universal-Omega the build has errored. [02:32:45] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.234 second response time [02:32:57] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14552 bytes in 0.186 second response time [02:33:28] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.335 second response time [02:33:32] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22202 bytes in 3.612 second response time [02:33:41] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet last ran 1 hour ago [02:33:48] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 1.697 second response time [02:35:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 8.96, 8.84 [02:36:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bf6bf1f0f410...4a0f8bb7710b [02:36:41] [02miraheze/mw-config] 07Universal-Omega 034a0f8bb - Update LocalSettings.php [02:36:42] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:37:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4a0f8bb7710b...41d0090a9d3b [02:37:23] [02miraheze/mw-config] 07Universal-Omega 0341d0090 - Update ManageWikiExtensions.php [02:37:24] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:37:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.81, 9.57, 9.04 [02:37:42] miraheze/mw-config - Universal-Omega the build has errored. [02:39:23] miraheze/mw-config - Universal-Omega the build has errored. [02:40:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03revert-247-Universal-Omega-patch-4 [+1/-0/±0] 13https://github.com/miraheze/CreateWiki/commit/e52f5d956da0 [02:40:05] [02miraheze/CreateWiki] 07Universal-Omega 03e52f5d9 - Revert "Delete DBListGenerator.php (#247)" [02:40:06] [02CreateWiki] 07Universal-Omega created branch 03revert-247-Universal-Omega-patch-4 - 13https://github.com/miraheze/CreateWiki [02:40:08] [02CreateWiki] 07Universal-Omega opened pull request 03#288: Revert "Delete DBListGenerator.php" - 13https://github.com/miraheze/CreateWiki/pull/288 [02:40:14] [02CreateWiki] 07Universal-Omega closed pull request 03#288: Revert "Delete DBListGenerator.php" - 13https://github.com/miraheze/CreateWiki/pull/288 [02:40:15] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/CreateWiki/compare/8a265d50e4e3...fe0c8b62aaf5 [02:40:16] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:40:17] [02miraheze/CreateWiki] 07Universal-Omega 03fe0c8b6 - Revert "Delete DBListGenerator.php (#247)" (#288) [02:40:18] [02CreateWiki] 07Universal-Omega deleted branch 03revert-247-Universal-Omega-patch-4 - 13https://github.com/miraheze/CreateWiki [02:40:20] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03revert-247-Universal-Omega-patch-4 [02:41:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.48, 10.35, 9.46 [02:43:51] [02WikiDiscover] 07lens0021 synchronize pull request 03#77: phpcs: Enable MediaWiki.Commenting.FunctionComment - 13https://github.com/miraheze/WikiDiscover/pull/77 [02:44:14] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:44:44] miraheze/CreateWiki - Universal-Omega the build passed. [02:45:34] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/55100f16f56f [02:45:35] [02miraheze/CreateWiki] 07Universal-Omega 0355100f1 - DBListGenerator: only DBname [02:45:37] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [02:45:38] [02CreateWiki] 07Universal-Omega opened pull request 03#289: DBListGenerator: only DBname - 13https://github.com/miraheze/CreateWiki/pull/289 [02:45:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 8.73, 8.66 [02:47:31] [02WikiDiscover] 07Universal-Omega reviewed pull request 03#77 commit - 13https://github.com/miraheze/WikiDiscover/pull/77#discussion_r794988154 [02:47:31] miraheze/CreateWiki - Universal-Omega the build passed. [02:48:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.22, 8.98, 9.24 [02:49:08] miraheze/WikiDiscover - lens0021 the build passed. [02:49:23] miraheze/CreateWiki - Universal-Omega the build has errored. [02:49:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.39, 7.40, 8.19 [02:50:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.89, 9.51, 9.96 [02:53:14] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/55100f16f56f...c375ce059aa7 [02:53:15] [02miraheze/CreateWiki] 07Universal-Omega 03c375ce0 - Update DBListGenerator.php [02:53:17] [02CreateWiki] 07Universal-Omega synchronize pull request 03#289: DBListGenerator: only DBname - 13https://github.com/miraheze/CreateWiki/pull/289 [02:53:43] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:50] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:54:03] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:54:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.91, 9.51, 9.82 [02:54:12] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:54:32] [02WikiDiscover] 07lens0021 reviewed pull request 03#77 commit - 13https://github.com/miraheze/WikiDiscover/pull/77#discussion_r794988865 [02:55:07] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:18] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.77, 6.63, 7.99 [02:57:05] [02CreateWiki] 07Universal-Omega closed pull request 03#289: DBListGenerator: only DBname - 13https://github.com/miraheze/CreateWiki/pull/289 [02:57:06] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/fe0c8b62aaf5...8003bfba3b57 [02:57:08] [02miraheze/CreateWiki] 07Universal-Omega 038003bfb - DBListGenerator: only DBname (#289) [02:57:09] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [02:57:11] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [02:57:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.44, 9.14, 9.63 [02:58:32] miraheze/CreateWiki - Universal-Omega the build passed. [02:59:23] [02WikiDiscover] 07Universal-Omega reviewed pull request 03#77 commit - 13https://github.com/miraheze/WikiDiscover/pull/77#discussion_r794989267 [02:59:37] [02WikiDiscover] 07Universal-Omega reviewed pull request 03#77 commit - 13https://github.com/miraheze/WikiDiscover/pull/77#discussion_r794989267 [02:59:51] [02WikiDiscover] 07Universal-Omega closed pull request 03#77: phpcs: Enable MediaWiki.Commenting.FunctionComment - 13https://github.com/miraheze/WikiDiscover/pull/77 [02:59:52] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/miraheze/WikiDiscover/compare/93157eedea45...76d114f8e382 [02:59:54] [02miraheze/WikiDiscover] 07lens0021 0376d114f - phpcs: Enable MediaWiki.Commenting.FunctionComment (#77) [03:00:02] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:00:29] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: CRITICAL - Cannot make SSL connection. [03:01:21] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [03:01:31] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [03:01:47] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 50742 MB (76% inode=90%); [03:01:52] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:01:55] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 3.043 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [03:02:00] miraheze/CreateWiki - Universal-Omega the build passed. [03:02:43] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [03:02:46] [02CreateWiki] 07Universal-Omega closed pull request 03#287: set defaults for cw_wikis - 13https://github.com/miraheze/CreateWiki/pull/287 [03:02:48] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8003bfba3b57...d79c23d96255 [03:02:49] [02miraheze/CreateWiki] 07Universal-Omega 03d79c23d - set defaults for cw_wikis (#287) [03:02:51] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [03:02:52] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/CreateWiki [03:03:20] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.003512084484 secs [03:04:19] miraheze/WikiDiscover - Universal-Omega the build passed. [03:04:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.77, 9.67, 9.61 [03:06:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.88, 1.94, 1.95 [03:07:43] miraheze/CreateWiki - Universal-Omega the build passed. [03:08:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.85, 9.60, 9.63 [03:11:00] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:11:22] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:11:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 8.57, 8.31 [03:11:45] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:11:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 10.86, 10.10 [03:12:25] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:13:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.51, 1.01, 1.52 [03:14:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.32, 8.16, 8.17 [03:15:16] [02CreateWiki] 07lens0021 commented on pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1024820025 [03:15:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.41, 9.94, 9.92 [03:17:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:18:50] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:19:25] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:20:08] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:20:27] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:21:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:21:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/41d0090a9d3b...d99a84c33474 [03:21:16] [02miraheze/mw-config] 07Universal-Omega 03d99a84c - Use extract() [03:22:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 9.69, 9.99 [03:22:24] miraheze/mw-config - Universal-Omega the build has errored. [03:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:23:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d99a84c33474...076b3313971f [03:23:18] [02miraheze/mw-config] 07Universal-Omega 03076b331 - Remove (array) [03:23:19] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:24:21] miraheze/mw-config - Universal-Omega the build has errored. [03:25:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.59, 9.99, 10.04 [03:25:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.11, 9.90, 9.82 [03:26:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.98, 8.66, 8.40 [03:29:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.32, 9.02, 9.48 [03:29:17] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [03:29:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/076b3313971f...95d5326d6a38 [03:29:22] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [03:29:23] [02miraheze/mw-config] 07Universal-Omega 0395d5326 - Remove unnecessary defaults [03:29:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:29:28] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [03:29:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.94, 7.86, 8.13 [03:30:07] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [03:30:29] miraheze/mw-config - Universal-Omega the build has errored. [03:32:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.52, 8.83, 9.59 [03:32:19] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.429 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [03:32:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:33:35] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.02557885647 secs [03:34:06] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:34:16] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [03:34:58] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet last ran 2 hours ago [03:35:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 10.26, 9.73 [03:36:43] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.547 second response time [03:36:54] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 0.044 second response time [03:37:59] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:38:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.71, 8.85, 9.29 [03:38:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.96, 9.38, 9.51 [03:38:18] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.225 second response time [03:38:28] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.040 second response time [03:38:35] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.326 second response time [03:39:30] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22202 bytes in 1.372 second response time [03:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:40:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/95d5326d6a38...5c9eae7ec651 [03:40:55] [02miraheze/mw-config] 07Universal-Omega 035c9eae7 - Update MirahezeFunctions.php [03:40:56] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:40:58] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Use new SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:41:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.28, 8.59, 9.19 [03:41:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.13, 9.37, 8.79 [03:41:55] miraheze/mw-config - Universal-Omega the build has errored. [03:42:03] miraheze/mw-config - Universal-Omega the build has errored. [03:44:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.92, 9.12, 8.79 [03:48:08] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.82, 8.08, 8.45 [03:50:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.34, 11.04, 9.94 [03:51:04] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:11] [02CreateWiki] 07Universal-Omega commented on pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1024825680 [03:54:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.70, 9.63, 9.59 [03:54:27] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22202 bytes in 1.610 second response time [04:02:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.64, 7.76, 8.33 [04:08:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.63, 9.33 [04:09:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 8.56, 8.54 [04:11:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.34, 9.77, 9.46 [04:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:18:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.38, 7.88, 8.20 [04:23:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.33, 10.10, 9.68 [04:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:24:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.01, 8.75, 8.55 [04:25:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.02, 1.79, 1.60 [04:26:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.32, 9.55, 9.55 [04:27:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.44, 9.08, 8.67 [04:28:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.37, 1.48 [04:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:30:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.33, 8.29, 8.45 [04:32:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.75, 3.39 [04:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:33:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.10, 8.22, 7.46 [04:35:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.27, 3.25, 3.23 [04:37:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.76, 9.52, 9.05 [04:39:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.36, 7.75, 7.51 [04:40:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.92, 9.45, 9.21 [04:43:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.74, 9.05, 9.09 [04:49:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.36, 9.77, 9.25 [04:52:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.96, 9.48, 9.21 [04:55:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.96, 2.12, 1.72 [04:55:37] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.75, 10.64, 9.72 [04:58:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.56, 1.55 [04:58:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.92, 9.86, 9.58 [05:01:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.70, 9.76, 9.56 [05:04:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.67, 8.88, 9.25 [05:07:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 9.18, 9.27 [05:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:08:13] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.29, 2.28, 1.91 [05:11:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.91, 1.62, 1.71 [05:13:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.72, 9.90, 9.70 [05:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:14:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.51, 1.20, 1.54 [05:18:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:18:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.78, 10.14, 9.81 [05:20:57] [02CreateWiki] 07lens0021 synchronize pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283 [05:21:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.59, 8.01, 8.45 [05:21:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.64, 9.13, 9.49 [05:23:46] miraheze/CreateWiki - lens0021 the build has errored. [05:30:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.93, 7.36, 8.43 [05:31:41] [02CreateWiki] 07lens0021 synchronize pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283 [05:36:47] miraheze/CreateWiki - lens0021 the build passed. [05:37:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:40:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.87, 9.07, 8.70 [05:46:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.26, 7.40, 8.12 [05:47:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:51:56] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/b81baa965f02...9f5034a5b195 [05:51:58] [02miraheze/dns] 07Reception123 039f5034a - add m33ks.ml zone [05:52:39] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/9f5034a5b195...230781429d30 [05:52:40] [02miraheze/dns] 07Reception123 032307814 - add microworldwiki.ga zone [05:54:33] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 0.72, 3.00, 2.22 [05:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:01:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.62, 1.38, 1.74 [06:02:57] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/379df1afdc47...83ca133ecea1 [06:02:59] [02miraheze/ssl] 07Reception123 0383ca133 - force redirects for some *.miraheze.org domains to custom domains (T8681) [06:03:23] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:03:53] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:04:11] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:04:13] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:04:30] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:05:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.06, 1.52 [06:06:14] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:07:17] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:07:35] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:07:38] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:12] [02phabricator-extensions] 07Reception123 closed pull request 03#19: Update workflow - 13https://github.com/miraheze/phabricator-extensions/pull/19 [06:08:13] [02miraheze/phabricator-extensions] 07Reception123 pushed 031 commit to 03master [+0/-1/±5] 13https://github.com/miraheze/phabricator-extensions/compare/ae17177b1e07...032e738d2106 [06:08:15] [02miraheze/phabricator-extensions] 07Universal-Omega 03032e738 - Update workflow (#19) [06:08:54] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:54] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:09:10] miraheze/phabricator-extensions - Reception123 the build passed. [06:09:39] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:09:55] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:09:57] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:11:13] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:12:22] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.359 second response time [06:12:22] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.034 second response time [06:13:21] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 50746 MB (76% inode=90%); [06:13:26] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.003783136606 secs [06:13:37] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [06:14:18] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [06:14:21] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22202 bytes in 1.417 second response time [06:14:26] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.129 second response time [06:14:28] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [06:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:14:37] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.312 second response time [06:14:53] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:14:56] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:14:56] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [06:15:00] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [06:15:03] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [06:16:12] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.212 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [06:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:28:27] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:48] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:54] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+1/-0/±2] 13https://github.com/miraheze/ssl/compare/83ca133ecea1...982a83231271 [06:29:56] [02miraheze/ssl] 07Reception123 036759b5a - add wiki.yunachannel.com cert [06:29:57] [02miraheze/ssl] 07Reception123 03982a832 - Merge branch 'master' of https://github.com/miraheze/ssl [06:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:31:47] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:32:05] [02ManageWiki] 07lens0021 opened pull request 03#337: Convert to PSR4 - 13https://github.com/miraheze/ManageWiki/pull/337 [06:35:08] miraheze/ManageWiki - lens0021 the build has errored. [06:36:11] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.321 second response time [06:36:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:38:16] !log restarted pokefanon import [06:38:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.10, 9.01, 8.24 [06:38:55]