[00:02:49] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:17] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 7789 MB (19% inode=96%); [00:05:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.99, 10.12, 9.03 [00:06:07] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 6.853 second response time [00:06:13] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:08:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.47, 9.92, 9.17 [00:18:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.29, 9.73, 9.29 [00:21:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.01, 9.69, 9.39 [00:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:31:08] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:32:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.81, 1.89, 1.77 [00:34:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.06, 1.84, 1.70 [00:35:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.70, 1.52, 1.66 [00:36:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.19, 9.37, 9.95 [00:37:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.47, 1.28, 1.51 [00:38:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:40:00] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.07, 7.02, 8.13 [00:43:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:45:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.43, 9.88, 9.74 [00:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:46:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.78, 3.40 [00:50:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.41, 3.35 [00:53:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.15, 3.22, 3.27 [00:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:22:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.73, 3.15 [01:23:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.80, 8.62, 9.59 [01:25:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.19, 3.04 [01:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:29:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 8.88, 9.27 [01:32:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.64, 8.31, 8.98 [01:35:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 9.26, 9.24 [01:38:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.13, 8.96, 9.14 [01:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:51:00] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.07, 7.75, 8.42 [01:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:01:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.88, 8.85, 8.58 [02:03:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:04:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.01, 9.93, 9.06 [02:07:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.31, 8.94, 8.84 [02:10:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.22, 10.29, 9.38 [02:11:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 8.66, 7.36 [02:13:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.89, 9.40, 8.52 [02:14:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.96, 8.67, 7.64 [02:16:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.53, 9.87, 8.85 [02:20:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.57, 8.58, 8.56 [02:20:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.45, 7.88, 7.67 [02:23:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.95, 8.23, 8.45 [02:30:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.08, 10.14, 9.19 [02:33:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.97, 9.83, 9.24 [02:39:17] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.82, 7.35, 8.47 [02:40:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.48, 1.96, 1.65 [02:42:06] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.99, 1.79, 1.52 [02:42:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 9.42, 9.89 [02:43:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.49, 1.23, 1.42 [02:44:00] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [02:45:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.56, 9.97, 10.03 [02:48:22] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.32, 1.68, 1.61 [02:48:55] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.019 second response time [03:01:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.39, 8.91, 8.64 [03:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 9.14, 8.74 [03:08:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.99, 8.87, 8.73 [03:09:00] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:11:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.06, 7.86, 8.38 [03:12:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.23, 9.24, 9.95 [03:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:16:07] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.010 second response time [03:18:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.42, 9.65, 9.81 [03:19:07] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.024 second response time [03:21:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.31, 8.56, 9.40 [03:25:45] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [03:31:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.26, 6.96, 8.24 [03:40:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:44:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.62, 10.24, 9.10 [03:47:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.31, 9.25, 8.94 [03:49:57] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:50:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.26, 7.93, 8.48 [03:50:23] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.009 second response time [03:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:53:16] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.025 second response time [03:57:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.77, 1.39 [04:00:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.53, 1.36 [04:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:14:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.58, 3.09 [04:20:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.35, 3.17 [04:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:49:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.24, 8.70, 8.05 [04:55:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.58, 8.24, 8.08 [05:38:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.47, 9.35, 8.33 [05:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:41:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.50, 8.93, 8.36 [05:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:44:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.18, 7.93, 8.09 [05:49:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.48, 3.24 [05:50:43] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.015 second response time [05:55:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.29, 3.29 [06:02:33] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 9.529 second response time [06:07:43] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.67, 4.96, 7.82 [06:13:34] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.64, 4.50, 6.70 [06:16:18] !log reception@mwtask111:/srv/mediawiki/w/extensions/ManageWiki/maintenance$ sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki unicodesubsetswiki [06:16:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:25:49] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.02, 1.87, 1.32 [06:28:45] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.84, 1.79, 1.39 [06:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:31:39] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.41, 1.57, 1.37 [06:42:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:50:21] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.96, 1.81, 1.52 [06:53:15] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.34, 1.98, 1.64 [06:56:10] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.53, 1.83, 1.64 [06:57:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:59:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.25, 1.58, 1.58 [07:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:46:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:56:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:11:52] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [09:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:31:30] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [09:33:22] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [09:35:58] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a69bb1e21875...7893277cfa13 [09:36:00] [02miraheze/mw-config] 07RhinosF1 037893277 - Update LocalSettings.php [09:36:01] [02mw-config] 07RhinosF1 synchronize pull request 03#4439: LS: disable usermerge - 13https://github.com/miraheze/mw-config/pull/4439 [09:36:35] [02mw-config] 07RhinosF1 closed pull request 03#4439: LS: disable usermerge - 13https://github.com/miraheze/mw-config/pull/4439 [09:36:37] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-4 [09:36:38] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/417720f3c087...d2455cb74e62 [09:36:40] [02miraheze/mw-config] 07RhinosF1 03d2455cb - LS: disable usermerge (#4439) [09:36:41] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-4 - 13https://github.com/miraheze/mw-config [09:36:52] miraheze/mw-config - RhinosF1 the build passed. [09:37:41] miraheze/mw-config - RhinosF1 the build passed. [09:38:12] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [09:41:47] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d2455cb74e62...36c9f51ff561 [09:41:49] [02miraheze/mw-config] 07RhinosF1 0336c9f51 - Logging: enable 2 streams [09:42:48] miraheze/mw-config - RhinosF1 the build passed. [09:48:02] !log [@test101] starting deploy of {'config': True} to skip [09:48:03] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [09:48:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:48:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:52:13] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [09:52:48] !log [@mwtask111] starting deploy of {'config': True} to all [09:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:52:57] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [09:53:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:18:43] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [10:53:53] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [10:56:43] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [10:59:27] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:20] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.026 second response time [11:02:30] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:08:39] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:10:45] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:11:03] Reception123: ^ has become rewrite mediawiki python stuff [11:11:50] ack [11:12:03] let me know when it will be ready to merge [11:12:27] Reception123: i'm down to 1 flake8 error for over complex [11:12:36] then i think an argument with mypy [11:14:44] ok [11:16:38] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:22:04] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:24:34] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:28:28] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:29:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.37, 2.42, 1.80 [11:31:54] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.66, 1.59, 1.58 [11:32:36] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:41:01] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:43:14] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:46:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.07, 2.69 [11:49:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 2.91, 2.70 [11:52:42] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [11:54:50] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [12:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:14:00] !log [rhinos@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@https://localhost [12:14:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:33] !log [rhinos@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@https://localhost [12:14:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:17:51] !log [rhinos@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@https://localhost [12:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:25:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.98, 1.85, 1.48 [12:28:33] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.19, 1.56, 1.43 [12:32:55] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [12:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:33:38] Reception123: I have 1 possible bug and UO's review fixes then good [12:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:40:22] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [12:47:34] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [12:48:24] ack [12:51:52] Reception123: i thought that would fixed it [12:52:01] now GHA is giving DNS errors [12:55:50] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.72, 1.68, 1.55 [12:58:49] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.40, 1.55, 1.52 [13:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:32:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:40:28] !log > INSERT INTO content_models (model_id, model_name) VALUES (1, 'wikitext'); INSERT INTO content_models (model_id, model_name) VALUES (2, 'css'); INSERT INTO content_models (model_id, model_name) VALUES (3, 'javascript'); INSERT INTO content_models (model_id, model_name) VALUES (4, 'Scribunto'); INSERT INTO content_models (model_id, model_name) VALUES (5, 'json'); [13:40:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:40:39] !log (spektrowiki) [13:40:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:48:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:53:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:54:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.29, 9.32, 8.39 [13:57:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 9.29, 8.56 [14:00:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.34, 8.49, 8.39 [14:16:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:34:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:40:07] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [14:42:51] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:23] Reception123: lgtm now [14:43:33] i doing a ttest [14:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:43:43] on test101 [14:43:56] !log [rhinos@test101] starting deploy of {'world': True} to all [14:44:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:44:16] let's see what it picks up [14:44:46] !log [rhinos@test101] starting deploy of {'config': True} to all [14:44:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:46:12] !log [rhinos@test101] starting deploy of {'config': True} to all [14:46:13] !log [rhinos@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [14:46:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:46:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:48:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:49:24] [02puppet] 07RhinosF1 synchronize pull request 03#2125: mwdeploy: rewrite rsync helper - 13https://github.com/miraheze/puppet/pull/2125 [14:50:17] !log [rhinos@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@mw1.miraheze.org [14:50:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 3.76, 3.20 [14:50:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:51:51] Reception123: it seems ok [14:52:11] the only change is if prep fails, it's not logging, i push a fix in a sec [14:54:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:56:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.32, 3.20 [15:02:00] !log [rhinos@test101] starting deploy of {'world': True, 'l10nupdate': True, 'l10n': True} to skip [15:02:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:04:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:27:00] !log [rhinos@test101] finished deploy of {'world': True, 'l10nupdate': True, 'l10n': True} to skip - SUCCESS in 1500s [15:27:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:28:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:31:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.66, 3.42 [15:31:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.63, 8.07, 7.52 [15:34:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.08, 4.31, 3.66 [15:34:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.47, 6.83, 7.13 [15:37:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.65, 3.53 [15:40:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.74, 3.06, 3.34 [15:43:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:48:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:53:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:09:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:16:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:49:11] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 5.197 second response time [16:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:55:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:07:45] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=elassintwiki --username-prefix=elassint --no-updates --report 1 /home/reception/Elassintpedia-finalmodded.xml (START) [17:07:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:07] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 9.44, 8.46 [17:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:13:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.51, 3.05 [17:16:09] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.210 second response time [17:16:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.35, 8.74, 7.73 [17:16:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 9.24, 8.26 [17:17:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.17, 3.24, 3.02 [17:17:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.52, 8.91, 8.37 [17:18:18] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sanctumoflightwiki --username-prefix=wikia:fireemblem --no-updates --report 1 /home/reception/Fire+Emblem+Wiki-20220205054129.xml (START) [17:18:19] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sanctumoflightwiki --username-prefix=wikia:fireemblem --no-updates --report 1 /home/reception/Fire+Emblem+Wiki-20220205054129.xml (END - exit=256) [17:18:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.63, 7.26, 7.33 [17:20:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 9.01, 8.46 [17:21:37] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=elassintwiki --username-prefix=elassint --no-updates --report 1 /home/reception/Elassintpedia-finalmodded.xml (END - exit=256) [17:21:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:22:04] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=elassintwiki (START) [17:22:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:22:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.74, 8.33, 8.15 [17:22:56] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sanctumoflightwiki (START) [17:23:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.64, 8.43, 8.33 [17:23:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.22, 8.30, 8.33 [17:26:58] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=elassintwiki (END - exit=0) [17:27:00] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sanctumoflightwiki (END - exit=0) [17:27:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:27:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:28:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 9.86, 9.10 [17:29:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.66, 9.11, 8.55 [17:30:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.46, 9.23, 8.82 [17:31:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.07, 8.70, 8.47 [17:31:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:31:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.68, 9.05, 8.93 [17:32:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.93, 8.18, 8.31 [17:33:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.11, 9.78, 9.07 [17:34:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.44, 8.22, 8.33 [17:34:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 9.54, 9.13 [17:36:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.55, 9.71, 9.25 [17:37:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.73, 9.21, 9.11 [17:38:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:39:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 8.87, 8.62 [17:40:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 9.57, 9.25 [17:41:33] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.008 second response time [17:42:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.45, 8.22, 8.42 [17:43:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.67, 8.61, 8.93 [17:50:45] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.043 second response time [17:51:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.73, 9.60, 9.20 [17:52:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.51, 7.97, 8.50 [17:53:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.45, 9.78, 8.72 [17:54:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 9.20, 9.13 [17:56:11] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.95, 2.24, 1.55 [17:56:25] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.81, 8.17, 6.53 [17:56:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.54, 9.38, 8.77 [17:58:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 9.98, 9.39 [17:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:59:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.17, 1.67, 1.45 [17:59:32] PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fbpml.org' expires in 15 day(s) (Mon 21 Feb 2022 17:45:02 GMT +0000). [18:01:55] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.qadrishattari.xyz' expires in 15 day(s) (Mon 21 Feb 2022 17:48:40 GMT +0000). [18:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:03:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b7b13e93f8e9...91d64d4fa34e [18:03:20] [02miraheze/ssl] 07MirahezeSSLBot 0391d64d4 - Bot: Update SSL cert for wiki.fbpml.org [18:05:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/91d64d4fa34e...d0544bc92442 [18:05:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.89, 9.66, 9.13 [18:05:59] [02miraheze/ssl] 07MirahezeSSLBot 03d0544bc - Bot: Update SSL cert for wiki.qadrishattari.xyz [18:06:42] PROBLEM - olwest.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'olwest.org' expires in 15 day(s) (Mon 21 Feb 2022 17:51:48 GMT +0000). [18:07:10] PROBLEM - wmworld.sktz.live - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wmworld.sktz.live' expires in 15 day(s) (Mon 21 Feb 2022 17:52:45 GMT +0000). [18:07:35] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'privacywiki.xyz' expires in 15 day(s) (Mon 21 Feb 2022 17:56:53 GMT +0000). [18:11:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.13, 3.41, 3.28 [18:12:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.76, 8.59, 8.49 [18:12:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.73, 9.86, 9.43 [18:13:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d0544bc92442...84dafef981e9 [18:13:34] [02miraheze/ssl] 07MirahezeSSLBot 0384dafef - Bot: Update SSL cert for olwest.org [18:14:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.88, 3.48 [18:15:02] PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.slimelab.net' expires in 15 day(s) (Mon 21 Feb 2022 17:46:50 GMT +0000). [18:15:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.19, 7.64, 8.15 [18:15:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.92, 9.94, 9.52 [18:17:39] PROBLEM - gp.ct777.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gp.ct777.cf' expires in 15 day(s) (Mon 21 Feb 2022 17:43:52 GMT +0000). [18:18:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/84dafef981e9...fe5d51735f6e [18:18:53] [02miraheze/ssl] 07MirahezeSSLBot 03fe5d517 - Bot: Update SSL cert for privacywiki.xyz [18:20:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.85, 2.73, 3.12 [18:22:02] PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.hoolehistoryheritagesociety.org.uk' expires in 15 day(s) (Mon 21 Feb 2022 17:50:45 GMT +0000). [18:25:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fe5d51735f6e...a8a063bd0c95 [18:25:09] [02miraheze/ssl] 07MirahezeSSLBot 03a8a063b - Bot: Update SSL cert for wiki.slimelab.net [18:25:58] PROBLEM - you.r-fit.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'you.r-fit.cc' expires in 15 day(s) (Mon 21 Feb 2022 17:53:45 GMT +0000). [18:26:56] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.009 second response time [18:27:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a8a063bd0c95...5f7f159297c8 [18:27:39] [02miraheze/ssl] 07MirahezeSSLBot 035f7f159 - Bot: Update SSL cert for gp.ct777.cf [18:27:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 9.60, 9.90 [18:29:09] RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Fri 06 May 2022 17:03:13 GMT +0000. [18:29:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5f7f159297c8...54722750ec8b [18:29:55] [02miraheze/ssl] 07MirahezeSSLBot 035472275 - Bot: Update SSL cert for www.hoolehistoryheritagesociety.org.uk [18:30:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 9.50, 9.79 [18:32:13] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Fri 06 May 2022 17:05:51 GMT +0000. [18:34:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/54722750ec8b...c702f87b06e5 [18:34:37] [02miraheze/ssl] 07MirahezeSSLBot 03c702f87 - Bot: Update SSL cert for wmworld.sktz.live [18:36:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c702f87b06e5...e504e699ce55 [18:36:13] [02miraheze/ssl] 07MirahezeSSLBot 03e504e69 - Bot: Update SSL cert for you.r-fit.cc [18:36:32] RECOVERY - olwest.org - LetsEncrypt on sslhost is OK: OK - Certificate 'olwest.org' will expire on Fri 06 May 2022 17:13:26 GMT +0000. [18:36:42] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.156 second response time [18:37:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.01, 9.69, 9.86 [18:38:27] RECOVERY - privacywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'privacywiki.xyz' will expire on Fri 06 May 2022 17:18:46 GMT +0000. [18:46:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.96, 9.62, 9.60 [18:46:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.41, 3.42, 3.18 [18:50:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 2.77, 2.97 [18:54:52] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [18:56:42] RECOVERY - you.r-fit.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Fri 06 May 2022 17:36:05 GMT +0000. [19:03:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.50, 9.07, 8.14 [19:07:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 8.24, 7.97 [19:07:14] RECOVERY - wmworld.sktz.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wmworld.sktz.live' will expire on Fri 06 May 2022 17:34:30 GMT +0000. [19:16:21] RECOVERY - wiki.slimelab.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimelab.net' will expire on Fri 06 May 2022 17:25:02 GMT +0000. [19:17:31] RECOVERY - gp.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'gp.ct777.cf' will expire on Fri 06 May 2022 17:27:32 GMT +0000. [19:21:09] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:22:23] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Fri 06 May 2022 17:29:48 GMT +0000. [19:28:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.32, 3.62, 3.22 [19:31:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.32, 3.19 [19:32:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.29, 8.79, 9.48 [19:35:41] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.78, 1.53 [19:36:28] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:37:29] PROBLEM - mon111 icinga.miraheze.org HTTPS on mon111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.009 second response time [19:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:38:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.46, 1.44 [19:39:05] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4289 MB (10% inode=96%); [19:39:35] RECOVERY - cloud11 SMART on cloud11 is OK: OK: [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean [19:41:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.28, 7.53, 8.49 [19:44:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.78, 3.49 [19:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:46:45] RECOVERY - mon111 icinga.miraheze.org HTTPS on mon111 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 2.609 second response time [19:48:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.40, 10.07, 9.38 [19:50:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 3.19, 3.36 [19:50:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.50, 8.52, 9.55 [19:51:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.08, 9.07, 9.11 [19:53:49]