[00:00:04] I want to see it work first [00:00:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:00:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-244d95a [+0/-0/±1] 13https://git.io/JDvjJ [00:01:00] [02miraheze/mediawiki] 07dependabot[bot] 03d8690a7 - Bump skins/Cosmos from `f66fddf` to `244d95a` [00:01:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:01:12] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [00:01:13] [02mediawiki] 07dependabot[bot] edited pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [00:02:37] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:37] well that workflow script is not very performant... [00:05:19] I'm not worried about performance too much [00:05:24] https://www.irccloud.com/pastebin/TLt97SPk/ [00:05:30] We'll just have to be very slow to merge them [00:05:46] Oh it failed [00:06:09] Fun [00:06:25] I'm off to sleep now though, it's after midnight [00:06:31] yeah, git pull failed for some reason. [00:06:59] https://github.com/Universal-Omega/mediawiki/pull/22 worked in my testing though... [00:07:00] [url] Bump extensions/MintyDocs from `b923810` to `7672867` by dependabot[bot] · Pull Request #22 · Universal-Omega/mediawiki · GitHub | github.com [00:07:24] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 590s [00:07:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 4.12, 3.64 [00:11:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:11:44] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6703 files, 53.55MB, 2021-11-30 00:08:00 (1.1 weeks ago) [00:11:52] I know why it failed now. [00:12:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.85, 3.68 [00:12:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [00:12:21] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDfeW [00:12:23] [02miraheze/mediawiki] 07Universal-Omega 03e26af24 - use REL1_37 for workflow [00:13:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:13:53] [02mediawiki] 07Universal-Omega commented on pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDfeE [00:14:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:14:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-244d95a [+0/-0/±1] 13https://git.io/JDfew [00:14:56] [02miraheze/mediawiki] 07dependabot[bot] 03e1954a1 - Bump skins/Cosmos from `f66fddf` to `244d95a` [00:15:24] [02mediawiki] 07dependabot[bot] edited pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [00:16:01] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [00:16:11] [02mediawiki] 07Universal-Omega deleted a comment on pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDfeS [00:16:21] [02mediawiki] 07dependabot[bot] edited pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [00:18:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.59, 3.59 [00:20:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.53, 3.56 [00:21:27] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:29] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.253 second response time [00:21:35] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:37] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [00:21:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:21:51] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:55] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.68, 3.60 [00:22:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:22:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [00:22:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#4440: Bump skins/Cosmos from `f66fddf` to `244d95a` - 13https://git.io/JDvpG [00:23:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw12 [00:23:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.58, 8.46, 4.41 [00:23:26] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 0.121 second response time [00:23:36] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw8 mw12 [00:23:38] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 9.581 second response time [00:23:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.68, 3.63 [00:24:36] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:12] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [00:25:37] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 2.163 second response time [00:25:44] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 8.443 second response time [00:25:56] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.742 second response time [00:26:04] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.423 second response time [00:26:38] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 4.552 second response time [00:26:52] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw9 mw11 mw12 mw13 [00:27:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.64, 2.82, 3.31 [00:27:58] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 11.50, 17.75, 22.89 [00:28:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.88, 6.30, 7.76 [00:28:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 2.56, 4.39, 5.62 [00:28:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.36, 6.29, 7.97 [00:29:06] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw12 [00:29:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.56, 6.16, 7.73 [00:29:30] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw12 [00:29:31] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:29:32] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:30:47] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [00:31:04] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [00:31:26] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [00:31:36] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.854 second response time [00:31:38] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:31:38] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.709 second response time [00:32:39] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [00:32:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.36, 6.34, 7.56 [00:33:03] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:20] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:33:24] [02mediawiki] 07Universal-Omega closed pull request 03#4385: Update to different labeler action - 13https://git.io/JMbVv [00:33:27] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [00:33:27] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:39] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:41] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:49] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:51] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:51] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:34:10] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03The-Voidwalker-patch-1 [00:34:17] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:34:26] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03REL1_36 [00:34:27] [02mediawiki] 07Universal-Omega deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbL5b [00:34:30] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:49] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [00:34:54] [02mediawiki] 07Universal-Omega deleted branch 03REL1_36 - 13https://git.io/vbL5b [00:35:04] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 1.258 second response time [00:35:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.25, 6.64, 7.87 [00:35:25] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:35:26] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.006 second response time [00:35:37] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.005 second response time [00:35:47] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15915 bytes in 4.234 second response time [00:35:49] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.336 second response time [00:35:54] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 7.243 second response time [00:36:29] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 21670 bytes in 2.718 second response time [00:36:39] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw8 mw9 mw10 mw12 mw13 [00:36:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 7.25, 7.71 [00:37:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.54, 4.76, 6.52 [00:37:26] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.313 second response time [00:37:33] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:37:33] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 2.14, 1.49 [00:37:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.68, 7.24, 7.96 [00:37:42] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw10 mw11 mw13 [00:37:45] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.850 second response time [00:37:55] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.48, 16.56, 20.22 [00:38:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.29, 3.80, 4.85 [00:38:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.87, 7.61, 7.78 [00:38:56] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15909 bytes in 0.330 second response time [00:39:32] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.83, 1.66, 1.39 [00:40:31] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:40:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.63, 6.34, 7.76 [00:40:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.56, 7.04, 7.54 [00:41:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.35, 0.82, 1.82 [00:42:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDfJH [00:42:05] [02miraheze/mw-config] 07Universal-Omega 030db38ed - Bump `$wgMajorSiteNoticeID` [00:42:32] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.353 second response time [00:42:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 7.01, 7.83 [00:42:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDfJA [00:42:46] [02miraheze/mw-config] 07Universal-Omega 0354d77c7 - allow opt-out [00:43:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDfJh [00:43:06] [02miraheze/mw-config] 07Universal-Omega 038df5ac2 - format [00:43:09] miraheze/mw-config - Universal-Omega the build passed. [00:43:41] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 1.438 second response time [00:43:47] miraheze/mw-config - Universal-Omega the build passed. [00:43:52] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:04] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:44:09] miraheze/mw-config - Universal-Omega the build passed. [00:44:23] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:44:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.37, 7.31, 7.51 [00:45:03] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:09] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:45:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.93, 0.85, 1.60 [00:45:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.24, 5.33, 6.64 [00:45:44] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:45:55] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 7.857 second response time [00:46:04] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15891 bytes in 0.659 second response time [00:46:28] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 6.152 second response time [00:46:53] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.361 second response time [00:47:03] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 0.454 second response time [00:47:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.05, 7.61, 7.44 [00:47:45] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.684 second response time [00:48:01] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 6.543 second response time [00:48:18] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [00:48:23] Keep getting nginx 502 [00:48:23] https://cdn.discordapp.com/attachments/808001911868489748/917940635531612222/IMG_7270.png [00:48:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.46, 7.37, 7.84 [00:48:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.16, 7.09, 7.37 [00:49:11] Never before have I seen the error page being nginx [00:49:15] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 2.478 second response time [00:49:35] All the servers seen to have recovered [00:49:42] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.30, 7.88, 7.36 [00:50:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDfU7 [00:50:20] [02miraheze/mw-config] 07Universal-Omega 0318ce34e - T8396: add cargo_backlinks to Cargo SQL [00:50:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.82, 7.32, 7.08 [00:50:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.76, 8.87, 8.34 [00:50:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.95, 8.19, 7.74 [00:51:00] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:51:08] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:08] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [00:51:12] !log [universalomega@mw11] DEPLOY ABORTED: Canary check failed for localhost [00:51:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:18] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.05, 7.97, 6.92 [00:51:27] miraheze/mw-config - Universal-Omega the build passed. [00:51:31] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:46] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw10 mw12 [00:52:09] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:12] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [00:52:13] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.93, 7.67, 7.26 [00:52:58] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.311 second response time [00:53:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:02] Is anybody having any slowdowns for the moment when you enter Miraheze? [00:53:21] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True} to all [00:53:22] everything is laggy for me [00:53:31] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 0.577 second response time [00:53:33] so laggy that my wiki's favicon doesn't show up [00:53:35] mw10, 12 and 13 seems to have gone down for a bit but are back [00:53:38] Hmph, I guess I'm not the only one then. [00:53:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.68, 7.85, 7.58 [00:53:43] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/cargo.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Cargo/sql/cargo_backlinks.sql [00:53:46] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [00:53:56] !log [universalomega@mw11] DEPLOY ABORTED: Canary check failed for mw9 [00:54:10] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw8 mw9 mw10 mw11 mw12 mw13 mediawiki [00:54:10] !log [universalomega@mw11] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [00:54:13] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.559 second response time [00:54:40] !log [universalomega@mw11] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 29s [00:54:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:06] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 3.385 second response time [00:55:12] I believe fixes are still going on [00:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:15] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 3.398 second response time [00:55:16] !log [@test3] starting deploy of {'config': True} to skip [00:55:17] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:55:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.94, 7.68, 7.08 [00:55:23] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.624 second response time [00:55:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.23, 8.74, 7.93 [00:55:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:06] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [00:56:10] PROBLEM - gluster4 Disk Space on gluster4 is CRITICAL: DISK CRITICAL - free space: / 64131 MB (5% inode=80%); [00:56:24] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.164 second response time [00:56:26] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 3.144 second response time [00:57:45] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:57:49] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.29, 21.23, 19.57 [00:57:50] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:57:52] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:29] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:49] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:50] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 8.18, 6.12, 4.91 [00:59:20] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:59:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.70, 8.16, 7.42 [00:59:48] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 7.324 second response time [00:59:48] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.13, 20.42, 19.51 [00:59:54] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.409 second response time [01:00:31] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 5.241 second response time [01:00:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.66, 5.58, 4.84 [01:00:50] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 21671 bytes in 3.840 second response time [01:00:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:56] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 7.588 second response time [01:01:18] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 0.452 second response time [01:01:48] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.73, 19.98, 19.51 [01:02:11] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.687 second response time [01:02:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.07, 7.28, 7.78 [01:02:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.02, 5.03, 4.72 [01:03:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.83, 7.31, 7.27 [01:03:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.90, 7.94, 7.90 [01:04:01] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.19, 7.40, 7.81 [01:05:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.26, 8.88, 7.82 [01:05:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.25, 8.54, 8.11 [01:05:46] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.53, 21.04, 20.08 [01:05:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:06:00] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 3.705 second response time [01:06:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.93, 8.09, 7.42 [01:06:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.98, 5.97, 5.22 [01:06:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.04, 8.50, 8.17 [01:07:45] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.77, 21.89, 20.48 [01:08:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:08:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.83, 6.14, 5.36 [01:09:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [01:10:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.78, 7.85, 7.75 [01:12:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [01:12:31] @CosmicAlpha https://phabricator.miraheze.org/T8340 [01:12:32] [url] ⚓ T8340 Unable to create item on Gratisdata | phabricator.miraheze.org [01:12:48] it is getting boring for me on GD... [01:13:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.59, 7.68, 7.97 [01:13:44] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.50, 22.82, 21.41 [01:13:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:14:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:14:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.06, 5.89, 5.57 [01:15:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.43, 8.18, 8.10 [01:16:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 5.96, 6.27, 5.77 [01:18:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:18:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.47, 7.87, 7.95 [01:19:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:20:15] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:20:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [01:20:36] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.63, 8.96, 8.31 [01:21:36] Ugochimobi: no idea how to fix that. Don't know where to even begin there. [01:21:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:22:12] ah, mhen, that's serious [01:22:12] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.062 second response time [01:22:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:22:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.42, 5.76, 5.70 [01:24:01] Ugochimobi: what about creating dummy items to fill in Q1, Q3, and Q15 so that it can get back on track? Would that work? [01:24:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:25:03] that works, but I have items all through till Q1157 and up, that means I have to create over 1000 dummy items, which isn't funny at all [01:25:44] oh, I thought only those 3 were missing, why do they need to even be numbered correctly, is there some significance to the order of items? [01:25:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:28:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:29:03] I have no idea at all, because from what I have observed all through this time, when I try creating items it tells me that the item already exist, which means, the item ID (Q###) so with that, I think it has to be ordered [01:30:12] if it tells you it already exists then how were you able to create Q3, that tells me the opposite of that. [01:30:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.09, 5.12, 5.29 [01:30:50] because Q3 wasn't existing again [01:31:01] probably after deletion [01:31:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.26, 7.58, 8.00 [01:31:36] yeah, that Q3 was deleted so when I try creating an item it starts from Q3 because it is not existing [01:31:38] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.05, 21.78, 21.13 [01:32:16] ugochimbi, then what item did you try to create that it said it exists? Please try to create Q15, then try to create another one like normal. [01:32:29] ugochimobi: ^ [01:32:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.10, 5.32, 5.36 [01:33:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.69, 8.18, 8.18 [01:33:37] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.59, 21.64, 21.14 [01:33:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:33:52] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [01:34:24] We are so damn slow right now... [01:34:43] nah, you can not create items randomly on wikibase, you have to pass through Special:NewItem , and when you fill in the details you need and press `Create item` then, let's say the last item is Q1189, then this item that you just created will be Q1190 [01:34:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.37, 5.88, 5.57 [01:35:05] s/last item/last item on the wiki [01:35:05] ugochimobi meant to say: nah, you can not create items randomly on wikibase, you have to pass through Special:NewItem , and when you fill in the details you need and press `Create item` then, let's say the last item on the wiki is Q1189, then this item that you just created will be Q1190 [01:35:19] ns1 and 2 keep going down and recovering... [01:35:30] feel like that aggravates the issue a bit [01:35:36] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.34, 22.97, 21.67 [01:35:48] all the mw servers do Agent, not ns1 and 2 themselves. [01:36:06] I see, reading icinga wrong then :P [01:36:12] Agent 4 sysadmin [01:37:16] PHP-FPM usage was near 100% usage on all mw servers, I have seen that alot today... [01:37:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:37:57] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.324 second response time [01:38:16] 🥲😶‍🌫️ [01:38:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:38:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.74, 5.52, 5.52 [01:39:37] I probably have around 50-100 emails from Grafana today about low PHP-FPM workers [01:39:40] yikes [01:39:43] Ugochimobi: OK, well then I don't know, and I can't debug since it only happens on your wiki. Sorry. I am just going to go for today. It has been a very long day, and this slowness and issues with our servers recently is really starting to agitate me. [01:40:28] I will be around if emergency... just ping me. [01:40:39] other then that, I am going. [01:41:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.47, 3.22 [01:41:35] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.73, 22.84, 22.11 [01:42:08] It's alright [01:42:12] ty for staying around [01:42:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.51, 5.70, 5.57 [01:43:25] No problem, Agent. [01:43:25] Most of the issues with 1.37 should be fixed, excluding an issue with Comments (which I did upstream patch for), Preloader, and Moderation. [01:43:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.64, 3.31 [01:44:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.73, 5.55, 5.53 [01:46:45] preloader? [01:46:57] https://phabricator.miraheze.org/T8396#169727 [01:46:58] [url] ⚓ T8396 Error after upgrading miraheze to 1.37.0 | phabricator.miraheze.org [01:47:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.93, 3.51 [01:47:34] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.52, 24.10, 22.84 [01:49:16] I will likely have to fork and maintain preloader to keep it working seeing as how it is mostly unmaintained. [01:49:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.59, 23.85, 22.91 [01:50:57] @CosmicAlpha I think I'd have to use pywikibot to make dummy item creation, thank goodness the content model stuff is fixed so, it's working fine now. [01:51:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 4.51, 3.83 [01:51:28] If you check famedata RC feed you'd find out that the bot has started creating items for famepedia pages [01:51:34] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 23.06, 24.26, 23.22 [01:53:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:54:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:54:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.71, 6.46, 5.86 [01:56:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.49, 5.89, 5.72 [01:57:24] https://github.com/Universal-Omega/Preloader/pull/1 so now just need to switch to my fork for that one. [01:57:24] [url] Replace usage of Revision class by Universal-Omega · Pull Request #1 · Universal-Omega/Preloader · GitHub | github.com [01:57:32] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.29, 23.83, 23.56 [01:58:50] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 8.40, 6.33, 5.86 [01:59:31] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.68, 24.60, 23.87 [02:00:13] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:00:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:02:11] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.116 second response time [02:02:17] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.963 second response time [02:03:27] PROBLEM - ping4 on jobchron1 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 376.54 ms [02:03:30] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.31, 23.88, 23.72 [02:04:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:04:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.05, 5.88, 5.88 [02:05:27] RECOVERY - ping4 on jobchron1 is OK: PING OK - Packet loss = 0%, RTA = 7.17 ms [02:05:30] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.39, 23.41, 23.51 [02:06:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.32, 6.17, 5.99 [02:07:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.99, 23.07, 23.39 [02:07:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:07:57] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/moderation.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/create_moderation_block_if_not_exists.sql [02:08:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.47, 5.47, 5.76 [02:10:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.73, 6.18, 5.98 [02:11:30] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.79, 24.49, 23.84 [02:15:30] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.04, 23.34, 23.58 [02:16:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.49, 5.35, 5.79 [02:17:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:18:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 10.22, 7.30, 6.46 [02:19:29] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 37.62, 26.56, 24.56 [02:20:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:20:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [02:23:49] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±2] 13https://git.io/JDfna [02:23:50] [02miraheze/mediawiki] 07Universal-Omega 0353557ee - Switch Preloader repository [02:24:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:25:52] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:27:34] !log [universalomega@mw11] starting deploy of {'world': True} to all [02:27:47] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:16] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:28:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:28:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:28:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:29:45] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 21673 bytes in 0.355 second response time [02:30:13] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.314 second response time [02:30:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:30:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:30:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:31:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.55, 4.00 [02:33:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.00, 3.99, 4.11 [02:36:50] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 554s [02:36:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:51] !log [universalomega@mw11] starting deploy of {'folders': 'w/extensions/Preloader'} to all [02:39:05] !log [universalomega@mw11] finished deploy of {'folders': 'w/extensions/Preloader'} to all - SUCCESS in 13s [02:39:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:45] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp20.miraheze.org - CNAME OK [02:40:26] Now all issues mentioned should be fixed! [02:40:40] Now I am really going for the night. [02:41:42] night [02:41:53] thanks! [02:42:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 2.52, 4.83, 5.94 [02:42:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.59, 6.25, 7.82 [02:44:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.17, 8.40, 8.43 [02:45:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:46:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.15, 5.28, 5.83 [02:48:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.89, 5.54, 5.86 [02:50:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:52:36] [02mw-config] 07Universal-Omega closed pull request 03#4268: site venue - 13https://git.io/JDvOV [02:52:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDf8J [02:52:39] [02miraheze/mw-config] 07Naleksuh 030918c8a - site venue (#4268) [02:53:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.58, 3.96 [02:53:43] miraheze/mw-config - Universal-Omega the build passed. [02:54:03] !log [@test3] starting deploy of {'config': True} to skip [02:54:04] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:54:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.95, 5.27, 5.56 [02:54:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:56:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.91, 5.06, 5.44 [02:57:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.58, 3.86 [02:58:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.61, 5.56, 5.58 [03:00:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.14, 5.08, 5.41 [03:01:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:02:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.45, 6.99, 7.98 [03:03:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.59, 3.82 [03:04:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.07, 1.83, 1.21 [03:04:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 8.03, 5.79, 5.58 [03:06:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.41, 1.32, 1.09 [03:06:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.06, 22.77, 23.97 [03:08:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.03, 8.67, 8.31 [03:08:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 23.33, 23.92, 24.32 [03:11:22] !log [@mw11] starting deploy of {'config': True} to all [03:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:29] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 6s [03:11:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.15, 5.17, 5.52 [03:13:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 16.75, 21.85, 23.49 [03:14:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.72, 6.46, 7.55 [03:15:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.74, 3.64 [03:16:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.45, 7.12, 7.95 [03:16:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.01, 7.60, 7.83 [03:16:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.70, 5.57, 5.54 [03:16:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.51, 22.08, 23.70 [03:18:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.68, 7.77, 8.08 [03:18:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.82, 7.37, 7.72 [03:18:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.69, 5.24, 5.42 [03:19:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.83, 3.72 [03:20:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.89, 6.89, 7.71 [03:20:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.08, 25.53, 24.66 [03:21:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 4.08, 3.82 [03:21:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.47, 6.44, 7.68 [03:22:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.04, 6.83, 7.88 [03:23:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.33, 7.12, 8.00 [03:23:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.63, 6.94, 7.69 [03:24:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.14, 7.53, 8.02 [03:24:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.53, 6.53, 7.13 [03:24:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.02, 5.61, 5.46 [03:25:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.89, 3.81 [03:25:29] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.83, 23.25, 23.30 [03:26:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.87, 6.12, 6.91 [03:26:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.37, 5.27, 5.37 [03:27:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.11, 21.35, 22.59 [03:28:36] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.67, 7.23, 7.31 [03:28:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.18, 6.59, 6.96 [03:29:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.99, 3.87 [03:29:19] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.50, 8.23, 8.09 [03:29:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 7.54, 7.79 [03:30:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.31, 7.50, 7.96 [03:30:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.97, 7.57, 7.44 [03:30:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.93, 6.71, 6.97 [03:31:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.93, 3.88 [03:32:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.45, 7.79, 7.52 [03:32:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.52, 5.93, 5.53 [03:32:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.99, 7.62, 7.28 [03:34:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.46, 7.96, 8.03 [03:36:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.49, 7.69, 7.95 [03:36:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.44, 7.02, 7.25 [03:36:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:29] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.17, 23.10, 22.54 [03:38:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.59, 7.75, 7.43 [03:38:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.05, 5.86, 5.68 [03:39:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.08, 7.58, 7.97 [03:39:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.41, 22.60, 22.39 [03:41:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 7.58, 7.92, 8.07 [03:41:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:43:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.68, 3.76 [03:43:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.09, 6.77, 7.62 [03:44:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.40, 6.06, 6.76 [03:44:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.85, 4.17, 5.05 [03:45:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.51, 3.70 [03:45:16] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.55, 7.41, 7.75 [03:46:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.43, 6.85, 6.93 [03:47:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.07, 7.37, 7.70 [03:49:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.12, 3.87 [03:49:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.81, 6.76, 6.97 [03:50:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.49, 8.20, 7.42 [03:51:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 6.75, 6.94 [03:52:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.66, 6.25, 6.78 [03:52:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.90, 7.43, 7.24 [03:52:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 7.83, 7.46 [03:53:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.12, 6.13, 6.69 [03:53:29] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.54, 18.35, 20.00 [03:54:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.77, 6.07, 6.76 [03:55:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.93, 3.91 [03:55:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.51, 6.72, 7.81 [03:56:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 2.96, 5.33, 6.52 [03:56:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 12.18, 18.30, 22.81 [03:57:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.37, 3.91, 5.77 [03:59:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.78, 5.74, 6.76 [04:01:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.46, 7.01, 7.50 [04:03:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.21, 2.85, 3.38 [04:03:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.65, 6.71, 7.32 [04:05:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.06, 7.61, 7.54 [04:05:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.24, 5.72, 5.85 [04:06:14] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.98, 6.61, 6.28 [04:06:59] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.75, 4.13, 4.04 [04:07:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.75, 3.61 [04:07:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.86, 6.93, 6.76 [04:07:30] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.87, 20.89, 19.71 [04:07:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 8.00, 7.93, 7.69 [04:07:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.81, 5.42, 5.71 [04:08:11] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.25, 7.25, 6.55 [04:08:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.17, 17.59, 20.25 [04:09:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.60, 3.57 [04:09:16] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.36, 6.31, 6.55 [04:09:29] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.85, 19.97, 19.51 [04:10:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.45, 6.79, 6.46 [04:10:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.83, 6.60, 6.26 [04:10:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.14, 5.32, 4.54 [04:12:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.16, 6.44, 6.27 [04:13:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.12, 21.84, 20.49 [04:13:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.64, 8.09, 7.75 [04:13:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.47, 7.49, 6.32 [04:14:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.29, 7.05, 6.68 [04:14:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.09, 5.20, 4.70 [04:14:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.02, 22.01, 21.35 [04:15:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.05, 3.32 [04:15:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 6.40, 6.10 [04:15:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 7.63, 7.61 [04:16:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.54, 4.99, 4.70 [04:17:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.87, 6.04, 6.00 [04:17:29] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.87, 19.51, 19.90 [04:17:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.73, 5.93, 5.95 [04:18:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.64, 25.15, 22.50 [04:19:16] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.50, 8.56, 7.23 [04:19:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.30, 6.89, 6.32 [04:20:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.56, 22.61, 21.92 [04:21:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.17, 6.87, 6.40 [04:21:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.40, 7.52, 7.01 [04:21:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.79, 5.42, 5.85 [04:21:51] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.91, 6.49, 6.61 [04:22:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.38, 3.40 [04:23:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.05, 5.88, 6.09 [04:24:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.00, 3.25 [04:25:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.36, 6.27, 6.60 [04:26:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.45, 18.74, 20.22 [04:27:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.91, 3.99, 5.10 [04:43:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.86, 8.30, 7.41 [04:43:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.69, 3.37 [04:45:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.45, 7.63, 6.73 [04:45:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.54, 3.36 [04:46:20] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.13, 22.60, 20.02 [04:47:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.64, 6.52, 6.43 [04:47:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.12, 7.82, 7.43 [04:47:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.12, 3.24, 3.26 [04:51:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 4.03, 3.59 [04:52:18] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.91, 18.40, 19.04 [04:53:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.34, 7.96, 7.48 [04:55:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.05, 7.50, 7.38 [04:55:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.04, 3.53, 3.53 [04:55:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.25, 4.73, 4.56 [04:57:49] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.64, 4.62, 4.54 [04:57:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDfyZ [04:57:55] [02miraheze/mediawiki] 07Universal-Omega 03d7f94b5 - Remove whitespace [04:59:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.95, 3.70 [05:01:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.11, 5.98, 6.74 [05:01:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.47, 3.55 [05:03:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JDfSX [05:03:35] [02miraheze/mw-config] 07Universal-Omega 03c9e7f51 - Update ManageWikiExtensions.php [05:03:37] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [05:03:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.69, 2.95, 3.36 [05:03:44] [02mw-config] 07Universal-Omega opened pull request 03#4272: Remove NumberedHeadings - 13https://git.io/JDfSD [05:04:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JDfSj [05:04:09] [02miraheze/mw-config] 07Universal-Omega 033a0eda7 - Update extension-list [05:04:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4272: Remove NumberedHeadings - 13https://git.io/JDfSD [05:04:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JDf9I [05:04:39] [02miraheze/mw-config] 07Universal-Omega 0323189d6 - Update LocalSettings.php [05:04:41] [02mw-config] 07Universal-Omega synchronize pull request 03#4272: Remove NumberedHeadings - 13https://git.io/JDfSD [05:04:48] miraheze/mw-config - Universal-Omega the build passed. [05:05:11] miraheze/mw-config - Universal-Omega the build passed. [05:05:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.24, 7.10, 7.06 [05:05:41] miraheze/mw-config - Universal-Omega the build passed. [05:07:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.82, 20.50, 19.88 [05:08:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.20, 7.24, 6.57 [05:09:37] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.86, 19.18, 19.48 [05:13:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.43, 6.38, 6.79 [05:13:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.14, 6.71, 6.61 [05:16:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 7.40, 6.23 [05:17:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.64, 6.89, 6.73 [05:18:38] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.56, 8.79, 6.85 [05:19:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 38.64, 25.28, 21.31 [05:19:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.01, 6.45, 6.61 [05:20:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 7.80, 6.37 [05:20:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.17, 5.26, 4.74 [05:21:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.19, 22.45, 20.74 [05:21:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.33, 4.10, 3.59 [05:22:04] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.33, 6.54, 6.07 [05:22:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.08, 7.88, 6.96 [05:22:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 5.00, 5.01, 4.69 [05:22:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.07, 20.40, 20.18 [05:23:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.65, 3.48 [05:24:28] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.63, 8.09, 6.72 [05:24:29] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.36, 7.11, 6.33 [05:24:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.33, 8.20, 7.15 [05:25:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.35, 8.46, 7.28 [05:25:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.91, 8.72, 7.59 [05:25:55] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.95, 8.21, 6.90 [05:26:07] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.87, 21.21, 19.41 [05:26:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.18, 7.09, 6.44 [05:26:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.73, 7.34, 6.94 [05:27:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 3.16, 3.33 [05:27:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.18, 7.53, 7.08 [05:27:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.52, 7.19, 6.68 [05:28:06] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.06, 19.38, 18.96 [05:28:24] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.22, 6.24, 6.19 [05:28:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.61, 7.43, 6.77 [05:28:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.35, 6.60, 6.72 [05:29:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.65, 7.71, 7.45 [05:29:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.51, 6.16, 6.36 [05:31:25] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.86, 5.84, 6.49 [05:32:22] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.31, 6.22, 6.47 [05:33:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.32, 7.62, 7.43 [05:35:15] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.47, 7.84, 7.12 [05:35:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.88, 3.55 [05:36:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.81, 20.52, 19.57 [05:36:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.83, 7.40, 6.65 [05:36:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.89, 7.59, 6.95 [05:37:12] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.57, 7.37, 7.03 [05:37:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.63, 3.50 [05:37:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.06, 7.55, 7.52 [05:38:02] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.46, 20.14, 19.54 [05:38:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.25, 6.51, 6.41 [05:38:11] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.75, 5.27, 4.25 [05:38:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.91, 6.96, 6.78 [05:39:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.30, 3.40 [05:40:06] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.08, 4.73, 4.18 [05:41:06] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.45, 8.13, 7.32 [05:41:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.26, 8.12, 7.70 [05:42:00] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.52, 22.45, 20.56 [05:42:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.23, 23.00, 20.70 [05:42:13] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.68, 8.19, 7.24 [05:43:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.46, 7.28, 6.58 [05:43:56] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.66, 5.39, 4.60 [05:43:59] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.92, 7.54, 6.90 [05:44:00] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 16.61, 20.50, 20.11 [05:44:11] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.11, 7.83, 7.24 [05:44:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.43, 7.66, 6.99 [05:44:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.39, 6.27, 5.33 [05:45:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.49, 7.03, 6.55 [05:45:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.73, 3.56 [05:45:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.99, 7.73, 7.67 [05:45:51] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.40, 4.51, 4.39 [05:45:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.25, 23.48, 21.41 [05:45:59] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.12, 19.57, 19.85 [05:46:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.64, 6.35, 6.59 [05:46:43] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.96, 5.02, 4.99 [05:46:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.94, 6.87, 7.10 [05:46:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.00, 6.23, 6.31 [05:47:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 3.16, 3.37 [05:47:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.76, 5.88, 6.40 [05:48:08] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.86, 6.01, 6.62 [05:50:50] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.35, 5.93, 6.65 [05:51:38] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.31, 22.39, 21.37 [05:51:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.68, 5.84, 6.77 [05:53:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.21, 21.21, 21.07 [05:57:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.10, 19.25, 20.38 [06:14:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.12, 9.57, 7.54 [06:14:45] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.54, 7.85, 6.27 [06:14:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.51, 22.79, 19.94 [06:15:06] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.17, 6.80, 5.98 [06:15:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.66, 8.91, 7.08 [06:15:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 15.37, 10.44, 7.22 [06:15:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.29, 6.82, 5.94 [06:15:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.23, 5.59, 4.82 [06:16:43] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.63, 6.94, 6.16 [06:16:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.23, 19.92, 19.23 [06:17:03] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.50, 6.22, 5.86 [06:17:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.88, 6.51, 5.93 [06:17:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.67, 8.00, 6.71 [06:17:49] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.40, 5.07, 4.72 [06:18:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.37, 3.23, 1.79 [06:18:42] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.25, 6.39, 6.05 [06:21:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.16, 7.84, 7.25 [06:21:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.18, 6.15, 6.25 [06:22:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.03, 2.00, 1.60 [06:22:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.77, 6.96, 6.37 [06:24:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.47, 1.48, 1.46 [06:24:20] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.83, 7.53, 7.41 [06:26:18] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.16, 8.36, 7.73 [06:27:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 6.12, 5.85 [06:28:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.81, 7.68, 7.55 [06:28:42] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.70, 7.39, 6.68 [06:29:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.88, 6.50, 6.79 [06:29:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.62, 5.80, 5.78 [06:30:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.81, 6.85, 6.58 [06:32:35] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.71, 5.68, 6.17 [06:33:49] PROBLEM - db13 APT on db13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:35:23] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 9.84, 7.37, 4.75 [06:35:44] RECOVERY - db13 APT on db13 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [06:36:07] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.97, 6.00, 6.73 [06:37:22] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 2.32, 5.27, 4.29 [06:37:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.17, 4.44, 4.25 [06:39:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.26, 4.41, 4.26 [06:47:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.79, 5.18, 4.62 [06:49:50] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.37, 4.56, 4.47 [06:55:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.70, 20.69, 19.31 [06:57:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.70, 18.62, 18.70 [06:59:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.59, 6.47, 5.29 [07:01:15] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.59, 24.53, 21.08 [07:03:54] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.64, 7.40, 6.16 [07:05:01] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/moderation.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Moderation/sql/patch-moderation-mod_tags.sql [07:05:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:05:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.73, 6.93, 6.16 [07:07:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.91, 5.97, 5.91 [07:08:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.86, 22.98, 22.08 [07:09:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.56, 5.72, 5.88 [07:12:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.44, 23.08, 22.26 [07:14:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.93, 20.17, 21.31 [07:19:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.54, 3.22 [07:19:49] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.07, 4.38, 5.06 [07:21:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.33, 3.19 [07:22:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.96, 18.32, 20.02 [07:23:56] !log reception@mwtask1:/home/universalomega$ sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/moderation.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Moderation/sql/patch-moderation-mod_type.sql [07:24:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:27:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.78, 7.29, 6.02 [07:27:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.72, 5.86, 5.31 [07:29:09] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.08, 7.60, 6.74 [07:29:25] !log reception@mwtask1:/home/universalomega$ sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/moderation.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Moderation/sql/patch-make-preload-unique.sql [07:29:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 3.76, 3.35 [07:29:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.23, 6.14 [07:29:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:30:11] !log deleted MWExt-SkinUpdateTracker GH repo [07:30:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:30:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.93, 6.46, 5.84 [07:30:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.67, 22.20, 21.09 [07:31:07] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.17, 7.62, 6.86 [07:31:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.46, 3.29 [07:31:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.05, 6.40, 5.97 [07:31:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.93, 5.93, 5.57 [07:32:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.67, 5.67, 5.62 [07:33:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.66, 3.22, 3.22 [07:33:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.06, 6.36, 5.77 [07:34:21] !log reception@mwtask1:/home/universalomega$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Moderation/sql/patch-make-preload-unique.sql --wiki allthetropeswiki [07:34:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:35:03] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.64, 6.76, 6.68 [07:37:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.11, 5.45, 5.60 [07:39:01] !log reception@mwtask1:/home/universalomega$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php /home/universalomega/patch-make-preload-unique.sql --wiki allthetropeswiki (altered query) [07:39:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:40:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.37, 17.51, 19.60 [07:42:33] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.37, 6.31, 5.94 [07:43:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.29, 6.62, 5.90 [07:44:32] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.04, 6.48, 6.06 [07:45:03] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49215 MB (11% inode=98%); [07:45:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.68, 5.38, 5.53 [07:47:49] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.21, 4.23, 5.09 [07:49:03] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48496 MB (10% inode=98%); [08:11:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.37, 7.39, 6.02 [08:13:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.03, 6.96, 5.52 [08:13:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.09, 6.41, 5.84 [08:15:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.08, 5.71, 5.23 [08:29:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.00, 18.26, 17.21 [08:31:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.98, 19.14, 17.70 [08:57:33] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.31, 6.90, 5.62 [08:59:32] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.72, 5.71, 5.33 [09:01:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.17, 6.25, 4.44 [09:02:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 10.08, 6.04, 3.61 [09:02:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.10, 21.37, 17.50 [09:03:22] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.27, 6.85, 5.03 [09:03:27] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 15.30, 9.91, 7.03 [09:03:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.60, 8.80, 6.05 [09:03:41] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.30, 6.35, 5.04 [09:04:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.50, 20.13, 15.88 [09:04:24] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.04, 7.58, 5.87 [09:04:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.14, 21.25, 18.00 [09:05:19] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.84, 6.14, 5.00 [09:05:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.48, 7.05, 5.74 [09:05:38] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.10, 5.66, 4.93 [09:05:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.39, 5.71, 4.74 [09:06:02] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.35, 18.04, 15.60 [09:06:21] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.90, 6.70, 5.75 [09:06:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 3.07, 5.21, 3.92 [09:06:56] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.38, 18.31, 17.31 [09:07:24] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.14, 6.60, 6.36 [09:07:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 2.59, 5.59, 5.37 [09:07:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.51, 4.71, 4.48 [09:08:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.66, 4.27, 3.73 [09:47:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.74, 19.22, 17.02 [09:48:38] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.22, 5.60, 4.45 [09:48:56] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.89, 21.69, 18.11 [09:50:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.68, 5.46, 4.53 [09:50:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.18, 19.23, 17.64 [09:52:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.15, 5.79, 4.76 [09:54:21] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.52, 4.85, 4.53 [10:01:48] .op [10:01:48] Attempting to OP... [10:01:57] .deop [10:01:57] Attempting to OP... [10:02:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.83, 23.57, 20.12 [10:02:56] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.80, 7.02, 5.56 [10:04:50] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.24, 5.90, 5.33 [10:05:43] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.75, 7.15, 5.58 [10:06:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.42, 22.46, 20.51 [10:07:42] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 7.05, 5.73 [10:09:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.98, 6.21, 5.58 [10:10:34] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.39, 4.91, 5.07 [10:10:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.25, 19.89, 19.90 [10:53:03] PROBLEM - db12 Disk Space on db12 is CRITICAL: DISK CRITICAL - free space: / 26689 MB (5% inode=98%); [10:53:59] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.12, 4.25, 3.96 [10:55:53] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.95, 3.92, 3.86 [11:05:03] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 41304 MB (9% inode=98%); [11:16:10] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.75, 5.32, 4.46 [11:16:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.95, 5.94, 5.03 [11:18:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.60, 5.85, 5.09 [11:23:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.85, 4.72, 4.67 [11:27:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.11, 5.15, 4.85 [11:28:08] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.15, 7.22, 5.98 [11:32:07] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.50, 6.41, 5.94 [11:33:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.79, 21.70, 19.67 [11:33:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.91, 4.61, 4.78 [11:35:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.29, 19.95, 19.24 [11:51:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.26, 6.18, 5.69 [11:54:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.64, 24.05, 20.40 [11:55:47] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.06, 5.54, 5.58 [11:55:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.40, 5.31, 4.62 [11:56:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.99, 21.12, 19.75 [11:57:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.22, 4.64, 4.47 [11:58:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.91, 18.38, 18.92 [12:18:12] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 7.48, 8.04, 6.86 [12:22:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.60, 3.06 [12:24:05] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.94, 7.75, 7.16 [12:24:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.21, 2.99 [12:26:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.02, 8.03, 7.32 [12:28:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.98, 7.01, 7.04 [12:32:31] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [12:35:52] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.22, 6.34, 6.74 [12:39:27] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:44:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.44, 7.72, 6.39 [12:44:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.57, 20.73, 18.71 [12:44:40] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.97, 7.46, 6.05 [12:45:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.77, 7.00, 6.05 [12:45:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.94, 7.19, 6.86 [12:46:15] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.53, 19.17, 18.37 [12:47:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.26, 7.08, 6.19 [12:47:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.92, 7.99, 7.21 [12:48:02] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.80, 7.27, 6.56 [12:48:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.72, 7.44, 6.14 [12:50:01] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.36, 8.97, 7.26 [12:51:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 16.55, 10.72, 7.30 [12:51:29] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.93, 6.33, 5.10 [12:51:33] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.60, 2.49, 1.50 [12:51:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.76, 24.77, 21.11 [12:52:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.73, 7.98, 6.88 [12:53:33] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.09, 1.86, 1.39 [12:53:48] PROBLEM - mem2 Puppet on mem2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[syslog-ng] [12:53:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.17, 23.92, 21.24 [12:54:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.37, 8.50, 7.20 [12:54:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.20, 7.88, 6.83 [12:55:46] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.93, 25.98, 22.31 [12:57:53] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.97, 20.82, 19.29 [13:02:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:04:10] PROBLEM - gluster4 Disk Space on gluster4 is WARNING: DISK WARNING - free space: / 64323 MB (6% inode=81%); [13:07:38] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:09:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:09:49] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.21, 20.08, 20.12 [13:13:47] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.41, 21.50, 20.66 [13:14:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:14:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.33, 23.10, 23.88 [13:15:47] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.32, 18.78, 19.77 [13:16:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.53, 5.48, 5.88 [13:18:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.23, 6.63, 7.96 [13:20:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.78, 5.85, 5.84 [13:20:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.09, 7.41, 8.07 [13:20:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.28, 24.47, 23.74 [13:21:48] RECOVERY - mem2 Puppet on mem2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [13:22:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:24:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:24:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.95, 23.69, 23.74 [13:26:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.19, 6.97, 7.85 [13:26:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.92, 23.57, 23.64 [13:27:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:27:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.39, 5.69, 5.96 [13:28:09] PROBLEM - gluster4 Disk Space on gluster4 is CRITICAL: DISK CRITICAL - free space: / 64145 MB (5% inode=81%); [13:28:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.38, 22.76, 23.34 [13:29:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.17, 5.78, 5.95 [13:30:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.43, 6.97, 7.90 [13:31:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.39, 5.31, 5.76 [13:33:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.53, 6.82, 7.93 [13:34:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.58, 8.06, 8.02 [13:36:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.56, 7.31, 7.75 [13:36:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.91, 7.04, 7.99 [13:36:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.27, 21.97, 22.34 [13:38:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.95, 20.99, 21.94 [13:39:24] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.05, 7.03, 7.89 [13:39:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.54, 7.26, 7.96 [13:40:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.58, 6.93, 7.40 [13:40:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.93, 22.91, 22.53 [13:42:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.04, 7.00, 7.18 [13:42:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.94, 7.38, 7.71 [13:43:21] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.65, 9.63, 8.64 [13:43:25] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.13, 8.52, 7.88 [13:43:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.66, 7.87, 8.02 [13:43:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.94, 6.03, 5.70 [13:44:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.40, 6.36, 6.92 [13:44:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.36, 7.36, 7.51 [13:44:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.81, 6.77, 7.45 [13:44:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.17, 23.32, 22.89 [13:45:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.40, 7.49, 7.57 [13:46:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.24, 6.93, 7.05 [13:47:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.39, 7.37, 7.81 [13:47:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.67, 5.80, 5.71 [13:48:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.11, 6.91, 7.02 [13:49:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.39, 6.92, 7.75 [13:50:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.66, 7.09, 7.22 [13:50:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.38, 8.39, 7.78 [13:50:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.11, 23.86, 23.14 [13:51:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.25, 7.93, 8.01 [13:51:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.32, 7.06, 7.23 [13:51:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.88, 8.22, 7.96 [13:51:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.99, 6.40, 5.93 [13:52:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.13, 7.83, 7.28 [13:52:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.51, 7.74, 7.62 [13:52:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.41, 22.70, 22.86 [13:53:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.13, 6.85, 7.15 [13:53:50] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.55, 5.83, 5.77 [13:54:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.22, 7.36, 7.19 [13:54:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.26, 7.16, 7.27 [13:55:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.31, 7.41, 7.81 [13:55:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.99, 7.26, 7.23 [13:55:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.15, 6.35, 5.95 [13:57:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.82, 6.35, 6.90 [13:58:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.18, 9.09, 7.99 [13:58:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.93, 8.43, 7.85 [13:58:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.35, 23.95, 23.34 [13:59:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 15.41, 9.63, 8.50 [14:00:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 13.21, 8.65, 7.61 [14:01:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.79, 8.02, 7.42 [14:03:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.67, 7.41, 7.27 [14:07:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.70, 8.05, 7.49 [14:08:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.81, 23.71, 23.82 [14:09:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.77, 7.43, 7.33 [14:10:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.17, 7.73, 7.68 [14:10:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.21, 23.99, 23.83 [14:11:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.20, 7.32, 7.27 [14:12:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.35, 8.46, 7.95 [14:16:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.66, 3.33 [14:17:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.16, 7.62, 7.61 [14:20:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:20:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.08, 7.55, 7.88 [14:22:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.79, 3.49 [14:22:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.20, 8.61, 8.22 [14:23:48] PROBLEM - mem2 Puppet on mem2 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Service[syslog-ng],Service[ferm],Service[prometheus-memcached-exporter],Service[memcached] [14:24:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.38, 3.38 [14:25:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:28:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:28:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.82, 3.57 [14:29:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.31, 7.71, 7.63 [14:32:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.51, 3.56, 3.55 [14:33:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:34:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:34:25] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.76, 20.43, 18.82 [14:36:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.05, 3.83, 3.62 [14:36:25] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 41.06, 25.89, 20.89 [14:38:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.54, 3.55 [14:38:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.25, 23.40, 20.59 [14:39:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:44:22] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.60, 19.24, 19.70 [14:48:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.22, 3.10, 3.38 [14:52:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.64, 3.55 [15:00:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.05, 3.33 [15:03:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.44, 7.22, 8.00 [15:04:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.97, 6.69, 7.76 [15:08:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.38, 6.50, 7.83 [15:11:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.79, 8.25, 7.83 [15:11:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.81, 7.44, 7.97 [15:12:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.14, 3.16 [15:13:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.50, 7.36, 7.58 [15:14:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 2.91, 3.07 [15:14:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.58, 6.30, 7.70 [15:15:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.05, 6.02, 7.79 [15:16:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.18, 6.61, 7.37 [15:16:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.04, 7.50, 7.92 [15:17:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.13, 6.64, 7.77 [15:18:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.32, 6.71, 7.57 [15:19:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.57, 7.54, 7.43 [15:19:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.96, 8.16, 7.86 [15:20:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.51, 7.46, 7.33 [15:20:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.38, 7.51, 7.77 [15:21:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.40, 7.06, 7.27 [15:22:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.79, 7.27, 7.63 [15:23:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.67, 3.35 [15:24:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.97, 8.54, 8.02 [15:25:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.71, 9.36, 8.11 [15:25:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:25:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.58, 3.34 [15:27:57] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.21, 3.23 [15:30:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:31:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.10, 7.79, 7.91 [15:35:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 13.64, 9.42, 8.41 [15:35:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.41, 3.31 [15:43:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.49, 7.56, 7.93 [15:43:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.58, 3.41 [15:44:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:45:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.54, 8.75, 8.38 [15:45:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.98, 3.39, 3.37 [15:49:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:49:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.41, 3.45, 3.45 [15:53:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.56, 3.47 [15:54:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:55:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.27, 3.38 [15:59:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:02:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.54, 4.33, 3.77 [16:04:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.96, 3.69 [16:08:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:13:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:14:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.56, 3.58 [16:17:50] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.16, 20.87, 19.77 [16:18:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.59, 3.62 [16:19:49] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 36.73, 25.46, 21.46 [16:21:25] PROBLEM - mem2 Current Load on mem2 is CRITICAL: connect to address 51.195.236.245 port 5666: Connection refusedconnect to host 51.195.236.245 port 5666: Connection refused [16:21:26] PROBLEM - mem2 APT on mem2 is CRITICAL: connect to address 51.195.236.245 port 5666: Connection refusedconnect to host 51.195.236.245 port 5666: Connection refused [16:21:37] PROBLEM - mem2 ferm_active on mem2 is CRITICAL: connect to address 51.195.236.245 port 5666: Connection refusedconnect to host 51.195.236.245 port 5666: Connection refused [16:21:47] PROBLEM - mem2 conntrack_table_size on mem2 is CRITICAL: connect to address 51.195.236.245 port 5666: Connection refusedconnect to host 51.195.236.245 port 5666: Connection refused [16:21:49] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.89, 23.62, 21.25 [16:21:55] PROBLEM - mem2 PowerDNS Recursor on mem2 is CRITICAL: connect to address 51.195.236.245 port 5666: Connection refusedconnect to host 51.195.236.245 port 5666: Connection refused [16:22:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:22:12] PROBLEM - mem2 NTP time on mem2 is CRITICAL: connect to address 51.195.236.245 port 5666: Connection refusedconnect to host 51.195.236.245 port 5666: Connection refused [16:22:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.236 second response time [16:22:19] PROBLEM - mem2 Disk Space on mem2 is CRITICAL: connect to address 51.195.236.245 port 5666: Connection refusedconnect to host 51.195.236.245 port 5666: Connection refused [16:22:58] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw9 [16:23:25] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.031 second response time [16:23:26] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.012 second response time [16:23:39] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.010 second response time [16:23:47] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.238 second response time [16:23:57] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw9 [16:24:29] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw9 [16:24:38] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw9 [16:24:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 1.19, 4.69, 7.91 [16:25:47] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.76, 23.72, 21.68 [16:27:47] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.66, 21.86, 21.26 [16:28:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.44, 3.50 [16:28:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 0.51, 2.39, 6.22 [16:32:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.09, 19.42, 23.32 [16:33:45] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 29.73, 25.59, 22.89 [16:33:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.69, 4.96, 5.86 [16:35:38] Reception123: around [16:35:44] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.10, 23.10, 22.32 [16:35:46] RhinosF1: yeah [16:36:05] Reception123: do you wanna work on mw8 [16:36:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.62, 3.50, 3.58 [16:36:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.36, 6.23, 7.87 [16:36:39] RhinosF1: hm? [16:36:48] Reception123: finding out why it's a pain [16:36:54] i need someone with root [16:37:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:37:13] ok, feel free to ping me for anything that needs root [16:39:22] Reception123: can you please type 'gdb -p 2121132' [16:39:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.82, 5.21, 5.61 [16:40:33] https://www.irccloud.com/pastebin/JSnbfbNT/ [16:40:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.88, 16.33, 20.23 [16:41:10] Reception123: do you get a prompt at the bottom [16:41:20] yeah, a CLI [16:41:25] Reception123: type run [16:41:44] https://www.irccloud.com/pastebin/YZe9nzin/ [16:42:31] Reception123: depool mw8 [16:42:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.76, 8.09, 7.98 [16:43:01] RhinosF1: what are you planning to do? [16:43:15] Reception123: drain php-fpm first so it restarts quicker [16:43:45] ok, will do [16:44:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:44:48] !log depool mw8 [16:44:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [16:44:49] RhinosF1: ^ [16:44:57] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.76, 22.52, 21.85 [16:45:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:45:14] !log restart php7.4-fpm on mw8 to drain it [16:45:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:33] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.53, 2.67, 3.23 [16:46:22] Reception123: now type 'gdb -p 2159618' [16:46:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.67, 7.69, 8.00 [16:46:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.79, 21.04, 21.45 [16:47:01] same thing basically [16:47:04] https://www.irccloud.com/pastebin/ZQ6TmiCK/ [16:47:05] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:47:10] Reception123: with a different number [16:47:19] then the run again at the prompt [16:47:22] hm? [16:47:33] Reception123: the number after -p is different [16:47:38] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21700 bytes in 5.665 second response time [16:47:38] when prompted press run [16:47:42] I did [16:47:48] urgh [16:47:54] ok change of plan [16:49:14] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.945 second response time [16:50:09] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.233 second response time [16:50:15] PROBLEM - mw8 php-fpm on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4' [16:50:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 0.85, 3.86, 6.35 [16:50:36] Reception123: try 'sudo gdb /usr/sbin/php-fpm7.4' [16:50:39] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.360 second response time [16:50:42] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.004 second response time [16:50:44] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.260 second response time [16:50:46] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.004 second response time [16:50:54] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.005 second response time [16:50:55] then 'run --nodaemonize --fpm-config /etc/php/7.4/fpm/php-fpm.conf' [16:51:02] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:10] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:13] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [16:51:14] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:14] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:35] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 3.869 second response time [16:51:58] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:05] PROBLEM - mem2 memcached on mem2 is CRITICAL: connect to address 51.195.236.245 and port 11211: Connection refused [16:52:40] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:43] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:47] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:54] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.97, 17.62, 20.00 [16:53:01] RhinosF1: doing first now [16:53:11] https://www.irccloud.com/pastebin/CH1hoKBY/ [16:53:33] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:53:38] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 12.96, 17.87, 19.98 [16:53:42] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.010 second response time [16:53:58] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 0.223 second response time [16:53:59] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:05] Reception123: okay, now without touching that console run 'sudo /usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74' [16:54:06] also I don't like what's going on with icinga warnings right now [16:54:09] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15899 bytes in 0.332 second response time [16:54:13] RECOVERY - mw8 php-fpm on mw8 is OK: PROCS OK: 33 processes with command name 'php-fpm7.4' [16:54:25] Reception123: well in a second you can repool mw8 and hopefully it'll calm down [16:54:30] Reception123: they've been like that for days now [16:54:36] RhinosF1: done, no output [16:54:41] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.316 second response time [16:54:42] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15899 bytes in 0.013 second response time [16:54:43] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15899 bytes in 0.322 second response time [16:54:44] Reception123: ok that's fine [16:54:46] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.012 second response time [16:54:46] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15891 bytes in 0.008 second response time [16:54:52] repool mw8 and wait for it to crash [16:54:53] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 21686 bytes in 0.151 second response time [16:55:09] ok, let's do that then [16:55:15] JohnLewis: hmm, I wonder why it started all of a sudden [16:55:26] Reception123: mw8 being depooled [16:55:31] Reception123: because mw8 was out [16:55:37] oh, it has such a huge effect? [16:55:41] yes [16:55:42] !log repool mw8 [16:55:45] There isn't enough capacity to regularly take out a production server [16:55:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.87, 4.62, 5.64 [16:55:53] I thought one mw* being out wouldn't be that terrible for a short period of time [16:55:56] Reception123: now when it crashes, we'll hopefully get a trace [16:56:03] in theory [16:56:05] ok, let's wait then [16:56:17] https://www.irccloud.com/pastebin/eoovv6PI/ [16:56:18] ^ so far [16:56:43] that's normal [16:56:45] ish [16:57:29] similar WARNINGs so far [16:57:30] !log stopped puppet so it doesn't take over php-fpm on mw8 [16:57:49] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 6.502 second response time [16:57:54] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: debugging - rf1 & reception, last run 15 minutes ago with 0 failures [16:57:55] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 5.164 second response time [16:58:13] i'm more concerned about when it stops [16:58:25] at least we know it's serving traffic [16:58:39] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 0.197 second response time [16:58:42] yeah, though is it really supposed to be saying ' executing too slow '? [16:58:47] no [16:59:08] but slow pages from loads of images aren't our issue [16:59:10] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.791 second response time [16:59:10] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 2.143 second response time [16:59:12] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.252 second response time [17:00:14] https://www.irccloud.com/pastebin/h1oZvpY1/ [17:00:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 4.911 second response time [17:00:19] ^ RhinosF1 execution timed out now [17:00:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 20.54, 11.94, 8.12 [17:00:36] still also getting some too slow ones but I saw that one too between them [17:00:55] Reception123: okay [17:00:59] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.005 second response time [17:01:07] as long as it stays running, i don't care [17:01:10] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:01:21] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:01:28] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 2.119 second response time [17:01:31] i can see mw11 [17:01:47] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 9.192 second response time [17:01:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.82, 3.80, 4.96 [17:02:11] !log restart php-fpm on mw11 as out of workers [17:02:18] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:02:55] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.015 second response time [17:03:00] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:03] hmm [17:03:03] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.68, 5.07, 4.14 [17:03:09] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.018 second response time [17:03:10] I just want to login really, been trying for 20 minutes now [17:03:17] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15910 bytes in 0.581 second response time [17:03:25] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.008 second response time [17:03:29] Same problem here [17:03:32] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:36] JohnLewis: php is running on all servers now, hopefully it won't take long until mw8 crashes [17:03:39] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:40] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:52] We should definitely refrain from depooling mw8 too much in that case [17:03:52] and if we find a cause, we can reimage back until we get a fix [17:04:08] well it's crashing every half an hour anyway [17:04:20] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.004 second response time [17:04:25] last OOM was at 16:22:39 [17:04:26] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 8.738 second response time [17:04:36] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 3.495 second response time [17:04:37] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:04:45] There's been alot of that "executing too slow" for quite some time, even before mw8 was PHP7.4. I've been trying to debug that also, like why Special:CreateAccount seems to always take 15-20 seconds to post after submitting. I just not sure I know how to debug it much more... but yeah I really do think mw8 should be reimaged back until a fix can be done, but then debugging wouldn't be possible either. [17:04:54] I'm thinking in future, it might be more effective to use test3 as the test bed in future, that can be free installed and put into production with negligible user impact [17:04:55] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:04:56] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 0.228 second response time [17:04:58] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 3.66, 4.50, 4.04 [17:05:00] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:05:06] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:05:19] JohnLewis: yeah, that's a good idea. As clearly without user traffic we can't test some things on test3 [17:05:33] JohnLewis: true, we never saw any impact without traffic [17:05:59] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:00] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:06:00] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 8.988 second response time [17:06:31] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:06:42] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:01] JohnLewis: the other suggestion i've seen is Excimer [17:07:12] but i don't know where to send it's logs [17:07:25] disk is gonna massively slow us down [17:07:30] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 3.145 second response time [17:07:32] and potentially fill up [17:07:40] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:09:08] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15899 bytes in 4.765 second response time [17:09:09] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15899 bytes in 4.885 second response time [17:09:13] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15899 bytes in 5.393 second response time [17:09:13] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.009 second response time [17:09:30] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [17:09:38] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [17:09:54] Reception123: status [17:10:09] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.323 second response time [17:10:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 2.242 second response time [17:10:46] RhinosF1: no OOM, same sort of outputs [17:10:50] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 1.272 second response time [17:10:50] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 1.216 second response time [17:10:52] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 1.601 second response time [17:10:52] Reception123: ok [17:10:53] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:11:15] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 5.134 second response time [17:11:28] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15891 bytes in 0.583 second response time [17:11:40] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 21686 bytes in 3.362 second response time [17:11:41] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.694 second response time [17:11:44] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.311 second response time [17:13:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.21, 4.44, 4.45 [17:14:46] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:50] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:03] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 8.293 second response time [17:15:05] JohnLewis: can you reboot mw13, it's showing as critical and not responding to a php restart [17:15:31] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.148 second response time [17:15:32] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:33] sure [17:15:48] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.65, 4.54, 4.49 [17:16:02] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.464 second response time [17:16:20] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 9.717 second response time [17:16:25] !log rebooting mw13, unresponsive [17:18:08] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:28] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:18:29] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:30] PROBLEM - mw13 NTP time on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:18:31] PROBLEM - mw13 Check Gluster Clients on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:18:38] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:18:44] PROBLEM - mw13 Puppet on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:18:53] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.006 second response time [17:18:56] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:00] PROBLEM - mw13 SSH on mw13 is CRITICAL: connect to address 51.195.236.251 and port 22: Connection refused [17:19:03] PROBLEM - mw13 JobRunner Service on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:19:06] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.022 second response time [17:19:09] PROBLEM - mw13 Disk Space on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:19:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.013 second response time [17:19:20] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:21] PROBLEM - mw13 conntrack_table_size on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:19:23] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:25] PROBLEM - mw13 ferm_active on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:19:29] PROBLEM - mw13 HTTPS on mw13 is CRITICAL: connect to address 51.195.236.251 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [17:19:31] PROBLEM - mw13 PowerDNS Recursor on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:19:33] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:35] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:37] RECOVERY - mem2 ferm_active on mem2 is OK: OK ferm input default policy is set [17:19:37] PROBLEM - mw13 APT on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:19:47] RECOVERY - mem2 conntrack_table_size on mem2 is OK: OK: nf_conntrack is 0 % full [17:19:48] RECOVERY - mem2 Puppet on mem2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:19:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.62, 5.51, 4.94 [17:19:56] RECOVERY - mem2 PowerDNS Recursor on mem2 is OK: DNS OK: 0.246 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [17:20:05] RECOVERY - mem2 memcached on mem2 is OK: TCP OK - 0.000 second response time on 51.195.236.245 port 11211 [17:20:07] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 21686 bytes in 0.296 second response time [17:20:12] RECOVERY - mem2 NTP time on mem2 is OK: NTP OK: Offset -3.850460052e-05 secs [17:20:13] PROBLEM - mw13 php-fpm on mw13 is CRITICAL: connect to address 51.195.236.251 port 5666: Connection refusedconnect to host 51.195.236.251 port 5666: Connection refused [17:20:13] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.008 second response time [17:20:19] RECOVERY - mem2 Disk Space on mem2 is OK: DISK OK - free space: / 7385 MB (77% inode=88%); [17:20:22] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15899 bytes in 0.029 second response time [17:20:34] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 5.523 second response time [17:20:38] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 4.085 second response time [17:20:48] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:58] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 3.626 second response time [17:20:59] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 0.594 second response time [17:21:01] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.016 second response time [17:21:09] RECOVERY - mem2 APT on mem2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:21:14] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 3.215 second response time [17:21:25] RECOVERY - mem2 Current Load on mem2 is OK: OK - load average: 0.04, 0.05, 0.07 [17:21:43] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 9.751 second response time [17:21:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.84, 6.50, 5.36 [17:21:50] RhinosF1: still no OOM [17:21:51] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15891 bytes in 6.625 second response time [17:22:01] Reception123: hopefully soon [17:22:17] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 6.800 second response time [17:22:23] RhinosF1: yeah. It's weird that for once we're hoping for an OOM to happen rather than the opposite [17:22:33] * Reception123 has never before hoped to see an OOM happen [17:22:50] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.306 second response time [17:22:55] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.036 second response time [17:23:24] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 0.831 second response time [17:23:30] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 8.241 second response time [17:23:32] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 7.751 second response time [17:23:40] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15875 bytes in 0.321 second response time [17:24:17] RECOVERY - mw13 php-fpm on mw13 is OK: PROCS OK: 33 processes with command name 'php-fpm7.3' [17:24:19] !log forcefully shutdown and start mw13 from virtualisation layer [17:24:30] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.889 second response time [17:24:35] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 0.179 second response time [17:24:35] RECOVERY - mw13 NTP time on mw13 is OK: NTP OK: Offset 0.0003311932087 secs [17:24:37] RECOVERY - mw13 Check Gluster Clients on mw13 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [17:24:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:24:50] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 34 minutes ago with 0 failures [17:25:05] RECOVERY - mw13 SSH on mw13 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [17:25:10] RECOVERY - mw13 JobRunner Service on mw13 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [17:25:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:25:16] RECOVERY - mw13 Disk Space on mw13 is OK: DISK OK - free space: / 10063 MB (55% inode=76%); [17:25:23] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.561 second response time [17:25:29] RECOVERY - mw13 conntrack_table_size on mw13 is OK: OK: nf_conntrack is 2 % full [17:25:29] before I see OOMs roughly ever 20 mins and of course now there's none [17:25:31] RECOVERY - mw13 ferm_active on mw13 is OK: OK ferm input default policy is set [17:25:37] RECOVERY - mw13 PowerDNS Recursor on mw13 is OK: DNS OK: 0.338 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [17:25:38] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 2.603 second response time [17:25:38] RECOVERY - mw13 HTTPS on mw13 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 559 bytes in 0.060 second response time [17:25:43] RECOVERY - mw13 APT on mw13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [17:25:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:25:55] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.044 second response time [17:27:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.21, 17.96, 15.56 [17:27:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.15, 23.93, 18.57 [17:28:54] [Wed Dec 8 17:28:06 2021] Out of memory: Killed process 2183642 (php-fpm7.4) total-vm:2030672kB, anon-rss:203064kB, file-rss:0kB, shmem-rss:325312kB, UID:33 pgtables:2032kB oom_score_adj:0 [17:28:56] RhinosF1: ^ !! [17:29:00] i can see mw9 [17:29:06] Reception123: paste everything [17:29:09] https://www.irccloud.com/pastebin/i9HUrNx9/ [17:29:17] all I've got for 28: so far [17:29:31] Reception123: is gdb still running? [17:29:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.44, 22.74, 18.70 [17:29:57] yeah though output has slowed down [17:29:59] https://www.irccloud.com/pastebin/BZYeTSy5/ [17:30:04] nothing of use [17:31:06] Reception123: there's no even drop in memory usage [17:31:14] why has that not given a fancy trace [17:31:25] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.255 second response time [17:31:25] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.007 second response time [17:31:26] hmm [17:31:29] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.12, 19.31, 16.68 [17:31:39] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 21696 bytes in 0.677 second response time [17:31:47] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.332 second response time [17:31:56] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 9 backends are healthy [17:32:08] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.339 second response time [17:32:37] Reception123: you are joking [17:32:38] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 9 backends are healthy [17:32:42] it killed a child [17:32:50] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [17:32:50] finally an OOM [17:32:52] we need it to kill the master [17:32:57] no the wrong oom [17:32:58] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 9 backends are healthy [17:33:01] we've still got nothing :( [17:33:06] ah [17:33:21] gdb: please kill the parent and stop killing children [17:33:33] someone better not taking this out of context :P [17:33:34] oh [17:33:34] JohnLewis: why [17:34:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:35:24] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.05, 23.26, 20.12 [17:35:30] * RhinosF1 for the record does not condone violence [17:35:38] :D [17:35:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.13, 7.86, 5.32 [17:35:55] RhinosF1: new OOM [17:35:57] https://www.irccloud.com/pastebin/ip389Z7M/ [17:36:02] at :32 now [17:36:17] Reception123: what in gdb [17:36:28] I'm about to put the PM on [17:36:37] nothing useful [17:37:01] Reception123: Paste it [17:37:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.37, 22.02, 20.00 [17:37:22] https://www.irccloud.com/pastebin/M53QYM09/ [17:37:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.20, 6.63, 5.18 [17:38:19] want the dmesg output too? [17:38:22] we got 3 OOMs in total [17:38:27] All of children [17:38:31] yeah [17:38:33] Why won't it kill the parent [17:38:39] JohnLewis: ideas? [17:38:41] no clue [17:39:18] OOM killer is designed to be the least impactful when killing things [17:39:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.60, 5.77, 6.00 [17:39:54] JohnLewis: but why doesn't it kill children normally [17:40:01] Then we'd only loose 1 worker [17:40:04] Not 32 [17:40:14] It does kill children normally [17:40:37] JohnLewis: but why does the master keep restarting then [17:41:06] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.96, 20.38, 19.87 [17:41:13] There’s a confit value in PHP that makes the master restart when a certain number of children get killed [17:41:28] JohnLewis: right [17:41:52] But if it's restarting normally anyway fairly quickly, why does killing the rest help [17:41:53] let's see what it is [17:43:04] Reception123: can you see anything from the same child in logs [17:43:09] Before the crash [17:43:39] here's the full log [17:43:41] https://www.irccloud.com/pastebin/SCKmNWMl/ [17:43:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.27, 7.28, 6.48 [17:44:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.66, 2.91 [17:44:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.64, 7.39, 4.71 [17:44:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.29, 25.71, 21.95 [17:45:07] Reception123: that might do something [17:45:13] Let's stop for now [17:45:21] why stop? [17:45:52] Reception123: because we're not going to get better output and that log might be useful [17:46:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.70, 4.26, 3.21 [17:46:13] RhinosF1: well isn't the master going to be killed soon after a number of children? [17:46:21] Reception123: well ye [17:46:22] so that would maybe give us something better? [17:46:25] but we have a log [17:46:31] we can try wait [17:46:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.05, 7.00, 4.92 [17:46:53] well I mean keeping gdb open doesn't harm anything does it? [17:47:54] apart from slightly worse performance [17:47:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.96, 3.22 [17:48:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.98, 7.71, 5.40 [17:48:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.63, 23.45, 21.91 [17:49:17] if it's very slight I'll keep it for another 30 mins or so and if we don't get the master killed I guess we can end it [17:49:27] but if all you wanted is the logs from dmesg I could've given you that without gdb [17:49:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.96, 3.31 [17:50:03] Reception123: 'sudo /usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74' needs running after [17:50:14] ok, will do [17:50:17] and then re-enable puppet and start normal php-fpm [17:50:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.48, 7.99, 5.82 [17:50:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.58, 24.35, 22.45 [17:51:24] ok [17:52:13] RhinosF1: we've got the next one [17:52:16] [Wed Dec 8 17:50:15 2021] Out of memory: Killed process 2180474 (php-fpm7.4) total-vm:2148888kB, anon-rss:321152kB, file-rss:0kB, shmem-rss:458148kB, UID:33 pgtables:2556kB oom_score_adj:0 [17:52:16] [Wed Dec 8 17:50:15 2021] oom_reaper: reaped process 2180474 (php-fpm7.4), now anon-rss:0kB, file-rss:0kB, shmem-rss:458148kB [17:53:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.57, 3.34 [17:54:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.03, 8.26, 6.62 [17:54:27] another one! [17:54:33] https://www.irccloud.com/pastebin/znHFwioP/ [17:54:48] https://www.irccloud.com/pastebin/VrbaDEPM/ [17:54:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.18, 8.54, 6.46 [17:55:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.06, 4.04, 3.53 [17:57:03] JohnLewis: could we decrease pm.max_requests [17:57:19] if it's a leak, that may help [17:57:47] It’s possible to decrease if that’s what the MW want to do [17:57:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.38, 3.34 [17:58:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.92, 7.59, 6.77 [17:58:45] JohnLewis: it's my so far only possible solution [17:58:59] another OOM at 17:56:41 [17:59:05] JohnLewis: does it really take these many childs for it to kill the master? [17:59:18] JohnLewis: a memory leak could make sense [17:59:27] and lowering it will hopefully stop it [17:59:38] but we don't want the master to restart after so many childs [17:59:44] It very much sounds like a memory leak, I’m thinking in an extension more than core [17:59:52] JohnLewis: agreed [17:59:55] yeah, I remember you mentioning that [18:00:00] Finding it is proving hard [18:00:09] I'm also thinking it the mean time stability [18:00:13] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.25, 8.29, 7.14 [18:00:15] yeah... how can we really find it? [18:00:40] Reception123: I don't know but I don't want annoyed users [18:00:54] RhinosF1: yeah, definitely not. We can't afford too many experiments involving depools [18:01:05] If restarts are more often because of max requests [18:01:16] I don't want it to restart the master more [18:01:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.76, 6.45, 7.73 [18:01:33] Because restarting the childs is keeping memory under control [18:01:38] And that will slow us down [18:02:11] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.39, 7.62, 7.02 [18:02:42] and another one at :01 [18:03:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.04, 6.71, 7.64 [18:05:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:05:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.16, 7.01, 7.65 [18:06:07] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.56, 7.99, 7.24 [18:07:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.39, 8.21, 8.02 [18:09:29] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.77, 19.70, 18.06 [18:10:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:11:30] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.20, 19.79, 18.29 [18:11:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.81, 3.36 [18:15:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:17:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.69, 3.50 [18:20:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:21:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.83, 3.59 [18:22:27] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.79, 20.79, 19.43 [18:23:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:23:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.61, 3.54 [18:24:26] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.96, 20.19, 19.40 [18:25:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.09, 4.23, 3.75 [18:28:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:28:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.04, 20.49, 19.79 [18:28:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.52, 7.74, 8.00 [18:30:24] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.21, 18.92, 19.30 [18:30:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.67, 7.83, 7.98 [18:31:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.86, 3.76 [18:32:46] [02mediawiki] 07Universal-Omega commented on pull request 03#4287: Bump extensions/CommentStreams from `75b713d` to `196ae05` - 13https://git.io/JDTco [18:32:48] [02mediawiki] 07dependabot[bot] edited pull request 03#4287: Bump extensions/CommentStreams from `75b713d` to `196ae05` - 13https://git.io/JM4kx [18:32:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.97, 7.40, 7.83 [18:33:41] [02mediawiki] 07dependabot[bot] edited pull request 03#4287: Bump extensions/CommentStreams from `75b713d` to `196ae05` - 13https://git.io/JM4kx [18:33:47] [02mediawiki] 07dependabot[bot] opened pull request 03#4441: Bump extensions/CommentStreams from `75b713d` to `f98dd3b` - 13https://git.io/JDTcS [18:33:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CommentStreams-f98dd3b [+0/-0/±1] 13https://git.io/JDTc9 [18:33:50] [02miraheze/mediawiki] 07dependabot[bot] 03a8e7bc4 - Bump extensions/CommentStreams from `75b713d` to `f98dd3b` [18:33:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#4441: Bump extensions/CommentStreams from `75b713d` to `f98dd3b` - 13https://git.io/JDTcS [18:33:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#4441: Bump extensions/CommentStreams from `75b713d` to `f98dd3b` - 13https://git.io/JDTcS [18:33:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CommentStreams-f98dd3b - 13https://git.io/vbL5b [18:34:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.45, 7.76, 7.88 [18:35:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#4441: Bump extensions/CommentStreams from `75b713d` to `f98dd3b` - 13https://git.io/JDTcS [18:35:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4441: Bump extensions/CommentStreams from `75b713d` to `f98dd3b` - 13https://git.io/JDTcS [18:36:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.17, 7.17, 7.66 [18:37:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.72, 6.77, 7.94 [18:37:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.97, 6.24, 7.62 [18:41:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.53, 7.65, 7.96 [18:41:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.09, 7.15, 7.64 [18:43:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.78, 3.70 [18:43:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.55, 6.56, 7.37 [18:44:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.43, 7.33, 7.43 [18:45:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.40, 7.65, 7.91 [18:45:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.94, 3.44, 3.58 [18:45:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.90, 7.29, 7.53 [18:46:05] [02puppet] 07RhinosF1 opened pull request 03#2149: php-fpm: reduce pm.max_requests - 13https://git.io/JDTW0 [18:46:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.46, 6.58, 7.14 [18:47:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.57, 2.78, 3.32 [18:47:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.71, 6.75, 7.31 [18:48:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 7.32, 7.90 [18:48:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.31, 7.40, 7.35 [18:50:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:50:54] [02puppet] 07JohnFLewis closed pull request 03#2149: php-fpm: reduce pm.max_requests - 13https://git.io/JDTW0 [18:50:56] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDTlL [18:50:57] [02miraheze/puppet] 07RhinosF1 03fe75e8b - php-fpm: reduce pm.max_requests (#2149) [18:51:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.04, 7.64, 7.45 [18:52:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.06, 7.94, 7.96 [18:54:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 7.86, 7.95 [18:55:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:58:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.56, 8.42, 8.11 [18:59:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:59:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.04, 7.76, 7.73 [18:59:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.96, 4.85, 5.94 [19:01:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.29, 8.29, 7.90 [19:01:45] !log started php7.4-fpm on mw8 [19:01:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.68, 6.35, 6.36 [19:02:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:02:07] Reception123: has puppet ran since the config change [19:03:19] RhinosF1: I'll run it now [19:03:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.09, 7.69, 7.35 [19:03:25] just re-enabled it [19:03:41] Reception123: you need to restart php again then [19:03:54] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:03:57] ah [19:04:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:05:53] Reception123: did puppet apply the config change [19:06:31] yes [19:06:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:09:41] Reception123: let's wait a few hours then if you've restarted php and hope it doesn't OOM [19:09:49] I'll keep an eye on grafana [19:10:53] yeah [19:11:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.63, 7.94, 7.87 [19:11:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:13:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.33, 8.65, 8.14 [19:14:22] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[php7.3-fpm] [19:14:35] Reception123: ^ [19:15:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.57, 3.38 [19:17:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.28, 3.29 [19:17:33] RhinosF1: that's fu [19:17:34] *fun [19:17:45] * RhinosF1 running puppet [19:19:07] if it's failing due to a config syntax error or similar, won't running puppet just hide it (since no changes = it won't get refreshed) until you reboot the server and get very confused [19:19:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.31, 7.79, 7.97 [19:19:44] majavah: service restarted fine [19:19:50] Active: active (running) since Wed 2021-12-08 19:19:19 UTC; 12s ago [19:20:14] why did the puppet run fail then? [19:20:21] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:20:24] Thats normal [19:20:29] majavah: no idea [19:20:34] let me try another host [19:20:45] no it's ran on 13 fine [19:20:49] Config changes kill puppet at first i thought [19:21:03] reload by puppet did fail (code=exited, status=254) [19:21:44] I'd recommend finding out and fixing it instead of always getting confused and having it bite you at some point as you won't notice that one syntax error after you're used to failure messages like that [19:22:22] * RhinosF1 suggests someone with access looks at puppet log [19:24:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.93, 1.71, 1.20 [19:24:32] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[php7.3-fpm] [19:26:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.66, 3.43 [19:26:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.36, 1.27, 1.10 [19:26:38] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[php7.3-fpm] [19:30:58] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[php7.3-fpm] [19:31:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.63, 6.78, 7.17 [19:31:27] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:31:38] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:31:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.35, 7.09, 7.98 [19:32:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:32:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [19:32:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 2.98, 3.21 [19:32:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.01, 2.47, 1.82 [19:33:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.41, 6.99, 7.21 [19:33:25] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.013 second response time [19:33:39] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.318 second response time [19:34:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:34:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.70, 1.83, 1.66 [19:36:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.78, 1.46, 1.54 [19:37:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:38:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.78, 3.51 [19:39:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.73, 5.59, 6.53 [19:39:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.10, 6.78, 7.44 [19:40:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.34, 3.38 [19:41:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.30, 6.70, 7.34 [19:43:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.44, 6.70, 6.78 [19:43:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.84, 7.55, 7.57 [19:44:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.65, 3.49 [19:45:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.30, 6.59, 6.77 [19:46:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.12, 6.83, 7.86 [19:47:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.51, 7.00, 7.39 [19:48:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.07, 3.31 [19:49:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.39, 7.46, 7.50 [19:50:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.04, 7.42, 7.79 [19:51:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:52:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.54, 3.44 [19:52:14] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.37, 6.88, 6.86 [19:52:31] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:52:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.57, 7.62, 7.84 [19:52:37] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:53:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.21, 7.03, 7.34 [19:54:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.00, 3.25 [19:56:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:58:00] !log mw10, mw11: restart php-fpm [19:58:03] err 12 [19:58:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:05] fixing [19:58:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.29, 6.72, 7.92 [19:59:05] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:59:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.17, 6.89, 7.12 [20:00:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.84, 6.29, 7.68 [20:02:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.52, 6.53, 6.78 [20:02:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.00, 6.89, 7.16 [20:02:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 15.14, 10.56, 9.10 [20:04:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.70, 6.76, 7.10 [20:04:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 7.38, 7.80 [20:05:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.41, 7.26, 7.30 [20:06:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.90, 6.18, 7.29 [20:08:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.16, 5.93, 6.69 [20:08:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.83, 20.27, 23.55 [20:11:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.90, 7.54, 7.28 [20:12:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 7.13, 7.03 [20:13:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.88, 7.16, 7.17 [20:14:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.99, 7.82, 7.53 [20:14:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.48, 23.35, 23.98 [20:15:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.11, 8.20, 7.54 [20:16:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.80, 7.99, 7.68 [20:17:36] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.59, 6.95, 6.81 [20:18:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.56, 7.88, 7.39 [20:19:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.74, 7.85, 7.56 [20:20:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.41, 8.37, 7.86 [20:21:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:21:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.92, 8.55, 7.46 [20:21:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.21, 8.41, 7.79 [20:22:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.50, 7.83, 7.46 [20:22:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.48, 3.15 [20:24:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.63, 3.24 [20:26:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:26:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.31, 8.42, 7.76 [20:26:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.72, 3.32 [20:28:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.85, 7.51, 7.51 [20:29:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.70, 7.02, 7.52 [20:30:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.58, 3.33 [20:31:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.55, 8.31, 7.93 [20:32:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.23, 8.39, 7.81 [20:36:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:41:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:42:34] PROBLEM - ping6 on jobchron1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 127.73 ms [20:42:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.53, 3.56, 3.60 [20:44:37] RECOVERY - ping6 on jobchron1 is OK: PING OK - Packet loss = 0%, RTA = 2.34 ms [20:44:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.03, 4.02, 3.76 [20:49:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.56, 6.66, 7.81 [20:50:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.59, 3.66 [20:53:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:54:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.87, 3.77 [20:58:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:00:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.82, 3.82 [21:02:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.99, 6.41, 7.88 [21:04:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.16, 7.62, 8.15 [21:06:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.14, 7.00, 7.86 [21:07:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.05, 5.85, 6.66 [21:09:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.15, 7.17, 7.95 [21:10:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.50, 3.59 [21:10:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.81, 8.00, 7.95 [21:11:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 13.88, 10.22, 9.00 [21:12:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.67, 3.65 [21:13:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:13:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.81, 10.41, 8.28 [21:16:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.32, 7.91, 7.97 [21:18:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:18:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.88, 8.46, 8.17 [21:20:24] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.14, 3.84 [21:28:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.90, 3.91 [21:31:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:40:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.17, 3.04, 3.40 [21:41:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:43:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:47:15] [02mediawiki] 07Universal-Omega commented on pull request 03#4432: Bump extensions/Comments from `821b75a` to `d472224` - 13https://git.io/JDTDl [21:47:17] [02mediawiki] 07dependabot[bot] edited pull request 03#4432: Bump extensions/Comments from `821b75a` to `d472224` - 13https://git.io/JDvxQ [21:48:06] [02mediawiki] 07dependabot[bot] edited pull request 03#4432: Bump extensions/Comments from `821b75a` to `d472224` - 13https://git.io/JDvxQ [21:48:13] [02mediawiki] 07dependabot[bot] opened pull request 03#4442: Bump extensions/Comments from `821b75a` to `2aeb769` - 13https://git.io/JDTDB [21:48:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Comments-2aeb769 [+0/-0/±1] 13https://git.io/JDTDR [21:48:16] [02miraheze/mediawiki] 07dependabot[bot] 03c784b13 - Bump extensions/Comments from `821b75a` to `2aeb769` [21:48:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Comments-2aeb769 - 13https://git.io/vbL5b [21:48:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#4442: Bump extensions/Comments from `821b75a` to `2aeb769` - 13https://git.io/JDTDB [21:48:21] [02mediawiki] 07dependabot[bot] labeled pull request 03#4442: Bump extensions/Comments from `821b75a` to `2aeb769` - 13https://git.io/JDTDB [21:48:22] [02mediawiki] 07dependabot[bot] commented on pull request 03#4432: Bump extensions/Comments from `821b75a` to `d472224` - 13https://git.io/JDTDu [21:48:24] [02mediawiki] 07dependabot[bot] closed pull request 03#4432: Bump extensions/Comments from `821b75a` to `d472224` - 13https://git.io/JDvxQ [21:48:25] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Comments-d472224 [21:48:27] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Comments-d472224 - 13https://git.io/vbL5b [21:53:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:57:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 4.06, 3.63 [21:59:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.87, 3.61 [22:07:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 3.05, 3.36 [22:09:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.31, 7.07, 7.94 [22:11:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.39, 7.75, 8.07 [22:12:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.45, 6.75, 7.86 [22:13:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.48, 7.20, 7.80 [22:13:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.36, 6.33, 7.71 [22:13:46] PROBLEM - cp30 PowerDNS Recursor on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:13:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [22:14:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:09] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:10] PROBLEM - ping6 on cp30 is CRITICAL: PING CRITICAL - Packet loss = 100% [22:14:15] PROBLEM - ping4 on cp30 is CRITICAL: PING CRITICAL - Packet loss = 100% [22:14:19] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:31] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:37] PROBLEM - cp30 HTTPS on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:39] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:42] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:43] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:43] PROBLEM - cp30 NTP time on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:46] PROBLEM - cp30 Stunnel Http for mon2 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:53] PROBLEM - cp30 conntrack_table_size on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:54] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:02] PROBLEM - cp30 SSH on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:10] PROBLEM - cp30 ferm_active on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [22:15:20] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:21] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:25] PROBLEM - cp30 APT on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:42] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:07] PROBLEM - Host cp30 is DOWN: PING CRITICAL - Packet loss = 100% [22:16:17] paladox, JohnLewis: ^ [22:16:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.18, 6.71, 7.92 [22:17:12] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDT9J [22:17:13] [02miraheze/mediawiki] 07Universal-Omega 03985f4d0 - Use pull_request_target [22:17:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.21, 7.40, 7.77 [22:17:45] [02mediawiki] 07Universal-Omega commented on pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDT9k [22:17:46] [02mediawiki] 07dependabot[bot] edited pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [22:18:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.20, 7.19, 7.62 [22:18:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 7.45, 8.06 [22:18:42] [02mediawiki] 07dependabot[bot] edited pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [22:18:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-788cef8 [+0/-0/±1] 13https://git.io/JDT9m [22:18:45] [02miraheze/mediawiki] 07dependabot[bot] 0387a8036 - Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` [22:18:47] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [22:19:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [22:19:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [22:19:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#4439: Bump extensions/GlobalNewFiles from `2ee8275` to `788cef8` - 13https://git.io/JDvpY [22:19:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.73, 7.34, 7.70 [22:20:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.33, 6.97, 7.47 [22:20:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.81, 1.57, 1.11 [22:21:08] !log reboot cp30 via ovh panel [22:21:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:22:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.32, 1.47, 1.13 [22:24:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.25, 7.18, 7.85 [22:25:23] [02mediawiki] 07Universal-Omega commented on pull request 03#4438: Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` - 13https://git.io/JDT9D [22:25:25] [02mediawiki] 07dependabot[bot] edited pull request 03#4438: Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` - 13https://git.io/JDvpq [22:25:36] [02mediawiki] 07Universal-Omega commented on pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDT9S [22:25:38] [02mediawiki] 07dependabot[bot] edited pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDvpT [22:25:48] [02mediawiki] 07Universal-Omega commented on pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDT97 [22:25:50] [02mediawiki] 07dependabot[bot] edited pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [22:26:02] [02mediawiki] 07Universal-Omega commented on pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDT9d [22:26:05] [02mediawiki] 07dependabot[bot] edited pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDvxp [22:26:19] [02mediawiki] 07dependabot[bot] edited pull request 03#4438: Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` - 13https://git.io/JDvpq [22:26:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SpriteSheet-b7a1c91 [+0/-0/±1] 13https://git.io/JDT9A [22:26:22] [02miraheze/mediawiki] 07dependabot[bot] 0365c2725 - Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` [22:26:24] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4438: Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` - 13https://git.io/JDvpq [22:26:28] [02mediawiki] 07dependabot[bot] edited pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDvpT [22:26:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RottenLinks-c400776 [+0/-0/±1] 13https://git.io/JDT9p [22:26:31] [02miraheze/mediawiki] 07dependabot[bot] 035349980 - Bump extensions/RottenLinks from `895c528` to `c400776` [22:26:33] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDvpT [22:26:38] [02mediawiki] 07dependabot[bot] edited pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [22:26:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DataDump-aa33858 [+0/-0/±1] 13https://git.io/JDT9h [22:26:42] [02miraheze/mediawiki] 07dependabot[bot] 03c6cbcd8 - Bump extensions/DataDump from `87f48f8` to `aa33858` [22:26:43] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [22:26:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#4438: Bump extensions/SpriteSheet from `d62639c` to `b7a1c91` - 13https://git.io/JDvpq [22:26:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDvpT [22:26:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#4437: Bump extensions/RottenLinks from `895c528` to `c400776` - 13https://git.io/JDvpT [22:26:58] [02mediawiki] 07dependabot[bot] edited pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDvxp [22:27:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-0cf64b5 [+0/-0/±1] 13https://git.io/JDTHv [22:27:01] [02miraheze/mediawiki] 07dependabot[bot] 03f435a43 - Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` [22:27:03] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDvxp [22:27:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [22:27:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [22:27:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#4436: Bump extensions/DataDump from `87f48f8` to `aa33858` - 13https://git.io/JDvpv [22:27:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDvxp [22:27:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4435: Bump extensions/UniversalLanguageSelector from `126936f` to `0cf64b5` - 13https://git.io/JDvxp [22:28:34] [02mediawiki] 07Universal-Omega commented on pull request 03#4434: Bump extensions/PageForms from `5d178f8` to `7d124a4` - 13https://git.io/JDTHL [22:28:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.54, 7.16, 7.64 [22:28:36] [02mediawiki] 07dependabot[bot] edited pull request 03#4434: Bump extensions/PageForms from `5d178f8` to `7d124a4` - 13https://git.io/JDvxA [22:28:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.66, 6.46, 7.70 [22:29:03] [02mediawiki] 07Universal-Omega commented on pull request 03#4433: Bump extensions/cldr from `937150a` to `2941387` - 13https://git.io/JDTHG [22:29:05] [02mediawiki] 07dependabot[bot] edited pull request 03#4433: Bump extensions/cldr from `937150a` to `2941387` - 13https://git.io/JDvxF [22:29:16] [02mediawiki] 07Universal-Omega commented on pull request 03#4431: Bump extensions/WikiForum from `ba358e1` to `e76b1cf` - 13https://git.io/JDTHc [22:29:18] [02mediawiki] 07dependabot[bot] edited pull request 03#4431: Bump extensions/WikiForum from `ba358e1` to `e76b1cf` - 13https://git.io/JDvx9 [22:29:33] [02mediawiki] 07Universal-Omega commented on pull request 03#4430: Bump extensions/PageSchemas from `23c8865` to `58064fe` - 13https://git.io/JDTHW [22:29:34] [02mediawiki] 07dependabot[bot] edited pull request 03#4434: Bump extensions/PageForms from `5d178f8` to `7d124a4` - 13https://git.io/JDvxA [22:29:36] [02mediawiki] 07dependabot[bot] edited pull request 03#4430: Bump extensions/PageSchemas from `23c8865` to `58064fe` - 13https://git.io/JDvxD [22:29:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-cc9850d [+0/-0/±1] 13https://git.io/JDTHl [22:29:41] [02miraheze/mediawiki] 07dependabot[bot] 030f87545 - Bump extensions/PageForms from `5d178f8` to `cc9850d` [22:29:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-cc9850d - 13https://git.io/vbL5b [22:29:44] [02mediawiki] 07dependabot[bot] opened pull request 03#4443: Bump extensions/PageForms from `5d178f8` to `cc9850d` - 13https://git.io/JDTH8 [22:29:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#4443: Bump extensions/PageForms from `5d178f8` to `cc9850d` - 13https://git.io/JDTH8 [22:29:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#4443: Bump extensions/PageForms from `5d178f8` to `cc9850d` - 13https://git.io/JDTH8 [22:29:48] [02mediawiki] 07dependabot[bot] commented on pull request 03#4434: Bump extensions/PageForms from `5d178f8` to `7d124a4` - 13https://git.io/JDTHB [22:29:50] [02mediawiki] 07dependabot[bot] closed pull request 03#4434: Bump extensions/PageForms from `5d178f8` to `7d124a4` - 13https://git.io/JDvxA [22:29:51] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-7d124a4 [22:29:53] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-7d124a4 - 13https://git.io/vbL5b [22:29:54] [02mediawiki] 07dependabot[bot] edited pull request 03#4433: Bump extensions/cldr from `937150a` to `2941387` - 13https://git.io/JDvxF [22:29:56] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/cldr-2941387 [+0/-0/±1] 13https://git.io/JDTHE [22:29:57] [02miraheze/mediawiki] 07dependabot[bot] 033fb0f46 - Bump extensions/cldr from `937150a` to `2941387` [22:29:59] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4433: Bump extensions/cldr from `937150a` to `2941387` - 13https://git.io/JDvxF [22:30:01] [02mediawiki] 07dependabot[bot] edited pull request 03#4431: Bump extensions/WikiForum from `ba358e1` to `e76b1cf` - 13https://git.io/JDvx9 [22:30:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiForum-e76b1cf [+0/-0/±1] 13https://git.io/JDTHu [22:30:05] [02miraheze/mediawiki] 07dependabot[bot] 034f45939 - Bump extensions/WikiForum from `ba358e1` to `e76b1cf` [22:30:06] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4431: Bump extensions/WikiForum from `ba358e1` to `e76b1cf` - 13https://git.io/JDvx9 [22:30:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#4443: Bump extensions/PageForms from `5d178f8` to `cc9850d` - 13https://git.io/JDTH8 [22:30:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#4443: Bump extensions/PageForms from `5d178f8` to `cc9850d` - 13https://git.io/JDTH8 [22:30:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#4433: Bump extensions/cldr from `937150a` to `2941387` - 13https://git.io/JDvxF [22:30:22] [02mediawiki] 07dependabot[bot] edited pull request 03#4430: Bump extensions/PageSchemas from `23c8865` to `58064fe` - 13https://git.io/JDvxD [22:30:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageSchemas-8cf8155 [+0/-0/±1] 13https://git.io/JDTHV [22:30:31] [02miraheze/mediawiki] 07dependabot[bot] 030ce6c21 - Bump extensions/PageSchemas from `23c8865` to `8cf8155` [22:30:32] [02mediawiki] 07dependabot[bot] opened pull request 03#4444: Bump extensions/PageSchemas from `23c8865` to `8cf8155` - 13https://git.io/JDTHw [22:30:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.81, 6.04, 6.76 [22:30:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-8cf8155 - 13https://git.io/vbL5b [22:30:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.40, 7.22, 7.60 [22:30:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#4431: Bump extensions/WikiForum from `ba358e1` to `e76b1cf` - 13https://git.io/JDvx9 [22:30:37] [02mediawiki] 07dependabot[bot] labeled pull request 03#4444: Bump extensions/PageSchemas from `23c8865` to `8cf8155` - 13https://git.io/JDTHw [22:30:38] [02mediawiki] 07dependabot[bot] labeled pull request 03#4444: Bump extensions/PageSchemas from `23c8865` to `8cf8155` - 13https://git.io/JDTHw [22:30:40] [02mediawiki] 07dependabot[bot] commented on pull request 03#4430: Bump extensions/PageSchemas from `23c8865` to `58064fe` - 13https://git.io/JDTHr [22:30:41] [02mediawiki] 07dependabot[bot] closed pull request 03#4430: Bump extensions/PageSchemas from `23c8865` to `58064fe` - 13https://git.io/JDvxD [22:30:43] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-58064fe - 13https://git.io/vbL5b [22:30:44] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-58064fe [22:30:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#4444: Bump extensions/PageSchemas from `23c8865` to `8cf8155` - 13https://git.io/JDTHw [22:30:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#4444: Bump extensions/PageSchemas from `23c8865` to `8cf8155` - 13https://git.io/JDTHw [22:30:56] [02mediawiki] 07Universal-Omega commented on pull request 03#4429: Bump extensions/RatePage from `b774f5c` to `1182c92` - 13https://git.io/JDTH6 [22:30:58] [02mediawiki] 07dependabot[bot] edited pull request 03#4429: Bump extensions/RatePage from `b774f5c` to `1182c92` - 13https://git.io/JDvxP [22:31:13] [02mediawiki] 07Universal-Omega commented on pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDTHM [22:31:15] [02mediawiki] 07dependabot[bot] edited pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDvxi [22:31:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.37, 6.20, 6.76 [22:31:31] [02mediawiki] 07Universal-Omega commented on pull request 03#4427: Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` - 13https://git.io/JDTHD [22:31:32] [02mediawiki] 07dependabot[bot] edited pull request 03#4427: Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` - 13https://git.io/JDvxw [22:31:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.68, 5.21, 6.53 [22:31:43] [02mediawiki] 07dependabot[bot] edited pull request 03#4429: Bump extensions/RatePage from `b774f5c` to `1182c92` - 13https://git.io/JDvxP [22:31:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RatePage-1182c92 [+0/-0/±1] 13https://git.io/JDTH9 [22:31:46] [02miraheze/mediawiki] 07dependabot[bot] 038e21194 - Bump extensions/RatePage from `b774f5c` to `1182c92` [22:31:48] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4429: Bump extensions/RatePage from `b774f5c` to `1182c92` - 13https://git.io/JDvxP [22:31:58] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JDTHH [22:31:59] [02miraheze/mw-config] 07RhinosF1 03ef5b0f1 - Profile: Profile mw8 [22:32:01] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [22:32:04] [02mw-config] 07RhinosF1 opened pull request 03#4273: Profile: Profile mw8 - 13https://git.io/JDTHQ [22:32:07] [02mediawiki] 07dependabot[bot] edited pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDvxi [22:32:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Moderation-956c332 [+0/-0/±1] 13https://git.io/JDTH5 [22:32:12] [02miraheze/mediawiki] 07dependabot[bot] 0305f89f2 - Bump extensions/Moderation from `916fa2d` to `956c332` [22:32:13] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDvxi [22:32:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#4429: Bump extensions/RatePage from `b774f5c` to `1182c92` - 13https://git.io/JDvxP [22:32:16] [02mediawiki] 07dependabot[bot] edited pull request 03#4427: Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` - 13https://git.io/JDvxw [22:32:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-3afdaf9 [+0/-0/±1] 13https://git.io/JDTHF [22:32:20] [02miraheze/mediawiki] 07dependabot[bot] 03df40093 - Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` [22:32:21] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4427: Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` - 13https://git.io/JDvxw [22:32:23] [02mediawiki] 07Universal-Omega commented on pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDTHb [22:32:25] [02mediawiki] 07dependabot[bot] edited pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [22:32:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDvxi [22:32:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4428: Bump extensions/Moderation from `916fa2d` to `956c332` - 13https://git.io/JDvxi [22:32:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#4427: Bump extensions/MirahezeMagic from `378ce3c` to `3afdaf9` - 13https://git.io/JDvxw [22:33:07] miraheze/mw-config - RhinosF1 the build has errored. [22:33:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-a4ee20b [+0/-0/±1] 13https://git.io/JDTQv [22:33:16] [02miraheze/mediawiki] 07dependabot[bot] 03c524c65 - Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` [22:33:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.04, 6.94, 7.96 [22:33:18] [02mediawiki] 07dependabot[bot] edited pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [22:33:19] [02mediawiki] 07dependabot[bot] synchronize pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [22:33:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [22:33:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [22:33:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#4426: Bump extensions/MatomoAnalytics from `9528cc3` to `a4ee20b` - 13https://git.io/JDvx2 [22:34:45] [02mw-config] 07Universal-Omega reviewed pull request 03#4273 commit - 13https://git.io/JDTQm [22:34:46] RECOVERY - Host cp30 is UP: PING OK - Packet loss = 0%, RTA = 81.78 ms [22:34:48] RECOVERY - cp30 ferm_active on cp30 is OK: OK ferm input default policy is set [22:34:48] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 13687 MB (35% inode=97%); [22:34:48] RECOVERY - cp30 PowerDNS Recursor on cp30 is OK: DNS OK: 1.607 second response time. miraheze.org returns 149.56.141.75,2607:5300:201:3100::5ebc [22:34:48] RECOVERY - cp30 APT on cp30 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:34:53] RECOVERY - ping6 on cp30 is OK: PING OK - Packet loss = 0%, RTA = 78.76 ms [22:35:17] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.11, 0.10, 0.04 [22:35:28] RECOVERY - ping4 on cp30 is OK: PING OK - Packet loss = 0%, RTA = 79.52 ms [22:35:28] RECOVERY - cp30 conntrack_table_size on cp30 is OK: OK: nf_conntrack is 0 % full [22:35:33] RECOVERY - cp30 SSH on cp30 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [22:35:48] RECOVERY - cp30 NTP time on cp30 is OK: NTP OK: Offset -0.006734669209 secs [22:36:01] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JDTQn [22:36:02] [02miraheze/mw-config] 07RhinosF1 03613ace9 - Update PhpAutoPrepend.php [22:36:04] [02mw-config] 07RhinosF1 synchronize pull request 03#4273: Profile: Profile mw8 - 13https://git.io/JDTHQ [22:36:04] CosmicAlpha: ^ [22:36:23] !log mw8: add '*/5 * * * * find /srv/mediawiki/cache/profile . -type f -mmin +40 -delete' to crontab [22:36:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.85, 5.14, 6.62 [22:37:02] miraheze/mw-config - RhinosF1 the build has errored. [22:38:25] [02mw-config] 07Universal-Omega commented on pull request 03#4273: Profile: Profile mw8 - 13https://git.io/JDTQV [22:39:00] [02mw-config] 07RhinosF1 commented on pull request 03#4273: Profile: Profile mw8 - 13https://git.io/JDTQr [22:39:16] CosmicAlpha: there's different config because we want to save output [22:40:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.06, 5.99, 6.70 [22:40:43] [02mw-config] 07Universal-Omega commented on pull request 03#4273: Profile: Profile mw8 - 13https://git.io/JDTQS [22:40:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.59, 21.78, 23.79 [22:41:16] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.38, 4.87, 6.56 [22:41:37] CosmicAlpha: it's supposed to exist for as short a period of time as possible [22:42:08] [02mw-config] 07Universal-Omega reviewed pull request 03#4273 commit - 13https://git.io/JDTQ7 [22:42:10] [02mw-config] 07Universal-Omega reviewed pull request 03#4273 commit - 13https://git.io/JDTQ5 [22:42:15] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JDTQF [22:42:17] [02miraheze/mw-config] 07RhinosF1 0355d5c3e - Update PhpAutoPrepend.php [22:42:18] [02mw-config] 07RhinosF1 synchronize pull request 03#4273: Profile: Profile mw8 - 13https://git.io/JDTHQ [22:42:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 37.83, 26.77, 25.30 [22:43:02] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JDTQp [22:43:03] [02miraheze/mw-config] 07RhinosF1 03c543a9b - Update PhpAutoPrepend.php [22:43:05] [02mw-config] 07RhinosF1 synchronize pull request 03#4273: Profile: Profile mw8 - 13https://git.io/JDTHQ [22:43:18] miraheze/mw-config - RhinosF1 the build has errored. [22:44:00] miraheze/mw-config - RhinosF1 the build passed. [22:44:42] [02puppet] 07RhinosF1 opened pull request 03#2150: mw8: enable tideways - 13https://git.io/JDT7k [22:44:47] paladox: ^ [22:44:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 6.44, 6.60 [22:45:13] what will that be used for? [22:45:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.69, 6.30, 6.81 [22:45:34] paladox: the profiling [22:45:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.51, 3.22 [22:45:53] ok, it won't help with looking at OOMs though, it's more used for performance [22:46:04] [02mw-config] 07paladox closed pull request 03#4273: Profile: Profile mw8 - 13https://git.io/JDTHQ [22:46:05] [02mw-config] 07paladox deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [22:46:07] [02miraheze/mw-config] 07paladox deleted branch 03RhinosF1-patch-3 [22:46:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDT7s [22:46:09] err wrong change [22:46:10] [02miraheze/mw-config] 07RhinosF1 031509404 - Profile: Profile mw8 (#4273) [22:46:17] [02puppet] 07paladox closed pull request 03#2150: mw8: enable tideways - 13https://git.io/JDT7k [22:46:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDT7n [22:46:20] [02miraheze/puppet] 07RhinosF1 0335f43c6 - mw8: enable tideways (#2150) [22:46:39] pulled RhinosF1 [22:46:42] you can run puppet [22:46:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.85, 21.98, 23.57 [22:47:01] paladox: it's a memory leak i think, i'm following https://scoutapm.com/blog/php-memory-leaks-how-to-find-and-fix-them [22:47:01] [url] PHP Memory Leaks: How to Find and Fix Them | Scout APM Blog | scoutapm.com [22:47:09] oh [22:47:15] miraheze/mw-config - paladox the build passed. [22:47:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.03, 6.92, 6.97 [22:47:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.99, 3.95, 3.41 [22:48:49] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.24, 6.24, 6.47 [22:48:53] RhinosF1: You might want to add sampling to the profiler actually. [22:49:11] CosmicAlpha: i added a cron to purge old files [22:49:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.72, 7.30, 7.11 [22:49:24] !log [@mw11] starting deploy of {'config': True} to all [22:49:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.79, 7.06, 6.91 [22:49:37] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 13s [22:49:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.82, 3.43 [22:50:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.22, 24.04, 24.03 [22:51:11] CosmicAlpha: i'm not seeing it [22:51:44] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 4.13, 3.58 [22:52:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.98, 22.81, 23.61 [22:53:16] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.23, 5.86, 6.59 [22:54:51] !log [@test3] starting deploy of {'config': True} to skip [22:54:52] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:54:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.79, 3.56 [22:55:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.97, 4.99, 5.95 [22:57:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 40.58, 26.62, 24.55 [22:57:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 15.77, 9.77, 7.92 [22:57:47] It’s taking longer to load pages [22:57:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 14.40, 8.52, 7.12 [22:59:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 6.43, 5.90 [22:59:16] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.53, 10.41, 8.27 [22:59:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 6.26, 5.72 [22:59:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 6.82, 8.02, 7.17 [23:00:01] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mw11] starting deploy of {'l10nupdate': True} to all [23:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:25] the patch isn't working though [23:00:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.16, 6.13, 5.84 [23:01:21] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.89, 7.81, 7.69 [23:01:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.30, 5.64, 5.57 [23:01:37] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.69, 7.44, 7.07 [23:03:13] CosmicAlpha: ideas? [23:03:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.19, 6.55, 7.18 [23:03:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.42, 6.28, 6.68 [23:03:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.66, 3.08, 3.37 [23:04:44] !log restart php-fpm to purge any cache / refresh config on mw8 [23:04:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 20.25, 11.39, 7.72 [23:05:37] paladox: cp30 is still failed [23:06:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:06:52] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.67, 19.28, 17.04 [23:06:52] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.68, 5.27, 6.50 [23:07:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.43, 7.12, 7.08 [23:08:30] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDTdC [23:08:32] [02miraheze/mw-config] 07RhinosF1 034993954 - profile-mw8: directly set $wgProfiler [23:08:51] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.30, 21.51, 18.11 [23:08:53] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 6.84, 3.78, 1.59 [23:09:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.77, 5.81, 6.73 [23:09:30] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15891 bytes in 0.342 second response time [23:09:32] miraheze/mw-config - RhinosF1 the build passed. [23:09:45] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15902 bytes in 0.324 second response time [23:09:48] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.347 second response time [23:09:49] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15909 bytes in 0.336 second response time [23:10:02] !log [@mw11] starting deploy of {'config': True} to all [23:10:06] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.616 second response time [23:10:11] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 9 backends are healthy [23:10:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:10:14] RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3108 bytes in 1.562 second response time [23:10:14] RhinosF1: fixed [23:10:21] RECOVERY - cp30 Stunnel Http for mon2 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 35417 bytes in 0.372 second response time [23:10:28] ty [23:10:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.63, 8.25, 6.61 [23:10:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 1.711 second response time [23:10:47] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 44s [23:10:52] PROBLEM - test3 Current Load on test3 is WARNING: WARNING - load average: 2.70, 3.68, 1.85 [23:10:54] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:11:07] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:11:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.87, 8.29, 7.44 [23:11:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.63, 6.99, 6.13 [23:12:01] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 2% [23:12:07] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:12:50] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.98, 22.05, 19.20 [23:12:52] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.94, 2.65, 1.69 [23:13:07] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:13:12] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.57, 8.53, 7.64 [23:14:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.71, 7.32, 6.59 [23:15:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.96, 7.68, 7.39 [23:15:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:15:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.69, 8.35, 7.26 [23:15:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.31, 7.74, 6.62 [23:16:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:49] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.77, 19.93, 18.95 [23:17:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.68, 7.59, 7.90 [23:18:18] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.707 second response time [23:18:55] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.80, 8.03, 7.55 [23:20:50] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.80, 5.30, 3.89 [23:20:54] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.54, 26.40, 22.17 [23:21:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:22:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.66, 3.39 [23:22:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.06, 5.54, 4.14 [23:23:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.82, 8.19, 7.91 [23:24:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.62, 8.65, 7.54 [23:25:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:26:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.58, 3.40 [23:26:50] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.06, 5.67, 4.53 [23:26:56] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:28:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.54, 3.41 [23:28:49] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.38, 6.10, 4.81 [23:28:51] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:28:52] PROBLEM - test3 Current Load on test3 is WARNING: WARNING - load average: 3.85, 2.87, 1.83 [23:29:00] !log [@mw11] finished deploy of {'l10nupdate': True} to all - FAIL: [35072, 0, 0, 0, 0, 0, 0, 0, 0] in 1737s [23:29:09] !log [@test3] starting deploy of {'config': True} to skip [23:29:10] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:29:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:27] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.75, 2.98, 3.23 [23:30:49] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.20, 5.47, 4.75 [23:30:50] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 17.90, 23.22, 22.84 [23:30:52] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 1.59, 2.37, 1.77 [23:31:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:32:35] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:33:47] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2025s [23:33:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:31] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.307 second response time [23:34:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.20, 7.67, 7.83 [23:35:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:37:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.16, 6.73, 7.84 [23:38:49] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.41, 4.97, 4.84 [23:40:47] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.57, 16.61, 19.98 [23:40:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.58, 7.22, 7.84 [23:42:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.97, 5.66, 6.78 [23:43:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.31, 6.93, 7.92 [23:44:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.00, 7.85, 7.89 [23:45:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.04, 5.46, 6.78 [23:47:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:48:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [23:49:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:51:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.73, 7.22, 7.53 [23:52:21] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:00] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:10] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:13] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:53:19] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.77, 8.94, 7.55 [23:53:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.47, 8.50, 7.44 [23:53:36] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:53:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.67, 7.39, 7.56 [23:53:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:54:16] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:54:21] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.35, 3.65, 3.14 [23:54:25] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 5.663 second response time [23:54:37] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.95, 7.82, 7.30 [23:55:42] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 8.235 second response time [23:56:17] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.22, 3.73, 3.22 [23:56:21] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 4.571 second response time [23:56:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 21696 bytes in 4.673 second response time [23:57:01] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15901 bytes in 0.008 second response time [23:57:12] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.005 second response time [23:57:13] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 21698 bytes in 0.119 second response time [23:57:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.68, 7.39, 7.27 [23:58:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:59:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.69, 7.78, 7.41 [23:59:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online