[00:01:37] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1436 MB (15% inode=80%); [00:02:12] JohnLewis, load times for phabricator.miraheze.org have gotten very bad. Takes forever for a new comment on a task to be posted [00:02:34] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.11, 9.39, 8.97 [00:03:38] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 792 failures. Last run 2 minutes ago with 792 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:03:40] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:44] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.87, 5.75, 5.12 [00:04:34] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6152 bytes in 2.271 second response time [00:04:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.29, 8.32, 8.64 [00:05:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.22, 7.88, 8.25 [00:05:44] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 6.24, 7.55, 6.40 [00:05:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.01, 4.21, 3.39 [00:07:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.94, 3.40 [00:08:29] dmehus: yeah, IO on cloud11's SSDs is pretty high because of piwik db migration [00:08:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.60, 7.90, 8.38 [00:09:41] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.09, 7.82, 6.78 [00:11:22] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.01, 5.54, 5.24 [00:11:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.07, 3.16 [00:13:16] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.17, 5.39, 5.22 [00:13:39] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.24, 7.71, 6.98 [00:14:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 8.36, 8.16 [00:16:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 8.36, 8.20 [00:16:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 9.00, 8.66 [00:17:36] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.02, 7.75, 7.15 [00:19:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 1.77, 1.47 [00:19:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.47, 8.92, 8.38 [00:19:35] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.42, 7.49, 7.12 [00:20:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.62, 8.52, 8.30 [00:20:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.50, 9.67, 8.77 [00:21:04] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.42, 1.66, 1.47 [00:21:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.15, 9.00, 8.48 [00:21:24] JohnLewis, ah [00:21:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.70, 7.98, 8.13 [00:22:02] what's piwik? [00:22:14] [[Tech:Piwik]] [00:22:14] https://meta.miraheze.org/wiki/Tech:Piwik [00:22:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.55, 9.07, 8.67 [00:22:49] Oh, Matomo [00:23:00] hrm, that could actually be a potentially useful redirect [00:23:32] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.80, 8.42, 7.54 [00:24:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.29, 8.45, 8.48 [00:24:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.52, 7.85, 8.41 [00:25:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.25, 10.10, 8.95 [00:25:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.34, 3.58, 3.15 [00:27:53] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.648 second response time [00:28:07] PROBLEM - puppet111 Current Load on puppet111 is CRITICAL: CRITICAL - load average: 6.67, 5.60, 3.77 [00:28:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.88, 3.11, 3.04 [00:29:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.51, 10.00, 9.24 [00:30:07] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 2.41, 4.33, 3.53 [00:31:26] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.17, 7.68, 7.63 [00:32:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.41, 9.30, 8.78 [00:33:37] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:40] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:34:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.95, 8.81, 8.67 [00:36:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 9.16, 8.42 [00:37:00] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.52, 4.31, 4.88 [00:40:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 8.59, 8.20 [00:42:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.18, 9.46, 8.56 [00:42:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.97, 7.95, 8.17 [00:43:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.27, 9.19, 8.96 [00:44:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.76, 9.40, 8.64 [00:45:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.49, 8.82, 8.86 [00:48:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.35, 9.58, 8.86 [00:48:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.38, 9.97, 9.29 [00:50:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 9.47, 8.92 [00:50:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 9.70, 9.27 [00:51:13] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.38, 7.45, 7.26 [00:52:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.91, 9.62, 9.00 [00:52:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.70, 10.04, 9.44 [00:53:12] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 6.93, 7.14, 7.17 [00:54:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.60, 8.69, 8.17 [00:55:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.08, 2.71 [00:57:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.91, 3.76, 3.08 [00:57:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 3.50, 2.89 [00:57:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.99, 9.47, 9.23 [00:58:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.65, 8.99, 8.45 [00:59:08] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 5.79, 6.27, 6.80 [00:59:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.48, 3.06 [00:59:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.41, 2.94 [01:00:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.53, 3.02 [01:01:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BlogPage-cd63768 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4b111a3b3df0 [01:01:13] [02miraheze/mediawiki] 07dependabot[bot] 034b111a3 - Bump extensions/BlogPage from `0b24c44` to `cd63768` [01:01:14] [02mediawiki] 07dependabot[bot] opened pull request 03#5761: Bump extensions/BlogPage from `0b24c44` to `cd63768` - 13https://github.com/miraheze/mediawiki/pull/5761 [01:01:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BlogPage-cd63768 - 13https://github.com/miraheze/mediawiki [01:01:17] [02mediawiki] 07dependabot[bot] commented on pull request 03#5665: Bump extensions/BlogPage from `0b24c44` to `ba8413d` - 13https://github.com/miraheze/mediawiki/pull/5665#issuecomment-1066231091 [01:01:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CheckUser-7b1eb4b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/61efd7a0a1dc [01:01:20] [02miraheze/mediawiki] 07dependabot[bot] 0361efd7a - Bump extensions/CheckUser from `c147485` to `7b1eb4b` [01:01:22] [02mediawiki] 07dependabot[bot] opened pull request 03#5762: Bump extensions/CheckUser from `c147485` to `7b1eb4b` - 13https://github.com/miraheze/mediawiki/pull/5762 [01:01:23] [02mediawiki] 07dependabot[bot] closed pull request 03#5665: Bump extensions/BlogPage from `0b24c44` to `ba8413d` - 13https://github.com/miraheze/mediawiki/pull/5665 [01:01:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CheckUser-7b1eb4b - 13https://github.com/miraheze/mediawiki [01:01:26] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/BlogPage-ba8413d [01:01:28] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/BlogPage-ba8413d - 13https://github.com/miraheze/mediawiki [01:01:29] [02mediawiki] 07dependabot[bot] commented on pull request 03#5740: Bump extensions/CheckUser from `c147485` to `a00cdce` - 13https://github.com/miraheze/mediawiki/pull/5740#issuecomment-1066231115 [01:01:31] [02mediawiki] 07dependabot[bot] closed pull request 03#5740: Bump extensions/CheckUser from `c147485` to `a00cdce` - 13https://github.com/miraheze/mediawiki/pull/5740 [01:01:32] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CheckUser-a00cdce [01:01:34] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CheckUser-a00cdce - 13https://github.com/miraheze/mediawiki [01:01:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/LoginNotify-a5d3ced [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4ea771bafbe4 [01:01:37] [02miraheze/mediawiki] 07dependabot[bot] 034ea771b - Bump extensions/LoginNotify from `03a3517` to `a5d3ced` [01:01:38] [02mediawiki] 07dependabot[bot] opened pull request 03#5763: Bump extensions/LoginNotify from `03a3517` to `a5d3ced` - 13https://github.com/miraheze/mediawiki/pull/5763 [01:01:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/LoginNotify-a5d3ced - 13https://github.com/miraheze/mediawiki [01:01:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AbuseFilter-bc22062 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d761fbdaf6e4 [01:01:43] [02miraheze/mediawiki] 07dependabot[bot] 03d761fbd - Bump extensions/AbuseFilter from `53e6c3e` to `bc22062` [01:01:44] [02mediawiki] 07dependabot[bot] opened pull request 03#5764: Bump extensions/AbuseFilter from `53e6c3e` to `bc22062` - 13https://github.com/miraheze/mediawiki/pull/5764 [01:01:46] [02mediawiki] 07dependabot[bot] commented on pull request 03#5669: Bump extensions/LoginNotify from `03a3517` to `71eb21b` - 13https://github.com/miraheze/mediawiki/pull/5669#issuecomment-1066231148 [01:01:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-bc22062 - 13https://github.com/miraheze/mediawiki [01:01:49] [02mediawiki] 07dependabot[bot] closed pull request 03#5669: Bump extensions/LoginNotify from `03a3517` to `71eb21b` - 13https://github.com/miraheze/mediawiki/pull/5669 [01:01:50] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/LoginNotify-71eb21b - 13https://github.com/miraheze/mediawiki [01:01:52] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/LoginNotify-71eb21b [01:01:53] [02mediawiki] 07dependabot[bot] commented on pull request 03#5721: Bump extensions/AbuseFilter from `53e6c3e` to `dfad58c` - 13https://github.com/miraheze/mediawiki/pull/5721#issuecomment-1066231163 [01:01:55] [02mediawiki] 07dependabot[bot] closed pull request 03#5721: Bump extensions/AbuseFilter from `53e6c3e` to `dfad58c` - 13https://github.com/miraheze/mediawiki/pull/5721 [01:01:56] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-2d7e772 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e425da242f6b [01:01:58] [02miraheze/mediawiki] 07dependabot[bot] 03e425da2 - Bump extensions/ConfirmEdit from `7848c32` to `2d7e772` [01:01:59] [02mediawiki] 07dependabot[bot] opened pull request 03#5765: Bump extensions/ConfirmEdit from `7848c32` to `2d7e772` - 13https://github.com/miraheze/mediawiki/pull/5765 [01:02:01] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-dfad58c [01:02:02] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-dfad58c - 13https://github.com/miraheze/mediawiki [01:02:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-2d7e772 - 13https://github.com/miraheze/mediawiki [01:02:05] [02mediawiki] 07dependabot[bot] commented on pull request 03#5736: Bump extensions/ConfirmEdit from `7848c32` to `22a569b` - 13https://github.com/miraheze/mediawiki/pull/5736#issuecomment-1066231192 [01:02:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ParserFunctions-10e1b57 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/55c2a3fb705f [01:02:08] [02miraheze/mediawiki] 07dependabot[bot] 0355c2a3f - Bump extensions/ParserFunctions from `4c8f825` to `10e1b57` [01:02:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5766: Bump extensions/ParserFunctions from `4c8f825` to `10e1b57` - 13https://github.com/miraheze/mediawiki/pull/5766 [01:02:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ParserFunctions-10e1b57 - 13https://github.com/miraheze/mediawiki [01:02:13] [02mediawiki] 07dependabot[bot] closed pull request 03#5736: Bump extensions/ConfirmEdit from `7848c32` to `22a569b` - 13https://github.com/miraheze/mediawiki/pull/5736 [01:02:14] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-22a569b - 13https://github.com/miraheze/mediawiki [01:02:16] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-22a569b [01:02:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TocTree-d6cd965 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/771f6d86ab7d [01:02:19] [02miraheze/mediawiki] 07dependabot[bot] 03771f6d8 - Bump extensions/TocTree from `dddbe61` to `d6cd965` [01:02:20] [02mediawiki] 07dependabot[bot] opened pull request 03#5767: Bump extensions/TocTree from `dddbe61` to `d6cd965` - 13https://github.com/miraheze/mediawiki/pull/5767 [01:02:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TocTree-d6cd965 - 13https://github.com/miraheze/mediawiki [01:02:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/NewestPages-7a2541b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b31b9dc3fb57 [01:02:25] [02miraheze/mediawiki] 07dependabot[bot] 03b31b9dc - Bump extensions/NewestPages from `13b8e23` to `7a2541b` [01:02:26] [02mediawiki] 07dependabot[bot] opened pull request 03#5768: Bump extensions/NewestPages from `13b8e23` to `7a2541b` - 13https://github.com/miraheze/mediawiki/pull/5768 [01:02:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/NewestPages-7a2541b - 13https://github.com/miraheze/mediawiki [01:02:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/3D-6b9d246 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cc9531f350db [01:02:31] [02miraheze/mediawiki] 07dependabot[bot] 03cc9531f - Bump extensions/3D from `cf97d54` to `6b9d246` [01:02:32] [02mediawiki] 07dependabot[bot] opened pull request 03#5769: Bump extensions/3D from `cf97d54` to `6b9d246` - 13https://github.com/miraheze/mediawiki/pull/5769 [01:02:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/3D-6b9d246 - 13https://github.com/miraheze/mediawiki [01:02:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Graph-8d983e7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f4d7e9d9eeec [01:02:37] [02miraheze/mediawiki] 07dependabot[bot] 03f4d7e9d - Bump extensions/Graph from `55dc205` to `8d983e7` [01:02:38] [02mediawiki] 07dependabot[bot] commented on pull request 03#5712: Bump extensions/3D from `cf97d54` to `a1beedf` - 13https://github.com/miraheze/mediawiki/pull/5712#issuecomment-1066231292 [01:02:40] [02mediawiki] 07dependabot[bot] opened pull request 03#5770: Bump extensions/Graph from `55dc205` to `8d983e7` - 13https://github.com/miraheze/mediawiki/pull/5770 [01:02:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#5761: Bump extensions/BlogPage from `0b24c44` to `cd63768` - 13https://github.com/miraheze/mediawiki/pull/5761 [01:02:43] [02mediawiki] 07dependabot[bot] closed pull request 03#5712: Bump extensions/3D from `cf97d54` to `a1beedf` - 13https://github.com/miraheze/mediawiki/pull/5712 [01:02:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#5762: Bump extensions/CheckUser from `c147485` to `7b1eb4b` - 13https://github.com/miraheze/mediawiki/pull/5762 [01:02:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Graph-8d983e7 - 13https://github.com/miraheze/mediawiki [01:02:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.76, 7.54, 8.30 [01:02:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.64, 9.08, 8.56 [01:02:47] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/3D-a1beedf [01:02:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.55, 9.75, 9.72 [01:02:49] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/3D-a1beedf - 13https://github.com/miraheze/mediawiki [01:02:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CiteThisPage-d86b730 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e001a0b9274f [01:02:52] [02miraheze/mediawiki] 07dependabot[bot] 03e001a0b - Bump extensions/CiteThisPage from `2972451` to `d86b730` [01:02:53] [02mediawiki] 07dependabot[bot] opened pull request 03#5771: Bump extensions/CiteThisPage from `2972451` to `d86b730` - 13https://github.com/miraheze/mediawiki/pull/5771 [01:02:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#5763: Bump extensions/LoginNotify from `03a3517` to `a5d3ced` - 13https://github.com/miraheze/mediawiki/pull/5763 [01:02:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-d86b730 - 13https://github.com/miraheze/mediawiki [01:02:58] [02mediawiki] 07dependabot[bot] commented on pull request 03#5744: Bump extensions/CiteThisPage from `2972451` to `2a93783` - 13https://github.com/miraheze/mediawiki/pull/5744#issuecomment-1066231358 [01:02:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-7269c88 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0bb225f64003 [01:03:01] [02miraheze/mediawiki] 07dependabot[bot] 030bb225f - Bump extensions/TimedMediaHandler from `4899bda` to `7269c88` [01:03:02] [02mediawiki] 07dependabot[bot] closed pull request 03#5744: Bump extensions/CiteThisPage from `2972451` to `2a93783` - 13https://github.com/miraheze/mediawiki/pull/5744 [01:03:04] [02mediawiki] 07dependabot[bot] opened pull request 03#5772: Bump extensions/TimedMediaHandler from `4899bda` to `7269c88` - 13https://github.com/miraheze/mediawiki/pull/5772 [01:03:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-7269c88 - 13https://github.com/miraheze/mediawiki [01:03:07] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-2a93783 - 13https://github.com/miraheze/mediawiki [01:03:08] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-2a93783 [01:03:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#5767: Bump extensions/TocTree from `dddbe61` to `d6cd965` - 13https://github.com/miraheze/mediawiki/pull/5767 [01:03:11] [02mediawiki] 07dependabot[bot] commented on pull request 03#5741: Bump extensions/TimedMediaHandler from `4899bda` to `68a4e38` - 13https://github.com/miraheze/mediawiki/pull/5741#issuecomment-1066231383 [01:03:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5766: Bump extensions/ParserFunctions from `4c8f825` to `10e1b57` - 13https://github.com/miraheze/mediawiki/pull/5766 [01:03:14] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.86, 5.47, 4.58 [01:03:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#5768: Bump extensions/NewestPages from `13b8e23` to `7a2541b` - 13https://github.com/miraheze/mediawiki/pull/5768 [01:03:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-9a5bef2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2f12ee6c2597 [01:03:17] [02miraheze/mediawiki] 07dependabot[bot] 032f12ee6 - Bump extensions/MirahezeMagic from `76c1e10` to `9a5bef2` [01:03:19] [02mediawiki] 07dependabot[bot] closed pull request 03#5741: Bump extensions/TimedMediaHandler from `4899bda` to `68a4e38` - 13https://github.com/miraheze/mediawiki/pull/5741 [01:03:20] [02mediawiki] 07dependabot[bot] opened pull request 03#5773: Bump extensions/MirahezeMagic from `76c1e10` to `9a5bef2` - 13https://github.com/miraheze/mediawiki/pull/5773 [01:03:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#5765: Bump extensions/ConfirmEdit from `7848c32` to `2d7e772` - 13https://github.com/miraheze/mediawiki/pull/5765 [01:03:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.41, 3.28, 3.10 [01:03:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-9a5bef2 - 13https://github.com/miraheze/mediawiki [01:03:25] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-68a4e38 - 13https://github.com/miraheze/mediawiki [01:03:26] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-68a4e38 [01:03:28] [02mediawiki] 07dependabot[bot] commented on pull request 03#5715: Bump extensions/MirahezeMagic from `76c1e10` to `2b3cfd5` - 13https://github.com/miraheze/mediawiki/pull/5715#issuecomment-1066231405 [01:03:29] [02mediawiki] 07dependabot[bot] closed pull request 03#5715: Bump extensions/MirahezeMagic from `76c1e10` to `2b3cfd5` - 13https://github.com/miraheze/mediawiki/pull/5715 [01:03:31] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-2b3cfd5 - 13https://github.com/miraheze/mediawiki [01:03:32] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-2b3cfd5 [01:03:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CategoryTree-e6024ec [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/360a0972d3a0 [01:03:35] [02miraheze/mediawiki] 07dependabot[bot] 03360a097 - Bump extensions/CategoryTree from `2f11e64` to `e6024ec` [01:03:37] [02mediawiki] 07dependabot[bot] opened pull request 03#5774: Bump extensions/CategoryTree from `2f11e64` to `e6024ec` - 13https://github.com/miraheze/mediawiki/pull/5774 [01:03:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CategoryTree-e6024ec - 13https://github.com/miraheze/mediawiki [01:03:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5764: Bump extensions/AbuseFilter from `53e6c3e` to `bc22062` - 13https://github.com/miraheze/mediawiki/pull/5764 [01:03:41] [02mediawiki] 07dependabot[bot] commented on pull request 03#5688: Bump extensions/CategoryTree from `2f11e64` to `e3fcd94` - 13https://github.com/miraheze/mediawiki/pull/5688#issuecomment-1066231438 [01:03:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.80, 3.20 [01:03:43] [02mediawiki] 07dependabot[bot] closed pull request 03#5688: Bump extensions/CategoryTree from `2f11e64` to `e3fcd94` - 13https://github.com/miraheze/mediawiki/pull/5688 [01:03:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ProofreadPage-720790c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9b9433b88317 [01:03:46] [02miraheze/mediawiki] 07dependabot[bot] 039b9433b - Bump extensions/ProofreadPage from `866ad8a` to `720790c` [01:03:47] [02mediawiki] 07dependabot[bot] opened pull request 03#5775: Bump extensions/ProofreadPage from `866ad8a` to `720790c` - 13https://github.com/miraheze/mediawiki/pull/5775 [01:03:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.76, 9.73, 9.38 [01:03:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-720790c - 13https://github.com/miraheze/mediawiki [01:03:50] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CategoryTree-e3fcd94 [01:03:52] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CategoryTree-e3fcd94 - 13https://github.com/miraheze/mediawiki [01:03:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AntiSpoof-0883351 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/356cbbbc5602 [01:03:55] [02miraheze/mediawiki] 07dependabot[bot] 03356cbbb - Bump extensions/AntiSpoof from `44f4504` to `0883351` [01:03:56] [02mediawiki] 07dependabot[bot] commented on pull request 03#5694: Bump extensions/ProofreadPage from `866ad8a` to `0d750a9` - 13https://github.com/miraheze/mediawiki/pull/5694#issuecomment-1066231476 [01:03:58] [02mediawiki] 07dependabot[bot] opened pull request 03#5776: Bump extensions/AntiSpoof from `44f4504` to `0883351` - 13https://github.com/miraheze/mediawiki/pull/5776 [01:03:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-0883351 - 13https://github.com/miraheze/mediawiki [01:04:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.85, 3.25 [01:04:01] [02mediawiki] 07dependabot[bot] closed pull request 03#5694: Bump extensions/ProofreadPage from `866ad8a` to `0d750a9` - 13https://github.com/miraheze/mediawiki/pull/5694 [01:04:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#5769: Bump extensions/3D from `cf97d54` to `6b9d246` - 13https://github.com/miraheze/mediawiki/pull/5769 [01:04:04] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-0d750a9 - 13https://github.com/miraheze/mediawiki [01:04:05] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-0d750a9 [01:04:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#5770: Bump extensions/Graph from `55dc205` to `8d983e7` - 13https://github.com/miraheze/mediawiki/pull/5770 [01:04:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Flow-ca77b81 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7bf90f019ab4 [01:04:10] [02miraheze/mediawiki] 07dependabot[bot] 037bf90f0 - Bump extensions/Flow from `c4c9f1c` to `ca77b81` [01:04:11] [02mediawiki] 07dependabot[bot] opened pull request 03#5777: Bump extensions/Flow from `c4c9f1c` to `ca77b81` - 13https://github.com/miraheze/mediawiki/pull/5777 [01:04:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Flow-ca77b81 - 13https://github.com/miraheze/mediawiki [01:04:14] [02mediawiki] 07dependabot[bot] commented on pull request 03#5708: Bump extensions/Flow from `c4c9f1c` to `a92d2fc` - 13https://github.com/miraheze/mediawiki/pull/5708#issuecomment-1066231535 [01:04:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#5771: Bump extensions/CiteThisPage from `2972451` to `d86b730` - 13https://github.com/miraheze/mediawiki/pull/5771 [01:04:17] [02mediawiki] 07dependabot[bot] closed pull request 03#5708: Bump extensions/Flow from `c4c9f1c` to `a92d2fc` - 13https://github.com/miraheze/mediawiki/pull/5708 [01:04:19] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Flow-a92d2fc [01:04:20] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Flow-a92d2fc - 13https://github.com/miraheze/mediawiki [01:04:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-3739389 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8f30df5ea7b7 [01:04:23] [02miraheze/mediawiki] 07dependabot[bot] 038f30df5 - Bump skins/Cosmos from `182cf3d` to `3739389` [01:04:25] [02mediawiki] 07dependabot[bot] opened pull request 03#5778: Bump skins/Cosmos from `182cf3d` to `3739389` - 13https://github.com/miraheze/mediawiki/pull/5778 [01:04:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Cosmos-3739389 - 13https://github.com/miraheze/mediawiki [01:04:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageImages-dd80ad2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5fccd97b7e90 [01:04:29] [02miraheze/mediawiki] 07dependabot[bot] 035fccd97 - Bump extensions/PageImages from `99bd702` to `dd80ad2` [01:04:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#5773: Bump extensions/MirahezeMagic from `76c1e10` to `9a5bef2` - 13https://github.com/miraheze/mediawiki/pull/5773 [01:04:32] [02mediawiki] 07dependabot[bot] opened pull request 03#5779: Bump extensions/PageImages from `99bd702` to `dd80ad2` - 13https://github.com/miraheze/mediawiki/pull/5779 [01:04:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5773: Bump extensions/MirahezeMagic from `76c1e10` to `9a5bef2` - 13https://github.com/miraheze/mediawiki/pull/5773 [01:04:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#5773: Bump extensions/MirahezeMagic from `76c1e10` to `9a5bef2` - 13https://github.com/miraheze/mediawiki/pull/5773 [01:04:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5772: Bump extensions/TimedMediaHandler from `4899bda` to `7269c88` - 13https://github.com/miraheze/mediawiki/pull/5772 [01:04:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageImages-dd80ad2 - 13https://github.com/miraheze/mediawiki [01:04:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5774: Bump extensions/CategoryTree from `2f11e64` to `e6024ec` - 13https://github.com/miraheze/mediawiki/pull/5774 [01:04:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#5776: Bump extensions/AntiSpoof from `44f4504` to `0883351` - 13https://github.com/miraheze/mediawiki/pull/5776 [01:04:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#5777: Bump extensions/Flow from `c4c9f1c` to `ca77b81` - 13https://github.com/miraheze/mediawiki/pull/5777 [01:04:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiEditor-cfe8d33 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9fdd0194a02d [01:04:46] [02miraheze/mediawiki] 07dependabot[bot] 039fdd019 - Bump extensions/WikiEditor from `6cde9e7` to `cfe8d33` [01:04:47] [02mediawiki] 07dependabot[bot] opened pull request 03#5780: Bump extensions/WikiEditor from `6cde9e7` to `cfe8d33` - 13https://github.com/miraheze/mediawiki/pull/5780 [01:04:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5775: Bump extensions/ProofreadPage from `866ad8a` to `720790c` - 13https://github.com/miraheze/mediawiki/pull/5775 [01:04:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiEditor-cfe8d33 - 13https://github.com/miraheze/mediawiki [01:04:52] [02mediawiki] 07dependabot[bot] commented on pull request 03#5696: Bump extensions/WikiEditor from `6cde9e7` to `69503e0` - 13https://github.com/miraheze/mediawiki/pull/5696#issuecomment-1066231676 [01:04:53] [02mediawiki] 07dependabot[bot] closed pull request 03#5696: Bump extensions/WikiEditor from `6cde9e7` to `69503e0` - 13https://github.com/miraheze/mediawiki/pull/5696 [01:04:55] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/WikiEditor-69503e0 - 13https://github.com/miraheze/mediawiki [01:04:57] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/WikiEditor-69503e0 [01:04:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageTriage-ba874b8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/67ab76e77f38 [01:04:59] [02miraheze/mediawiki] 07dependabot[bot] 0367ab76e - Bump extensions/PageTriage from `914c4b1` to `ba874b8` [01:05:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5781: Bump extensions/PageTriage from `914c4b1` to `ba874b8` - 13https://github.com/miraheze/mediawiki/pull/5781 [01:05:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageTriage-ba874b8 - 13https://github.com/miraheze/mediawiki [01:05:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/QuizGame-a9fb944 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/de14ee9cc2ae [01:05:05] [02miraheze/mediawiki] 07dependabot[bot] 03de14ee9 - Bump extensions/QuizGame from `9ea8dfa` to `a9fb944` [01:05:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5782: Bump extensions/QuizGame from `9ea8dfa` to `a9fb944` - 13https://github.com/miraheze/mediawiki/pull/5782 [01:05:08] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.07, 4.95, 4.49 [01:05:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/QuizGame-a9fb944 - 13https://github.com/miraheze/mediawiki [01:05:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#5779: Bump extensions/PageImages from `99bd702` to `dd80ad2` - 13https://github.com/miraheze/mediawiki/pull/5779 [01:05:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#5778: Bump skins/Cosmos from `182cf3d` to `3739389` - 13https://github.com/miraheze/mediawiki/pull/5778 [01:05:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/OATHAuth-2ca2ed1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/27a91b24be43 [01:05:14] [02miraheze/mediawiki] 07dependabot[bot] 0327a91b2 - Bump extensions/OATHAuth from `5dfe3dd` to `2ca2ed1` [01:05:16] [02mediawiki] 07dependabot[bot] commented on pull request 03#5750: Bump extensions/QuizGame from `9ea8dfa` to `fc3c56a` - 13https://github.com/miraheze/mediawiki/pull/5750#issuecomment-1066231767 [01:05:17] [02mediawiki] 07dependabot[bot] opened pull request 03#5783: Bump extensions/OATHAuth from `5dfe3dd` to `2ca2ed1` - 13https://github.com/miraheze/mediawiki/pull/5783 [01:05:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/OATHAuth-2ca2ed1 - 13https://github.com/miraheze/mediawiki [01:05:20] [02mediawiki] 07dependabot[bot] closed pull request 03#5750: Bump extensions/QuizGame from `9ea8dfa` to `fc3c56a` - 13https://github.com/miraheze/mediawiki/pull/5750 [01:05:22] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/QuizGame-fc3c56a [01:05:23] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/QuizGame-fc3c56a - 13https://github.com/miraheze/mediawiki [01:05:25] [02mediawiki] 07dependabot[bot] commented on pull request 03#5723: Bump extensions/OATHAuth from `5dfe3dd` to `ec90172` - 13https://github.com/miraheze/mediawiki/pull/5723#issuecomment-1066231789 [01:05:26] [02mediawiki] 07dependabot[bot] closed pull request 03#5723: Bump extensions/OATHAuth from `5dfe3dd` to `ec90172` - 13https://github.com/miraheze/mediawiki/pull/5723 [01:05:28] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/OATHAuth-ec90172 - 13https://github.com/miraheze/mediawiki [01:05:29] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/OATHAuth-ec90172 [01:05:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-e6b25a7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7615fe45674a [01:05:32] [02miraheze/mediawiki] 07dependabot[bot] 037615fe4 - Bump extensions/TemplateSandbox from `937f237` to `e6b25a7` [01:05:34] [02mediawiki] 07dependabot[bot] opened pull request 03#5784: Bump extensions/TemplateSandbox from `937f237` to `e6b25a7` - 13https://github.com/miraheze/mediawiki/pull/5784 [01:05:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-e6b25a7 - 13https://github.com/miraheze/mediawiki [01:05:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MobileFrontend-f479efc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6b7859fd86f0 [01:05:38] [02miraheze/mediawiki] 07dependabot[bot] 036b7859f - Bump extensions/MobileFrontend from `6222894` to `f479efc` [01:05:40] [02mediawiki] 07dependabot[bot] opened pull request 03#5785: Bump extensions/MobileFrontend from `6222894` to `f479efc` - 13https://github.com/miraheze/mediawiki/pull/5785 [01:05:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-f479efc - 13https://github.com/miraheze/mediawiki [01:05:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.38, 3.37, 3.12 [01:05:43] [02mediawiki] 07dependabot[bot] commented on pull request 03#5676: Bump extensions/MobileFrontend from `6222894` to `f1ff16a` - 13https://github.com/miraheze/mediawiki/pull/5676#issuecomment-1066231887 [01:05:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Comments-fbcdea7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/114f2dc3ab42 [01:05:46] [02miraheze/mediawiki] 07dependabot[bot] 03114f2dc - Bump extensions/Comments from `ad12023` to `fbcdea7` [01:05:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#5780: Bump extensions/WikiEditor from `6cde9e7` to `cfe8d33` - 13https://github.com/miraheze/mediawiki/pull/5780 [01:05:49] [02mediawiki] 07dependabot[bot] opened pull request 03#5786: Bump extensions/Comments from `ad12023` to `fbcdea7` - 13https://github.com/miraheze/mediawiki/pull/5786 [01:05:50] [02mediawiki] 07dependabot[bot] closed pull request 03#5676: Bump extensions/MobileFrontend from `6222894` to `f1ff16a` - 13https://github.com/miraheze/mediawiki/pull/5676 [01:05:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Comments-fbcdea7 - 13https://github.com/miraheze/mediawiki [01:05:53] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-f1ff16a - 13https://github.com/miraheze/mediawiki [01:05:55] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-f1ff16a [01:05:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#5781: Bump extensions/PageTriage from `914c4b1` to `ba874b8` - 13https://github.com/miraheze/mediawiki/pull/5781 [01:05:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/vendor-5739cd6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/01cd1320f4e3 [01:05:59] [02miraheze/mediawiki] 07dependabot[bot] 0301cd132 - Bump vendor from `56c2e8a` to `5739cd6` [01:06:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5787: Bump vendor from `56c2e8a` to `5739cd6` - 13https://github.com/miraheze/mediawiki/pull/5787 [01:06:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/vendor-5739cd6 - 13https://github.com/miraheze/mediawiki [01:06:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#5782: Bump extensions/QuizGame from `9ea8dfa` to `a9fb944` - 13https://github.com/miraheze/mediawiki/pull/5782 [01:06:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#5782: Bump extensions/QuizGame from `9ea8dfa` to `a9fb944` - 13https://github.com/miraheze/mediawiki/pull/5782 [01:06:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#5784: Bump extensions/TemplateSandbox from `937f237` to `e6b25a7` - 13https://github.com/miraheze/mediawiki/pull/5784 [01:06:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Echo-6abb2af [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f4f3101a2320 [01:06:10] [02miraheze/mediawiki] 07dependabot[bot] 03f4f3101 - Bump extensions/Echo from `7501007` to `6abb2af` [01:06:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#5783: Bump extensions/OATHAuth from `5dfe3dd` to `2ca2ed1` - 13https://github.com/miraheze/mediawiki/pull/5783 [01:06:13] [02mediawiki] 07dependabot[bot] opened pull request 03#5788: Bump extensions/Echo from `7501007` to `6abb2af` - 13https://github.com/miraheze/mediawiki/pull/5788 [01:06:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Echo-6abb2af - 13https://github.com/miraheze/mediawiki [01:06:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-2c5a4d1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b2055f4ba00e [01:06:17] [02miraheze/mediawiki] 07dependabot[bot] 03b2055f4 - Bump extensions/MultimediaViewer from `7d5845f` to `2c5a4d1` [01:06:19] [02mediawiki] 07dependabot[bot] opened pull request 03#5789: Bump extensions/MultimediaViewer from `7d5845f` to `2c5a4d1` - 13https://github.com/miraheze/mediawiki/pull/5789 [01:06:20] [02mediawiki] 07dependabot[bot] commented on pull request 03#5693: Bump extensions/Echo from `7501007` to `1fa5aaa` - 13https://github.com/miraheze/mediawiki/pull/5693#issuecomment-1066231979 [01:06:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-2c5a4d1 - 13https://github.com/miraheze/mediawiki [01:06:23] [02mediawiki] 07dependabot[bot] closed pull request 03#5693: Bump extensions/Echo from `7501007` to `1fa5aaa` - 13https://github.com/miraheze/mediawiki/pull/5693 [01:06:25] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Echo-1fa5aaa - 13https://github.com/miraheze/mediawiki [01:06:26] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Echo-1fa5aaa [01:06:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SimpleBlogPage-20e0923 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/317c1dc8b104 [01:06:29] [02miraheze/mediawiki] 07dependabot[bot] 03317c1dc - Bump extensions/SimpleBlogPage from `0bf4784` to `20e0923` [01:06:31] [02mediawiki] 07dependabot[bot] commented on pull request 03#5697: Bump extensions/MultimediaViewer from `7d5845f` to `58434db` - 13https://github.com/miraheze/mediawiki/pull/5697#issuecomment-1066231998 [01:06:32] [02mediawiki] 07dependabot[bot] opened pull request 03#5790: Bump extensions/SimpleBlogPage from `0bf4784` to `20e0923` - 13https://github.com/miraheze/mediawiki/pull/5790 [01:06:34] [02mediawiki] 07dependabot[bot] closed pull request 03#5697: Bump extensions/MultimediaViewer from `7d5845f` to `58434db` - 13https://github.com/miraheze/mediawiki/pull/5697 [01:06:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SimpleBlogPage-20e0923 - 13https://github.com/miraheze/mediawiki [01:06:37] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-58434db - 13https://github.com/miraheze/mediawiki [01:06:38] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-58434db [01:06:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CreateWiki-251e4e2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/473dc681e556 [01:06:41] [02miraheze/mediawiki] 07dependabot[bot] 03473dc68 - Bump extensions/CreateWiki from `8fa2186` to `251e4e2` [01:06:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5791: Bump extensions/CreateWiki from `8fa2186` to `251e4e2` - 13https://github.com/miraheze/mediawiki/pull/5791 [01:06:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-251e4e2 - 13https://github.com/miraheze/mediawiki [01:06:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5786: Bump extensions/Comments from `ad12023` to `fbcdea7` - 13https://github.com/miraheze/mediawiki/pull/5786 [01:06:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.25, 8.57, 8.57 [01:06:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SemanticMediaWiki-dd6dbe4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4232019853d2 [01:06:49] [02miraheze/mediawiki] 07dependabot[bot] 034232019 - Bump extensions/SemanticMediaWiki from `77164bc` to `dd6dbe4` [01:06:50] [02mediawiki] 07dependabot[bot] opened pull request 03#5792: Bump extensions/SemanticMediaWiki from `77164bc` to `dd6dbe4` - 13https://github.com/miraheze/mediawiki/pull/5792 [01:06:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SemanticMediaWiki-dd6dbe4 - 13https://github.com/miraheze/mediawiki [01:06:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#5785: Bump extensions/MobileFrontend from `6222894` to `f479efc` - 13https://github.com/miraheze/mediawiki/pull/5785 [01:06:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Citizen-c4b9abd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/531b4c34df17 [01:06:56] [02miraheze/mediawiki] 07dependabot[bot] 03531b4c3 - Bump skins/Citizen from `3e6a240` to `c4b9abd` [01:06:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Citizen-c4b9abd - 13https://github.com/miraheze/mediawiki [01:06:59] [02mediawiki] 07dependabot[bot] opened pull request 03#5793: Bump skins/Citizen from `3e6a240` to `c4b9abd` - 13https://github.com/miraheze/mediawiki/pull/5793 [01:07:01] [02mediawiki] 07dependabot[bot] commented on pull request 03#5751: Bump extensions/SemanticMediaWiki from `77164bc` to `7e41316` - 13https://github.com/miraheze/mediawiki/pull/5751#issuecomment-1066232088 [01:07:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cite-66dda67 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a4313e7dd789 [01:07:04] [02miraheze/mediawiki] 07dependabot[bot] 03a4313e7 - Bump extensions/Cite from `293a854` to `66dda67` [01:07:05] [02mediawiki] 07dependabot[bot] closed pull request 03#5751: Bump extensions/SemanticMediaWiki from `77164bc` to `7e41316` - 13https://github.com/miraheze/mediawiki/pull/5751 [01:07:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5794: Bump extensions/Cite from `293a854` to `66dda67` - 13https://github.com/miraheze/mediawiki/pull/5794 [01:07:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cite-66dda67 - 13https://github.com/miraheze/mediawiki [01:07:10] [02mediawiki] 07dependabot[bot] commented on pull request 03#5689: Bump skins/Citizen from `3e6a240` to `0c6f22b` - 13https://github.com/miraheze/mediawiki/pull/5689#issuecomment-1066232100 [01:07:12] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/SemanticMediaWiki-7e41316 - 13https://github.com/miraheze/mediawiki [01:07:13] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/SemanticMediaWiki-7e41316 [01:07:15] [02mediawiki] 07dependabot[bot] closed pull request 03#5689: Bump skins/Citizen from `3e6a240` to `0c6f22b` - 13https://github.com/miraheze/mediawiki/pull/5689 [01:07:16] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Citizen-0c6f22b [01:07:17] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Citizen-0c6f22b - 13https://github.com/miraheze/mediawiki [01:07:19] [02mediawiki] 07dependabot[bot] commented on pull request 03#5727: Bump extensions/Cite from `293a854` to `cda920e` - 13https://github.com/miraheze/mediawiki/pull/5727#issuecomment-1066232115 [01:07:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CommentStreams-5643c86 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c4ccca4257b8 [01:07:22] [02miraheze/mediawiki] 07dependabot[bot] 03c4ccca4 - Bump extensions/CommentStreams from `93ea9a7` to `5643c86` [01:07:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5795: Bump extensions/CommentStreams from `93ea9a7` to `5643c86` - 13https://github.com/miraheze/mediawiki/pull/5795 [01:07:25] [02mediawiki] 07dependabot[bot] closed pull request 03#5727: Bump extensions/Cite from `293a854` to `cda920e` - 13https://github.com/miraheze/mediawiki/pull/5727 [01:07:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CommentStreams-5643c86 - 13https://github.com/miraheze/mediawiki [01:07:28] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Cite-cda920e - 13https://github.com/miraheze/mediawiki [01:07:30] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Cite-cda920e [01:07:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-30f5d63 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0b8ff08a686c [01:07:33] [02miraheze/mediawiki] 07dependabot[bot] 030b8ff08 - Bump extensions/WikibaseQualityConstraints from `f25e5c1` to `30f5d63` [01:07:34] [02mediawiki] 07dependabot[bot] opened pull request 03#5796: Bump extensions/WikibaseQualityConstraints from `f25e5c1` to `30f5d63` - 13https://github.com/miraheze/mediawiki/pull/5796 [01:07:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-30f5d63 - 13https://github.com/miraheze/mediawiki [01:07:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5789: Bump extensions/MultimediaViewer from `7d5845f` to `2c5a4d1` - 13https://github.com/miraheze/mediawiki/pull/5789 [01:07:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#5790: Bump extensions/SimpleBlogPage from `0bf4784` to `20e0923` - 13https://github.com/miraheze/mediawiki/pull/5790 [01:07:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralNotice-7245cba [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a97704d7a377 [01:07:42] [02miraheze/mediawiki] 07dependabot[bot] 03a97704d - Bump extensions/CentralNotice from `f08dcd0` to `7245cba` [01:07:43] [02mediawiki] 07dependabot[bot] commented on pull request 03#5749: Bump extensions/WikibaseQualityConstraints from `f25e5c1` to `b46d206` - 13https://github.com/miraheze/mediawiki/pull/5749#issuecomment-1066232153 [01:07:45] [02mediawiki] 07dependabot[bot] opened pull request 03#5797: Bump extensions/CentralNotice from `f08dcd0` to `7245cba` - 13https://github.com/miraheze/mediawiki/pull/5797 [01:07:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.35, 9.30, 9.31 [01:07:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-7245cba - 13https://github.com/miraheze/mediawiki [01:07:48] [02mediawiki] 07dependabot[bot] closed pull request 03#5749: Bump extensions/WikibaseQualityConstraints from `f25e5c1` to `b46d206` - 13https://github.com/miraheze/mediawiki/pull/5749 [01:07:49] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-b46d206 - 13https://github.com/miraheze/mediawiki [01:07:51] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-b46d206 [01:07:52] [02mediawiki] 07dependabot[bot] commented on pull request 03#5685: Bump extensions/CentralNotice from `f08dcd0` to `37052df` - 13https://github.com/miraheze/mediawiki/pull/5685#issuecomment-1066232171 [01:07:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5788: Bump extensions/Echo from `7501007` to `6abb2af` - 13https://github.com/miraheze/mediawiki/pull/5788 [01:07:55] [02mediawiki] 07dependabot[bot] closed pull request 03#5685: Bump extensions/CentralNotice from `f08dcd0` to `37052df` - 13https://github.com/miraheze/mediawiki/pull/5685 [01:07:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TranslationNotifications-ca74bc5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1c421f116253 [01:07:58] [02miraheze/mediawiki] 07dependabot[bot] 031c421f1 - Bump extensions/TranslationNotifications from `7956b32` to `ca74bc5` [01:08:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5788: Bump extensions/Echo from `7501007` to `6abb2af` - 13https://github.com/miraheze/mediawiki/pull/5788 [01:08:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.87, 3.39 [01:08:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5798: Bump extensions/TranslationNotifications from `7956b32` to `ca74bc5` - 13https://github.com/miraheze/mediawiki/pull/5798 [01:08:03] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-37052df - 13https://github.com/miraheze/mediawiki [01:08:04] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-37052df [01:08:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#5793: Bump skins/Citizen from `3e6a240` to `c4b9abd` - 13https://github.com/miraheze/mediawiki/pull/5793 [01:08:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#5791: Bump extensions/CreateWiki from `8fa2186` to `251e4e2` - 13https://github.com/miraheze/mediawiki/pull/5791 [01:08:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TranslationNotifications-ca74bc5 - 13https://github.com/miraheze/mediawiki [01:08:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#5791: Bump extensions/CreateWiki from `8fa2186` to `251e4e2` - 13https://github.com/miraheze/mediawiki/pull/5791 [01:08:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#5793: Bump skins/Citizen from `3e6a240` to `c4b9abd` - 13https://github.com/miraheze/mediawiki/pull/5793 [01:08:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5791: Bump extensions/CreateWiki from `8fa2186` to `251e4e2` - 13https://github.com/miraheze/mediawiki/pull/5791 [01:08:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#5793: Bump skins/Citizen from `3e6a240` to `c4b9abd` - 13https://github.com/miraheze/mediawiki/pull/5793 [01:08:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#5791: Bump extensions/CreateWiki from `8fa2186` to `251e4e2` - 13https://github.com/miraheze/mediawiki/pull/5791 [01:08:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/JsonConfig-629b30c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/36a19a521c5c [01:08:19] [02miraheze/mediawiki] 07dependabot[bot] 0336a19a5 - Bump extensions/JsonConfig from `fadfd5e` to `629b30c` [01:08:21] [02mediawiki] 07dependabot[bot] opened pull request 03#5799: Bump extensions/JsonConfig from `fadfd5e` to `629b30c` - 13https://github.com/miraheze/mediawiki/pull/5799 [01:08:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/JsonConfig-629b30c - 13https://github.com/miraheze/mediawiki [01:08:24] [02mediawiki] 07dependabot[bot] commented on pull request 03#5714: Bump extensions/JsonConfig from `fadfd5e` to `48bfd09` - 13https://github.com/miraheze/mediawiki/pull/5714#issuecomment-1066232206 [01:08:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#5792: Bump extensions/SemanticMediaWiki from `77164bc` to `dd6dbe4` - 13https://github.com/miraheze/mediawiki/pull/5792 [01:08:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/MinervaNeue-3360981 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7fe14e1b0e9e [01:08:28] [02miraheze/mediawiki] 07dependabot[bot] 037fe14e1 - Bump skins/MinervaNeue from `e13fe16` to `3360981` [01:08:30] [02mediawiki] 07dependabot[bot] opened pull request 03#5800: Bump skins/MinervaNeue from `e13fe16` to `3360981` - 13https://github.com/miraheze/mediawiki/pull/5800 [01:08:31] [02mediawiki] 07dependabot[bot] closed pull request 03#5714: Bump extensions/JsonConfig from `fadfd5e` to `48bfd09` - 13https://github.com/miraheze/mediawiki/pull/5714 [01:08:33] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/JsonConfig-48bfd09 - 13https://github.com/miraheze/mediawiki [01:08:34] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/JsonConfig-48bfd09 [01:08:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-3360981 - 13https://github.com/miraheze/mediawiki [01:08:37] [02mediawiki] 07dependabot[bot] commented on pull request 03#5711: Bump skins/MinervaNeue from `e13fe16` to `819dae6` - 13https://github.com/miraheze/mediawiki/pull/5711#issuecomment-1066232227 [01:08:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/StandardDialogs-3a4b06d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b1e4d54f56f3 [01:08:40] [02miraheze/mediawiki] 07dependabot[bot] 03b1e4d54 - Bump extensions/StandardDialogs from `89795dc` to `3a4b06d` [01:08:42] [02mediawiki] 07dependabot[bot] opened pull request 03#5801: Bump extensions/StandardDialogs from `89795dc` to `3a4b06d` - 13https://github.com/miraheze/mediawiki/pull/5801 [01:08:43] [02mediawiki] 07dependabot[bot] closed pull request 03#5711: Bump skins/MinervaNeue from `e13fe16` to `819dae6` - 13https://github.com/miraheze/mediawiki/pull/5711 [01:08:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/StandardDialogs-3a4b06d - 13https://github.com/miraheze/mediawiki [01:08:46] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-819dae6 [01:08:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 8.29, 8.34 [01:08:48] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-819dae6 - 13https://github.com/miraheze/mediawiki [01:08:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiDiscover-348f5af [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f478e4401110 [01:08:51] [02miraheze/mediawiki] 07dependabot[bot] 03f478e44 - Bump extensions/WikiDiscover from `87282ea` to `348f5af` [01:08:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5802: Bump extensions/WikiDiscover from `87282ea` to `348f5af` - 13https://github.com/miraheze/mediawiki/pull/5802 [01:08:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiDiscover-348f5af - 13https://github.com/miraheze/mediawiki [01:08:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#5794: Bump extensions/Cite from `293a854` to `66dda67` - 13https://github.com/miraheze/mediawiki/pull/5794 [01:08:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DiscussionTools-c506739 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/299dfe24e5d9 [01:08:58] [02miraheze/mediawiki] 07dependabot[bot] 03299dfe2 - Bump extensions/DiscussionTools from `4c0f69d` to `c506739` [01:09:00] [02mediawiki] 07dependabot[bot] opened pull request 03#5803: Bump extensions/DiscussionTools from `4c0f69d` to `c506739` - 13https://github.com/miraheze/mediawiki/pull/5803 [01:09:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 10.13, 6.40, 5.17 [01:09:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-c506739 - 13https://github.com/miraheze/mediawiki [01:09:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5795: Bump extensions/CommentStreams from `93ea9a7` to `5643c86` - 13https://github.com/miraheze/mediawiki/pull/5795 [01:09:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#5796: Bump extensions/WikibaseQualityConstraints from `f25e5c1` to `30f5d63` - 13https://github.com/miraheze/mediawiki/pull/5796 [01:09:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#5799: Bump extensions/JsonConfig from `fadfd5e` to `629b30c` - 13https://github.com/miraheze/mediawiki/pull/5799 [01:09:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#5797: Bump extensions/CentralNotice from `f08dcd0` to `7245cba` - 13https://github.com/miraheze/mediawiki/pull/5797 [01:09:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateData-a7fa94d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d2b97abb45b0 [01:09:10] [02miraheze/mediawiki] 07dependabot[bot] 03d2b97ab - Bump extensions/TemplateData from `3489558` to `a7fa94d` [01:09:12] [02mediawiki] 07dependabot[bot] opened pull request 03#5804: Bump extensions/TemplateData from `3489558` to `a7fa94d` - 13https://github.com/miraheze/mediawiki/pull/5804 [01:09:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5800: Bump skins/MinervaNeue from `e13fe16` to `3360981` - 13https://github.com/miraheze/mediawiki/pull/5800 [01:09:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateData-a7fa94d - 13https://github.com/miraheze/mediawiki [01:09:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#5798: Bump extensions/TranslationNotifications from `7956b32` to `ca74bc5` - 13https://github.com/miraheze/mediawiki/pull/5798 [01:09:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/cldr-e23652a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6d66006b859d [01:09:19] [02miraheze/mediawiki] 07dependabot[bot] 036d66006 - Bump extensions/cldr from `dacffff` to `e23652a` [01:09:21] [02mediawiki] 07dependabot[bot] commented on pull request 03#5698: Bump extensions/TemplateData from `3489558` to `492cca1` - 13https://github.com/miraheze/mediawiki/pull/5698#issuecomment-1066232350 [01:09:22] [02mediawiki] 07dependabot[bot] opened pull request 03#5805: Bump extensions/cldr from `dacffff` to `e23652a` - 13https://github.com/miraheze/mediawiki/pull/5805 [01:09:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/cldr-e23652a - 13https://github.com/miraheze/mediawiki [01:09:25] [02mediawiki] 07dependabot[bot] closed pull request 03#5698: Bump extensions/TemplateData from `3489558` to `492cca1` - 13https://github.com/miraheze/mediawiki/pull/5698 [01:09:27] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/TemplateData-492cca1 [01:09:28] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/TemplateData-492cca1 - 13https://github.com/miraheze/mediawiki [01:09:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ReplaceText-a3211d6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e304f61f7462 [01:09:31] [02miraheze/mediawiki] 07dependabot[bot] 03e304f61 - Bump extensions/ReplaceText from `dbd95b2` to `a3211d6` [01:09:33] [02mediawiki] 07dependabot[bot] opened pull request 03#5806: Bump extensions/ReplaceText from `dbd95b2` to `a3211d6` - 13https://github.com/miraheze/mediawiki/pull/5806 [01:09:34] [02mediawiki] 07dependabot[bot] commented on pull request 03#5719: Bump extensions/cldr from `dacffff` to `3499e3f` - 13https://github.com/miraheze/mediawiki/pull/5719#issuecomment-1066232367 [01:09:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ReplaceText-a3211d6 - 13https://github.com/miraheze/mediawiki [01:09:37] [02mediawiki] 07dependabot[bot] closed pull request 03#5719: Bump extensions/cldr from `dacffff` to `3499e3f` - 13https://github.com/miraheze/mediawiki/pull/5719 [01:09:39] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/cldr-3499e3f [01:09:40] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/cldr-3499e3f - 13https://github.com/miraheze/mediawiki [01:09:42] [02mediawiki] 07dependabot[bot] commented on pull request 03#5720: Bump extensions/ReplaceText from `dbd95b2` to `98eefa2` - 13https://github.com/miraheze/mediawiki/pull/5720#issuecomment-1066232384 [01:09:43] [02mediawiki] 07dependabot[bot] closed pull request 03#5720: Bump extensions/ReplaceText from `dbd95b2` to `98eefa2` - 13https://github.com/miraheze/mediawiki/pull/5720 [01:09:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ShortDescription-34f950b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8163548b4940 [01:09:46] [02miraheze/mediawiki] 07dependabot[bot] 038163548 - Bump extensions/ShortDescription from `26b10a9` to `34f950b` [01:09:48] [02mediawiki] 07dependabot[bot] opened pull request 03#5807: Bump extensions/ShortDescription from `26b10a9` to `34f950b` - 13https://github.com/miraheze/mediawiki/pull/5807 [01:09:49] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ReplaceText-98eefa2 [01:09:51] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ReplaceText-98eefa2 - 13https://github.com/miraheze/mediawiki [01:09:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#5801: Bump extensions/StandardDialogs from `89795dc` to `3a4b06d` - 13https://github.com/miraheze/mediawiki/pull/5801 [01:09:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ShortDescription-34f950b - 13https://github.com/miraheze/mediawiki [01:09:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SocialProfile-c59c238 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f8e2446bcd61 [01:09:57] [02miraheze/mediawiki] 07dependabot[bot] 03f8e2446 - Bump extensions/SocialProfile from `f1d6a5b` to `c59c238` [01:09:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-c59c238 - 13https://github.com/miraheze/mediawiki [01:10:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5802: Bump extensions/WikiDiscover from `87282ea` to `348f5af` - 13https://github.com/miraheze/mediawiki/pull/5802 [01:10:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.38, 3.38, 3.27 [01:10:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5808: Bump extensions/SocialProfile from `f1d6a5b` to `c59c238` - 13https://github.com/miraheze/mediawiki/pull/5808 [01:10:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5802: Bump extensions/WikiDiscover from `87282ea` to `348f5af` - 13https://github.com/miraheze/mediawiki/pull/5802 [01:10:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#5802: Bump extensions/WikiDiscover from `87282ea` to `348f5af` - 13https://github.com/miraheze/mediawiki/pull/5802 [01:10:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Vector-0a762d0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/10c8d261ed2b [01:10:07] [02miraheze/mediawiki] 07dependabot[bot] 0310c8d26 - Bump skins/Vector from `28e5d64` to `0a762d0` [01:10:09] [02mediawiki] 07dependabot[bot] opened pull request 03#5809: Bump skins/Vector from `28e5d64` to `0a762d0` - 13https://github.com/miraheze/mediawiki/pull/5809 [01:10:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Vector-0a762d0 - 13https://github.com/miraheze/mediawiki [01:10:12] [02mediawiki] 07dependabot[bot] commented on pull request 03#5709: Bump extensions/SocialProfile from `f1d6a5b` to `0286d0e` - 13https://github.com/miraheze/mediawiki/pull/5709#issuecomment-1066232450 [01:10:13] [02mediawiki] 07dependabot[bot] closed pull request 03#5709: Bump extensions/SocialProfile from `f1d6a5b` to `0286d0e` - 13https://github.com/miraheze/mediawiki/pull/5709 [01:10:15] [02mediawiki] 07dependabot[bot] commented on pull request 03#5704: Bump skins/Vector from `28e5d64` to `0628ee1` - 13https://github.com/miraheze/mediawiki/pull/5704#issuecomment-1066232469 [01:10:16] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-0286d0e [01:10:18] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-0286d0e - 13https://github.com/miraheze/mediawiki [01:10:19] [02mediawiki] 07dependabot[bot] closed pull request 03#5704: Bump skins/Vector from `28e5d64` to `0628ee1` - 13https://github.com/miraheze/mediawiki/pull/5704 [01:10:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-381cd5d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8449f0419495 [01:10:22] [02miraheze/mediawiki] 07dependabot[bot] 038449f04 - Bump extensions/DynamicPageList3 from `db51e49` to `381cd5d` [01:10:24] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Vector-0628ee1 - 13https://github.com/miraheze/mediawiki [01:10:25] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Vector-0628ee1 [01:10:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#5803: Bump extensions/DiscussionTools from `4c0f69d` to `c506739` - 13https://github.com/miraheze/mediawiki/pull/5803 [01:10:28] [02mediawiki] 07dependabot[bot] opened pull request 03#5810: Bump extensions/DynamicPageList3 from `db51e49` to `381cd5d` - 13https://github.com/miraheze/mediawiki/pull/5810 [01:10:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-381cd5d - 13https://github.com/miraheze/mediawiki [01:10:31] [02mediawiki] 07dependabot[bot] commented on pull request 03#5686: Bump extensions/DynamicPageList3 from `db51e49` to `6d77c08` - 13https://github.com/miraheze/mediawiki/pull/5686#issuecomment-1066232509 [01:10:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/mw-ScratchBlocks4-2cd8da5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/477954244c77 [01:10:34] [02miraheze/mediawiki] 07dependabot[bot] 034779542 - Bump extensions/mw-ScratchBlocks4 from `dcaa20b` to `2cd8da5` [01:10:36] [02mediawiki] 07dependabot[bot] closed pull request 03#5686: Bump extensions/DynamicPageList3 from `db51e49` to `6d77c08` - 13https://github.com/miraheze/mediawiki/pull/5686 [01:10:37] [02mediawiki] 07dependabot[bot] opened pull request 03#5811: Bump extensions/mw-ScratchBlocks4 from `dcaa20b` to `2cd8da5` - 13https://github.com/miraheze/mediawiki/pull/5811 [01:10:39] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-6d77c08 - 13https://github.com/miraheze/mediawiki [01:10:40] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-6d77c08 [01:10:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/mw-ScratchBlocks4-2cd8da5 - 13https://github.com/miraheze/mediawiki [01:10:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#5806: Bump extensions/ReplaceText from `dbd95b2` to `a3211d6` - 13https://github.com/miraheze/mediawiki/pull/5806 [01:10:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#5804: Bump extensions/TemplateData from `3489558` to `a7fa94d` - 13https://github.com/miraheze/mediawiki/pull/5804 [01:10:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5805: Bump extensions/cldr from `dacffff` to `e23652a` - 13https://github.com/miraheze/mediawiki/pull/5805 [01:10:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#5805: Bump extensions/cldr from `dacffff` to `e23652a` - 13https://github.com/miraheze/mediawiki/pull/5805 [01:10:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Video-858fb24 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1fb8a464f8a1 [01:10:51] [02miraheze/mediawiki] 07dependabot[bot] 031fb8a46 - Bump extensions/Video from `b50585f` to `858fb24` [01:10:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5812: Bump extensions/Video from `b50585f` to `858fb24` - 13https://github.com/miraheze/mediawiki/pull/5812 [01:10:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Video-858fb24 - 13https://github.com/miraheze/mediawiki [01:10:55] [02mediawiki] 07dependabot[bot] commented on pull request 03#5748: Bump extensions/Video from `b50585f` to `7ee05b0` - 13https://github.com/miraheze/mediawiki/pull/5748#issuecomment-1066232566 [01:10:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SpamBlacklist-a5f9218 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b6b7ef84b22e [01:10:58] [02miraheze/mediawiki] 07dependabot[bot] 03b6b7ef8 - Bump extensions/SpamBlacklist from `7dfdfad` to `a5f9218` [01:11:00] [02mediawiki] 07dependabot[bot] closed pull request 03#5748: Bump extensions/Video from `b50585f` to `7ee05b0` - 13https://github.com/miraheze/mediawiki/pull/5748 [01:11:00] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.88, 5.83, 5.09 [01:11:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5813: Bump extensions/SpamBlacklist from `7dfdfad` to `a5f9218` - 13https://github.com/miraheze/mediawiki/pull/5813 [01:11:03] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Video-7ee05b0 [01:11:04] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Video-7ee05b0 - 13https://github.com/miraheze/mediawiki [01:11:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SpamBlacklist-a5f9218 - 13https://github.com/miraheze/mediawiki [01:11:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#5809: Bump skins/Vector from `28e5d64` to `0a762d0` - 13https://github.com/miraheze/mediawiki/pull/5809 [01:11:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#5807: Bump extensions/ShortDescription from `26b10a9` to `34f950b` - 13https://github.com/miraheze/mediawiki/pull/5807 [01:11:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#5808: Bump extensions/SocialProfile from `f1d6a5b` to `c59c238` - 13https://github.com/miraheze/mediawiki/pull/5808 [01:11:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#5808: Bump extensions/SocialProfile from `f1d6a5b` to `c59c238` - 13https://github.com/miraheze/mediawiki/pull/5808 [01:11:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-9ae8c76 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/dcd14a688835 [01:11:15] [02miraheze/mediawiki] 07dependabot[bot] 03dcd14a6 - Bump extensions/Translate from `411d06c` to `9ae8c76` [01:11:16] [02mediawiki] 07dependabot[bot] opened pull request 03#5814: Bump extensions/Translate from `411d06c` to `9ae8c76` - 13https://github.com/miraheze/mediawiki/pull/5814 [01:11:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-9ae8c76 - 13https://github.com/miraheze/mediawiki [01:11:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/LinkTitles-c0e5396 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/62ec3f099ef6 [01:11:21] [02miraheze/mediawiki] 07dependabot[bot] 0362ec3f0 - Bump extensions/LinkTitles from `4e868e3` to `c0e5396` [01:11:22] [02mediawiki] 07dependabot[bot] opened pull request 03#5815: Bump extensions/LinkTitles from `4e868e3` to `c0e5396` - 13https://github.com/miraheze/mediawiki/pull/5815 [01:11:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/LinkTitles-c0e5396 - 13https://github.com/miraheze/mediawiki [01:11:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#5810: Bump extensions/DynamicPageList3 from `db51e49` to `381cd5d` - 13https://github.com/miraheze/mediawiki/pull/5810 [01:11:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/VisualEditor-a768a68 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/61bb55b030f8 [01:11:28] [02miraheze/mediawiki] 07dependabot[bot] 0361bb55b - Bump extensions/VisualEditor from `40ee3ad` to `a768a68` [01:11:30] [02mediawiki] 07dependabot[bot] opened pull request 03#5816: Bump extensions/VisualEditor from `40ee3ad` to `a768a68` - 13https://github.com/miraheze/mediawiki/pull/5816 [01:11:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#5810: Bump extensions/DynamicPageList3 from `db51e49` to `381cd5d` - 13https://github.com/miraheze/mediawiki/pull/5810 [01:11:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5810: Bump extensions/DynamicPageList3 from `db51e49` to `381cd5d` - 13https://github.com/miraheze/mediawiki/pull/5810 [01:11:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-a768a68 - 13https://github.com/miraheze/mediawiki [01:11:36] [02mediawiki] 07dependabot[bot] commented on pull request 03#5737: Bump extensions/VisualEditor from `40ee3ad` to `371a81a` - 13https://github.com/miraheze/mediawiki/pull/5737#issuecomment-1066232676 [01:11:37] [02mediawiki] 07dependabot[bot] closed pull request 03#5737: Bump extensions/VisualEditor from `40ee3ad` to `371a81a` - 13https://github.com/miraheze/mediawiki/pull/5737 [01:11:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/apex-4b8e3ca [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/28a54b4bb8d1 [01:11:40] [02miraheze/mediawiki] 07dependabot[bot] 0328a54b4 - Bump skins/apex from `5448eaf` to `4b8e3ca` [01:11:42] [02mediawiki] 07dependabot[bot] opened pull request 03#5817: Bump skins/apex from `5448eaf` to `4b8e3ca` - 13https://github.com/miraheze/mediawiki/pull/5817 [01:11:43] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-371a81a [01:11:45] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-371a81a - 13https://github.com/miraheze/mediawiki [01:11:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/apex-4b8e3ca - 13https://github.com/miraheze/mediawiki [01:11:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PortableInfobox-c901aa9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a22c25b2afbc [01:11:49] [02miraheze/mediawiki] 07dependabot[bot] 03a22c25b - Bump extensions/PortableInfobox from `4df958b` to `c901aa9` [01:11:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5812: Bump extensions/Video from `b50585f` to `858fb24` - 13https://github.com/miraheze/mediawiki/pull/5812 [01:11:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5818: Bump extensions/PortableInfobox from `4df958b` to `c901aa9` - 13https://github.com/miraheze/mediawiki/pull/5818 [01:11:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-c901aa9 - 13https://github.com/miraheze/mediawiki [01:11:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-4b9c66c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6a9d11bdbaa2 [01:11:57] [02miraheze/mediawiki] 07dependabot[bot] 036a9d11b - Bump extensions/PageForms from `1d74db6` to `4b9c66c` [01:11:58] [02mediawiki] 07dependabot[bot] opened pull request 03#5819: Bump extensions/PageForms from `1d74db6` to `4b9c66c` - 13https://github.com/miraheze/mediawiki/pull/5819 [01:12:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-4b9c66c - 13https://github.com/miraheze/mediawiki [01:12:01] [02mediawiki] 07dependabot[bot] commented on pull request 03#5747: Bump extensions/PageForms from `1d74db6` to `16589e0` - 13https://github.com/miraheze/mediawiki/pull/5747#issuecomment-1066232761 [01:12:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5813: Bump extensions/SpamBlacklist from `7dfdfad` to `a5f9218` - 13https://github.com/miraheze/mediawiki/pull/5813 [01:12:04] [02mediawiki] 07dependabot[bot] closed pull request 03#5747: Bump extensions/PageForms from `1d74db6` to `16589e0` - 13https://github.com/miraheze/mediawiki/pull/5747 [01:12:06] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-16589e0 - 13https://github.com/miraheze/mediawiki [01:12:07] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-16589e0 [01:12:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Newsletter-851640e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8dc691d79095 [01:12:10] [02miraheze/mediawiki] 07dependabot[bot] 038dc691d - Bump extensions/Newsletter from `31d866c` to `851640e` [01:12:12] [02mediawiki] 07dependabot[bot] opened pull request 03#5820: Bump extensions/Newsletter from `31d866c` to `851640e` - 13https://github.com/miraheze/mediawiki/pull/5820 [01:12:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Newsletter-851640e - 13https://github.com/miraheze/mediawiki [01:12:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Kartographer-4c87d80 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/840796a6953f [01:12:16] [02miraheze/mediawiki] 07dependabot[bot] 03840796a - Bump extensions/Kartographer from `396d49b` to `4c87d80` [01:12:18] [02mediawiki] 07dependabot[bot] commented on pull request 03#5757: Bump extensions/Newsletter from `31d866c` to `e1cb886` - 13https://github.com/miraheze/mediawiki/pull/5757#issuecomment-1066232797 [01:12:19] [02mediawiki] 07dependabot[bot] opened pull request 03#5821: Bump extensions/Kartographer from `396d49b` to `4c87d80` - 13https://github.com/miraheze/mediawiki/pull/5821 [01:12:21] [02mediawiki] 07dependabot[bot] closed pull request 03#5757: Bump extensions/Newsletter from `31d866c` to `e1cb886` - 13https://github.com/miraheze/mediawiki/pull/5757 [01:12:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Kartographer-4c87d80 - 13https://github.com/miraheze/mediawiki [01:12:24] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Newsletter-e1cb886 [01:12:25] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Newsletter-e1cb886 - 13https://github.com/miraheze/mediawiki [01:12:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#5814: Bump extensions/Translate from `411d06c` to `9ae8c76` - 13https://github.com/miraheze/mediawiki/pull/5814 [01:12:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#5814: Bump extensions/Translate from `411d06c` to `9ae8c76` - 13https://github.com/miraheze/mediawiki/pull/5814 [01:12:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#5814: Bump extensions/Translate from `411d06c` to `9ae8c76` - 13https://github.com/miraheze/mediawiki/pull/5814 [01:12:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#5815: Bump extensions/LinkTitles from `4e868e3` to `c0e5396` - 13https://github.com/miraheze/mediawiki/pull/5815 [01:12:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5815: Bump extensions/LinkTitles from `4e868e3` to `c0e5396` - 13https://github.com/miraheze/mediawiki/pull/5815 [01:12:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-efbfbe8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d618553b6ac9 [01:12:36] [02miraheze/mediawiki] 07dependabot[bot] 03d618553 - Bump skins/Evelution from `732289b` to `efbfbe8` [01:12:37] [02mediawiki] 07dependabot[bot] opened pull request 03#5822: Bump skins/Evelution from `732289b` to `efbfbe8` - 13https://github.com/miraheze/mediawiki/pull/5822 [01:12:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-efbfbe8 - 13https://github.com/miraheze/mediawiki [01:12:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5817: Bump skins/apex from `5448eaf` to `4b8e3ca` - 13https://github.com/miraheze/mediawiki/pull/5817 [01:12:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-93a34c8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f24d12cc7ac5 [01:12:43] [02miraheze/mediawiki] 07dependabot[bot] 03f24d12c - Bump extensions/UniversalLanguageSelector from `32ee5f5` to `93a34c8` [01:12:45] [02mediawiki] 07dependabot[bot] opened pull request 03#5823: Bump extensions/UniversalLanguageSelector from `32ee5f5` to `93a34c8` - 13https://github.com/miraheze/mediawiki/pull/5823 [01:12:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-93a34c8 - 13https://github.com/miraheze/mediawiki [01:12:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.66, 8.07, 8.38 [01:12:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#5818: Bump extensions/PortableInfobox from `4df958b` to `c901aa9` - 13https://github.com/miraheze/mediawiki/pull/5818 [01:12:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5818: Bump extensions/PortableInfobox from `4df958b` to `c901aa9` - 13https://github.com/miraheze/mediawiki/pull/5818 [01:12:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5816: Bump extensions/VisualEditor from `40ee3ad` to `a768a68` - 13https://github.com/miraheze/mediawiki/pull/5816 [01:12:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Math-3586c44 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/90e3276d3e92 [01:12:54] [02miraheze/mediawiki] 07dependabot[bot] 0390e3276 - Bump extensions/Math from `e415034` to `3586c44` [01:12:55] [02mediawiki] 07dependabot[bot] opened pull request 03#5824: Bump extensions/Math from `e415034` to `3586c44` - 13https://github.com/miraheze/mediawiki/pull/5824 [01:12:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Math-3586c44 - 13https://github.com/miraheze/mediawiki [01:12:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#5821: Bump extensions/Kartographer from `396d49b` to `4c87d80` - 13https://github.com/miraheze/mediawiki/pull/5821 [01:13:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5819: Bump extensions/PageForms from `1d74db6` to `4b9c66c` - 13https://github.com/miraheze/mediawiki/pull/5819 [01:13:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#5819: Bump extensions/PageForms from `1d74db6` to `4b9c66c` - 13https://github.com/miraheze/mediawiki/pull/5819 [01:13:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-a6c1f0e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/36496e6e1bad [01:13:04] [02miraheze/mediawiki] 07dependabot[bot] 0336496e6 - Bump extensions/GlobalWatchlist from `4cff4b3` to `a6c1f0e` [01:13:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateStyles-eec8c6c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b591345075dd [01:13:07] [02miraheze/mediawiki] 07dependabot[bot] 03b591345 - Bump extensions/TemplateStyles from `b7e6c34` to `eec8c6c` [01:13:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-a6c1f0e - 13https://github.com/miraheze/mediawiki [01:13:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5825: Bump extensions/GlobalWatchlist from `4cff4b3` to `a6c1f0e` - 13https://github.com/miraheze/mediawiki/pull/5825 [01:13:12] [02mediawiki] 07dependabot[bot] opened pull request 03#5826: Bump extensions/TemplateStyles from `b7e6c34` to `eec8c6c` - 13https://github.com/miraheze/mediawiki/pull/5826 [01:13:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5820: Bump extensions/Newsletter from `31d866c` to `851640e` - 13https://github.com/miraheze/mediawiki/pull/5820 [01:13:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateStyles-eec8c6c - 13https://github.com/miraheze/mediawiki [01:13:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-3ead915 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a64676c741ff [01:13:18] [02miraheze/mediawiki] 07dependabot[bot] 03a64676c - Bump extensions/Cargo from `0f06913` to `3ead915` [01:13:19] [02mediawiki] 07dependabot[bot] opened pull request 03#5827: Bump extensions/Cargo from `0f06913` to `3ead915` - 13https://github.com/miraheze/mediawiki/pull/5827 [01:13:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-3ead915 - 13https://github.com/miraheze/mediawiki [01:13:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-6c98593 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/577763a9216f [01:13:24] [02miraheze/mediawiki] 07dependabot[bot] 03577763a - Bump extensions/ApprovedRevs from `bd12cb0` to `6c98593` [01:13:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-6c98593 - 13https://github.com/miraheze/mediawiki [01:13:27] [02mediawiki] 07dependabot[bot] opened pull request 03#5828: Bump extensions/ApprovedRevs from `bd12cb0` to `6c98593` - 13https://github.com/miraheze/mediawiki/pull/5828 [01:13:28] [02mediawiki] 07dependabot[bot] commented on pull request 03#5722: Bump extensions/Cargo from `0f06913` to `01f0790` - 13https://github.com/miraheze/mediawiki/pull/5722#issuecomment-1066233013 [01:13:30] [02mediawiki] 07dependabot[bot] closed pull request 03#5722: Bump extensions/Cargo from `0f06913` to `01f0790` - 13https://github.com/miraheze/mediawiki/pull/5722 [01:13:31] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-01f0790 - 13https://github.com/miraheze/mediawiki [01:13:33] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-01f0790 [01:13:34] [02mediawiki] 07dependabot[bot] opened pull request 03#5829: Bump extensions/ArticlePlaceholder from `f17427e` to `8d6e792` - 13https://github.com/miraheze/mediawiki/pull/5829 [01:13:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-8d6e792 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d77a278b788a [01:13:37] [02miraheze/mediawiki] 07dependabot[bot] 03d77a278 - Bump extensions/ArticlePlaceholder from `f17427e` to `8d6e792` [01:13:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-8d6e792 - 13https://github.com/miraheze/mediawiki [01:13:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5824: Bump extensions/Math from `e415034` to `3586c44` - 13https://github.com/miraheze/mediawiki/pull/5824 [01:13:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#5823: Bump extensions/UniversalLanguageSelector from `32ee5f5` to `93a34c8` - 13https://github.com/miraheze/mediawiki/pull/5823 [01:13:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.53, 3.21 [01:13:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#5823: Bump extensions/UniversalLanguageSelector from `32ee5f5` to `93a34c8` - 13https://github.com/miraheze/mediawiki/pull/5823 [01:13:45] [02mediawiki] 07dependabot[bot] commented on pull request 03#5718: Bump extensions/ArticlePlaceholder from `f17427e` to `bc719fb` - 13https://github.com/miraheze/mediawiki/pull/5718#issuecomment-1066233078 [01:13:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DataDump-b9d7aaa [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c3330856c02b [01:13:48] [02miraheze/mediawiki] 07dependabot[bot] 03c333085 - Bump extensions/DataDump from `88351b6` to `b9d7aaa` [01:13:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5822: Bump skins/Evelution from `732289b` to `efbfbe8` - 13https://github.com/miraheze/mediawiki/pull/5822 [01:13:51] [02mediawiki] 07dependabot[bot] opened pull request 03#5830: Bump extensions/DataDump from `88351b6` to `b9d7aaa` - 13https://github.com/miraheze/mediawiki/pull/5830 [01:13:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#5823: Bump extensions/UniversalLanguageSelector from `32ee5f5` to `93a34c8` - 13https://github.com/miraheze/mediawiki/pull/5823 [01:13:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5822: Bump skins/Evelution from `732289b` to `efbfbe8` - 13https://github.com/miraheze/mediawiki/pull/5822 [01:13:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DataDump-b9d7aaa - 13https://github.com/miraheze/mediawiki [01:13:57] [02mediawiki] 07dependabot[bot] closed pull request 03#5718: Bump extensions/ArticlePlaceholder from `f17427e` to `bc719fb` - 13https://github.com/miraheze/mediawiki/pull/5718 [01:13:58] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-bc719fb [01:14:00] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-bc719fb - 13https://github.com/miraheze/mediawiki [01:14:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:14:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Thanks-c52644c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fb734a6103f1 [01:14:03] [02miraheze/mediawiki] 07dependabot[bot] 03fb734a6 - Bump extensions/Thanks from `841e1f7` to `c52644c` [01:14:04] [02mediawiki] 07dependabot[bot] opened pull request 03#5831: Bump extensions/Thanks from `841e1f7` to `c52644c` - 13https://github.com/miraheze/mediawiki/pull/5831 [01:14:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Thanks-c52644c - 13https://github.com/miraheze/mediawiki [01:14:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-26b91bb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5772b6691580 [01:14:09] [02miraheze/mediawiki] 07dependabot[bot] 035772b66 - Bump extensions/WikimediaIncubator from `2719187` to `26b91bb` [01:14:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5832: Bump extensions/WikimediaIncubator from `2719187` to `26b91bb` - 13https://github.com/miraheze/mediawiki/pull/5832 [01:14:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-26b91bb - 13https://github.com/miraheze/mediawiki [01:14:13] [02mediawiki] 07dependabot[bot] commented on pull request 03#5753: Bump extensions/Thanks from `841e1f7` to `f3f3463` - 13https://github.com/miraheze/mediawiki/pull/5753#issuecomment-1066233137 [01:14:15] [02mediawiki] 07dependabot[bot] closed pull request 03#5753: Bump extensions/Thanks from `841e1f7` to `f3f3463` - 13https://github.com/miraheze/mediawiki/pull/5753 [01:14:16] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Thanks-f3f3463 [01:14:18] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Thanks-f3f3463 - 13https://github.com/miraheze/mediawiki [01:14:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#5826: Bump extensions/TemplateStyles from `b7e6c34` to `eec8c6c` - 13https://github.com/miraheze/mediawiki/pull/5826 [01:14:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#5825: Bump extensions/GlobalWatchlist from `4cff4b3` to `a6c1f0e` - 13https://github.com/miraheze/mediawiki/pull/5825 [01:14:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Linter-96a903f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c60edc636115 [01:14:24] [02miraheze/mediawiki] 07dependabot[bot] 03c60edc6 - Bump extensions/Linter from `5b70116` to `96a903f` [01:14:25] [02mediawiki] 07dependabot[bot] opened pull request 03#5833: Bump extensions/Linter from `5b70116` to `96a903f` - 13https://github.com/miraheze/mediawiki/pull/5833 [01:14:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Linter-96a903f - 13https://github.com/miraheze/mediawiki [01:14:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Report-1fe9705 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5ac4b785f32f [01:14:30] [02miraheze/mediawiki] 07dependabot[bot] 035ac4b78 - Bump extensions/Report from `b10a2dc` to `1fe9705` [01:14:31] [02mediawiki] 07dependabot[bot] opened pull request 03#5834: Bump extensions/Report from `b10a2dc` to `1fe9705` - 13https://github.com/miraheze/mediawiki/pull/5834 [01:14:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Report-1fe9705 - 13https://github.com/miraheze/mediawiki [01:14:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5828: Bump extensions/ApprovedRevs from `bd12cb0` to `6c98593` - 13https://github.com/miraheze/mediawiki/pull/5828 [01:14:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#5827: Bump extensions/Cargo from `0f06913` to `3ead915` - 13https://github.com/miraheze/mediawiki/pull/5827 [01:14:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5828: Bump extensions/ApprovedRevs from `bd12cb0` to `6c98593` - 13https://github.com/miraheze/mediawiki/pull/5828 [01:14:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#5827: Bump extensions/Cargo from `0f06913` to `3ead915` - 13https://github.com/miraheze/mediawiki/pull/5827 [01:14:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Babel-6c205bf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9e99ef1d09f7 [01:14:42] [02miraheze/mediawiki] 07dependabot[bot] 039e99ef1 - Bump extensions/Babel from `265a94d` to `6c205bf` [01:14:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5835: Bump extensions/Babel from `265a94d` to `6c205bf` - 13https://github.com/miraheze/mediawiki/pull/5835 [01:14:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Babel-6c205bf - 13https://github.com/miraheze/mediawiki [01:14:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralAuth-3e1f6d8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8d345034e3a1 [01:14:48] [02miraheze/mediawiki] 07dependabot[bot] 038d34503 - Bump extensions/CentralAuth from `306fa3a` to `3e1f6d8` [01:14:49] [02mediawiki] 07dependabot[bot] opened pull request 03#5836: Bump extensions/CentralAuth from `306fa3a` to `3e1f6d8` - 13https://github.com/miraheze/mediawiki/pull/5836 [01:14:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-3e1f6d8 - 13https://github.com/miraheze/mediawiki [01:14:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Wikibase-f86b6ab [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bda2b32b6e99 [01:14:54] [02miraheze/mediawiki] 07dependabot[bot] 03bda2b32 - Bump extensions/Wikibase from `5c27652` to `f86b6ab` [01:14:55] [02mediawiki] 07dependabot[bot] opened pull request 03#5837: Bump extensions/Wikibase from `5c27652` to `f86b6ab` - 13https://github.com/miraheze/mediawiki/pull/5837 [01:14:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Wikibase-f86b6ab - 13https://github.com/miraheze/mediawiki [01:14:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/QuickInstantCommons-1b051fa [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2c3db51d47e0 [01:15:00] [02miraheze/mediawiki] 07dependabot[bot] 032c3db51 - Bump extensions/QuickInstantCommons from `cd72b29` to `1b051fa` [01:15:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 7.54, 6.20, 5.36 [01:15:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5838: Bump extensions/QuickInstantCommons from `cd72b29` to `1b051fa` - 13https://github.com/miraheze/mediawiki/pull/5838 [01:15:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/QuickInstantCommons-1b051fa - 13https://github.com/miraheze/mediawiki [01:15:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MintyDocs-dc21776 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ea4e599e4321 [01:15:06] [02miraheze/mediawiki] 07dependabot[bot] 03ea4e599 - Bump extensions/MintyDocs from `22c7541` to `dc21776` [01:15:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5839: Bump extensions/MintyDocs from `22c7541` to `dc21776` - 13https://github.com/miraheze/mediawiki/pull/5839 [01:15:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MintyDocs-dc21776 - 13https://github.com/miraheze/mediawiki [01:15:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ManageWiki-f86d703 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/252cd54023d7 [01:15:12] [02miraheze/mediawiki] 07dependabot[bot] 03252cd54 - Bump extensions/ManageWiki from `35b5466` to `f86d703` [01:15:13] [02mediawiki] 07dependabot[bot] opened pull request 03#5840: Bump extensions/ManageWiki from `35b5466` to `f86d703` - 13https://github.com/miraheze/mediawiki/pull/5840 [01:15:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-f86d703 - 13https://github.com/miraheze/mediawiki [01:15:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Moderation-72822a0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/159b6a6b8083 [01:15:18] [02miraheze/mediawiki] 07dependabot[bot] 03159b6a6 - Bump extensions/Moderation from `0579326` to `72822a0` [01:15:19] [02mediawiki] 07dependabot[bot] opened pull request 03#5841: Bump extensions/Moderation from `0579326` to `72822a0` - 13https://github.com/miraheze/mediawiki/pull/5841 [01:15:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Moderation-72822a0 - 13https://github.com/miraheze/mediawiki [01:15:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ProtectSite-1117627 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7c209b833811 [01:15:24] [02miraheze/mediawiki] 07dependabot[bot] 037c209b8 - Bump extensions/ProtectSite from `8eca4df` to `1117627` [01:15:25] [02mediawiki] 07dependabot[bot] opened pull request 03#5842: Bump extensions/ProtectSite from `8eca4df` to `1117627` - 13https://github.com/miraheze/mediawiki/pull/5842 [01:15:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ProtectSite-1117627 - 13https://github.com/miraheze/mediawiki [01:15:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PollNY-060c926 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bd24e58eadf2 [01:15:30] [02miraheze/mediawiki] 07dependabot[bot] 03bd24e58 - Bump extensions/PollNY from `a8b1780` to `060c926` [01:15:31] [02mediawiki] 07dependabot[bot] opened pull request 03#5843: Bump extensions/PollNY from `a8b1780` to `060c926` - 13https://github.com/miraheze/mediawiki/pull/5843 [01:15:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PollNY-060c926 - 13https://github.com/miraheze/mediawiki [01:15:34] [02mediawiki] 07dependabot[bot] commented on pull request 03#5758: Bump extensions/PollNY from `a8b1780` to `e744e28` - 13https://github.com/miraheze/mediawiki/pull/5758#issuecomment-1066233507 [01:15:36] [02mediawiki] 07dependabot[bot] closed pull request 03#5758: Bump extensions/PollNY from `a8b1780` to `e744e28` - 13https://github.com/miraheze/mediawiki/pull/5758 [01:15:37] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PollNY-e744e28 - 13https://github.com/miraheze/mediawiki [01:15:39] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PollNY-e744e28 [01:15:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5829: Bump extensions/ArticlePlaceholder from `f17427e` to `8d6e792` - 13https://github.com/miraheze/mediawiki/pull/5829 [01:15:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#5830: Bump extensions/DataDump from `88351b6` to `b9d7aaa` - 13https://github.com/miraheze/mediawiki/pull/5830 [01:15:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#5830: Bump extensions/DataDump from `88351b6` to `b9d7aaa` - 13https://github.com/miraheze/mediawiki/pull/5830 [01:15:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#5831: Bump extensions/Thanks from `841e1f7` to `c52644c` - 13https://github.com/miraheze/mediawiki/pull/5831 [01:15:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5833: Bump extensions/Linter from `5b70116` to `96a903f` - 13https://github.com/miraheze/mediawiki/pull/5833 [01:15:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#5839: Bump extensions/MintyDocs from `22c7541` to `dc21776` - 13https://github.com/miraheze/mediawiki/pull/5839 [01:15:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5842: Bump extensions/ProtectSite from `8eca4df` to `1117627` - 13https://github.com/miraheze/mediawiki/pull/5842 [01:15:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5834: Bump extensions/Report from `b10a2dc` to `1fe9705` - 13https://github.com/miraheze/mediawiki/pull/5834 [01:15:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#5838: Bump extensions/QuickInstantCommons from `cd72b29` to `1b051fa` - 13https://github.com/miraheze/mediawiki/pull/5838 [01:15:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5840: Bump extensions/ManageWiki from `35b5466` to `f86d703` - 13https://github.com/miraheze/mediawiki/pull/5840 [01:15:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#5838: Bump extensions/QuickInstantCommons from `cd72b29` to `1b051fa` - 13https://github.com/miraheze/mediawiki/pull/5838 [01:15:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5832: Bump extensions/WikimediaIncubator from `2719187` to `26b91bb` - 13https://github.com/miraheze/mediawiki/pull/5832 [01:15:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#5840: Bump extensions/ManageWiki from `35b5466` to `f86d703` - 13https://github.com/miraheze/mediawiki/pull/5840 [01:16:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5835: Bump extensions/Babel from `265a94d` to `6c205bf` - 13https://github.com/miraheze/mediawiki/pull/5835 [01:16:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#5843: Bump extensions/PollNY from `a8b1780` to `060c926` - 13https://github.com/miraheze/mediawiki/pull/5843 [01:16:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5840: Bump extensions/ManageWiki from `35b5466` to `f86d703` - 13https://github.com/miraheze/mediawiki/pull/5840 [01:16:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#5841: Bump extensions/Moderation from `0579326` to `72822a0` - 13https://github.com/miraheze/mediawiki/pull/5841 [01:16:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#5837: Bump extensions/Wikibase from `5c27652` to `f86b6ab` - 13https://github.com/miraheze/mediawiki/pull/5837 [01:16:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#5836: Bump extensions/CentralAuth from `306fa3a` to `3e1f6d8` - 13https://github.com/miraheze/mediawiki/pull/5836 [01:16:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.68, 8.41, 8.47 [01:17:00] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.19, 5.81, 5.32 [01:17:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.64, 7.32, 8.32 [01:19:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.57, 6.00, 5.44 [01:19:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.19, 3.18, 3.16 [01:20:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:20:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.34, 7.62, 8.49 [01:21:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.84, 8.74, 8.71 [01:23:00] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.62, 4.79, 5.11 [01:24:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 8.49, 8.68 [01:25:00] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.99, 4.25, 4.87 [01:27:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.21, 9.05, 8.85 [01:28:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.69, 9.69, 9.15 [01:29:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.93, 9.14, 8.90 [01:30:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.77, 9.17, 9.04 [01:30:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.33, 8.92, 8.52 [01:31:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.86, 9.82, 9.19 [01:34:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.45, 10.20, 9.45 [01:34:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 9.44, 8.79 [01:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 9.86, 9.42 [01:36:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.81, 8.99, 8.72 [01:41:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.68, 9.52, 9.53 [01:42:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.73, 7.72, 6.97 [01:43:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.80, 10.19, 9.78 [01:44:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.10, 7.53, 7.00 [01:44:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.86, 8.60, 8.05 [01:45:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.56, 9.96, 9.76 [01:47:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.72, 10.28, 9.87 [01:48:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 9.10, 8.36 [01:50:56] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.54, 1.96, 1.29 [01:51:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.65, 1.28 [01:52:06] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:08] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.157 second response time [01:52:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 20.08, 9.20, 4.29 [01:52:39] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.08, 7.80, 8.46 [01:52:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.10, 1.63, 1.25 [01:53:13] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.95, 7.83, 6.37 [01:53:16] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:21] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:53:37] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.54, 1.29 [01:53:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.99, 9.74, 9.82 [01:54:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.58, 7.61, 8.01 [01:55:37] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 2.725 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:57:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [01:58:02] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server' [01:58:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.76, 8.09, 7.53 [01:58:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.47, 7.93, 8.49 [01:59:11] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.51, 7.73, 6.89 [01:59:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [02:00:05] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [02:00:11] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.47, 8.20, 7.64 [02:01:11] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.59, 8.57, 7.31 [02:01:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.82, 1.51 [02:02:41] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:03:40] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.51, 1.73, 1.52 [02:03:58] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:05:24] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:05:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.63, 6.85, 8.34 [02:05:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 4.15, 3.36 [02:05:46] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.89, 1.60 [02:05:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 3.35, 2.84 [02:06:15] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:06:32] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:07:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.74, 3.31 [02:07:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.31, 2.87 [02:08:12] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:08:12] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.338 second response time [02:08:32] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.852 second response time [02:08:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [02:09:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.93, 4.00, 3.10 [02:09:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [02:09:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [02:09:19] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:09:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.13, 3.49 [02:09:43] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:09:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.87, 1.67 [02:09:47] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:09:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.56, 3.01 [02:10:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:11:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [02:11:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:11:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.78, 3.18 [02:11:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.59, 3.38 [02:11:43] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.70, 1.63 [02:11:45] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1205 MB (12% inode=80%); [02:11:46] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 51 minutes ago with 0 failures [02:11:47] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [02:11:49] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [02:11:55] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.005568474531 secs [02:12:37] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [02:13:06] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [02:13:10] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [02:13:13] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.85, 7.63, 7.55 [02:13:14] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 3.464 second response time [02:13:20] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.889 second response time [02:13:24] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 0.811 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [02:13:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.26, 4.24, 3.41 [02:13:42] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.705 second response time [02:13:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.95, 3.54 [02:13:44] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.663 second response time [02:14:14] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.038 second response time [02:14:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.88, 3.40 [02:15:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.69, 3.31 [02:15:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:16:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.25, 8.08, 7.21 [02:16:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.47, 4.85, 3.77 [02:17:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.93, 3.66 [02:18:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.62, 7.88, 7.25 [02:19:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.89, 3.45 [02:19:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.64, 1.59 [02:20:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.11, 8.72, 8.18 [02:20:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 8.68, 7.66 [02:21:10] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.29, 6.11, 6.73 [02:21:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.62, 3.41 [02:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:21:32] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.50, 1.55 [02:22:27] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.77, 8.29, 8.10 [02:22:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.47, 7.77, 7.48 [02:23:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 3.29, 3.32 [02:23:36] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:38] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:38] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:38] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:08] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:24:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.87, 3.56, 3.75 [02:24:48] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:56] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:24:58] PROBLEM - cp31 Stunnel HTTP for mon111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:25:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.20, 6.62, 6.79 [02:25:26] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:27] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:30] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:36] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.47, 2.91, 3.37 [02:25:53] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:26:06] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:26:37] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:41] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:26:56] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.241 second response time [02:27:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.06, 6.37, 6.67 [02:27:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw101 mw102 mw122 [02:27:29] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.008 second response time [02:27:30] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.478 second response time [02:27:36] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.526 second response time [02:27:36] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.210 second response time [02:27:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw102 mw111 mw112 mw121 mw122 [02:27:53] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.353 second response time [02:28:02] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:09] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:19] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.006324976683 secs [02:28:21] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:32] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:57] RECOVERY - cp31 Stunnel HTTP for mon111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 33915 bytes in 1.185 second response time [02:29:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [02:29:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [02:30:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.30, 2.80, 3.33 [02:31:07] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.54, 6.84, 6.83 [02:33:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.42, 6.67, 6.77 [02:33:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [02:34:17] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.751 second response time [02:34:24] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:34:26] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:34:31] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 3.518 second response time [02:34:35] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:34:40] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:34:51] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [02:34:51] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [02:35:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [02:35:21] PROBLEM - cp31 Stunnel HTTP for phab121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:35:36] SRE: persistent 503s on multiple wikis [02:35:54] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.312 second response time [02:36:21] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.325 second response time [02:36:22] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.995 second response time [02:36:34] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.358 second response time [02:37:19] RECOVERY - cp31 Stunnel HTTP for phab121 on cp31 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.297 second response time [02:37:48] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:37:54] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:37:58] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [02:39:54] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:39:55] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.765 second response time [02:40:04] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:09] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:40:13] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.88, 1.68 [02:40:22] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:40:47] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:59] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [02:41:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:30] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:42:03] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.047 second response time [02:42:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.45, 1.67, 1.62 [02:42:18] .give Agent coffee [02:42:25] erm [02:42:33] I guess it doesn't work anymore :P [02:42:43] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.230 second response time [02:42:45] .goofy give Agent coffee [02:42:50] .help [02:42:50] dmehus: I've published a list of my commands at: https://sopel.mirahezebots.org/help_prod.html [02:42:58] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.778 second response time [02:43:06] .help goofy [02:43:06] dmehus: Unknown command "goofy" [02:43:25] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [02:43:26] .coffee Agent [02:43:26] * MirahezeBot gives Agent a nice warm cup of coffee. [02:43:27] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.883 second response time [02:43:28] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.006964504719 secs [02:43:29] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures [02:43:31] ah there we go [02:43:32] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [02:44:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.36, 2.96 [02:45:08] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1136 MB (12% inode=80%); [02:45:15] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [02:46:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.47, 3.04 [02:47:02] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.30, 6.89, 6.83 [02:47:03] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:48:09] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.838 second response time [02:48:16] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [02:48:23] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [02:48:36] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 0.808 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [02:48:38] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [02:48:56] dmehus: thanks for the coffee lol [02:49:01] I really need it right now [02:49:01] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 7.446 second response time [02:49:01] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.333 second response time [02:49:02] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.64, 5.89, 6.48 [02:49:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw102 mw112 [02:49:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw121 mw122 [02:49:57] Agent, np heh [02:50:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.80, 1.65 [02:50:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 2.319 second response time [02:51:19] PROBLEM - cp30 Varnish Backends on cp30 is WARNING: No backends detected. If this is an error, see readme.txt [02:51:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.19, 5.08, 3.53 [02:51:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.02, 5.24, 3.83 [02:51:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [02:51:58] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.712 second response time [02:52:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.58, 1.59 [02:52:11] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 8.795 second response time [02:52:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.97, 4.49, 3.51 [02:53:01] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.514 second response time [02:53:16] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [02:54:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.35, 1.84, 1.37 [02:57:43] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:58:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.46, 1.33 [02:58:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.75, 3.51, 3.48 [02:58:43] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [02:58:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:59] PROBLEM - cp31 Stunnel HTTP for mail121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:59:03] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:59:07] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:59:14] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:59:14] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:59:16] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:59:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.10, 3.77, 3.60 [02:59:24] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:59:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.53, 3.32, 3.59 [02:59:55] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:59:55] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:00:12] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:00:21] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:22] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.41, 2.79, 3.21 [03:00:29] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:00:31] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:35] PROBLEM - cp31 Stunnel HTTP for mon111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:00:38] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:47] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:00:48] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:00:53] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:00:58] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:01:09] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:12] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.00, 2.90, 3.30 [03:01:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw101 mw111 mw112 mw121 mw122 [03:01:30] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.239 second response time [03:01:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.00, 2.52, 3.26 [03:01:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw101 mw102 mw112 mw121 mw122 [03:01:52] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:54] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.923 second response time [03:01:55] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.939 second response time [03:01:57] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.66, 2.08, 1.77 [03:02:08] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:02:10] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:02:10] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:02:12] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.312 second response time [03:02:14] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [03:02:26] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.464 second response time [03:02:34] RECOVERY - cp31 Stunnel HTTP for mon111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 33954 bytes in 0.342 second response time [03:02:47] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.318 second response time [03:02:50] RECOVERY - cp31 Stunnel HTTP for mail121 on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.235 second response time [03:02:55] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.870 second response time [03:02:56] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.335 second response time [03:02:56] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.337 second response time [03:03:13] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.314 second response time [03:03:14] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.323 second response time [03:03:23] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [03:03:28] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.315 second response time [03:04:08] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.020 second response time [03:04:12] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.297 second response time [03:04:17] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.060 second response time [03:04:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw102 mw111 [03:04:35] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.913 second response time [03:04:48] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.332 second response time [03:05:19] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 11% [03:06:40] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:47] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:48] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.490 second response time [03:06:48] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:57] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:01] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:10] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.818 second response time [03:07:11] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [03:07:16] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.278 second response time [03:07:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [03:07:51] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.95, 1.83 [03:08:06] yo miraheze is down [03:08:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [03:08:46] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.712 second response time [03:08:49] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 6.401 second response time [03:08:49] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.263 second response time [03:08:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 3.50, 3.25 [03:08:56] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [03:08:57] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.602 second response time [03:08:58] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.557 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [03:09:02] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 3.587 second response time [03:09:10] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.404 second response time [03:09:16] we are back on line thank you guys so much [03:09:26] i appreciate all the work you guys to [03:09:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 4.08, 3.56 [03:10:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.60, 3.24 [03:10:10] JackTemporary, thanks, yeah I started a task related to this [03:10:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.18, 7.75, 7.01 [03:11:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.28, 3.41, 3.38 [03:11:47] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.58, 1.70 [03:12:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.78, 2.97, 3.05 [03:12:42] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:12:44] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:12:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 0.86, 2.60, 2.99 [03:12:50] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.62, 7.03, 6.84 [03:12:56] PROBLEM - cp30 Stunnel HTTP for mon111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:12:59] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [03:13:00] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [03:13:02] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [03:13:03] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [03:13:03] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1116 MB (11% inode=80%); [03:13:08] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.008310765028 secs [03:13:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 2.98, 3.21 [03:14:47] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.357 second response time [03:14:55] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:15:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw102 mw112 mw121 mw122 [03:15:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw101 mw102 mw111 mw112 mw121 [03:16:40] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.331 second response time [03:16:50] RECOVERY - cp30 Stunnel HTTP for mon111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 33961 bytes in 0.338 second response time [03:16:50] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [03:18:36] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:19:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 2.06, 1.84 [03:20:13] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [03:20:36] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 3.172 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [03:20:52] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.22, 7.19, 6.98 [03:21:03] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.312 second response time [03:21:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [03:21:53] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.07, 1.69, 1.73 [03:22:52] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.24, 6.69, 6.81 [03:23:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [03:23:51] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.10, 1.50, 1.66 [03:25:10] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:26:34] !sre [03:26:39] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:51] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.97, 6.63, 6.77 [03:27:01] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:27:05] PROBLEM - cp31 Stunnel HTTP for phab121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:27:07] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.177 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [03:27:34] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:36] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:48] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [03:27:52] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.317 second response time [03:27:56] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [03:27:58] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.008 second response time [03:28:22] Can reproduce the above persistently [03:28:33] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:28:42] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:28:46] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:05] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:29:05] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:29:05] RECOVERY - cp31 Stunnel HTTP for phab121 on cp31 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.282 second response time [03:29:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [03:29:19] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw101 mw102 mw112 mw121 mw122 [03:30:28] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [03:30:34] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:30:42] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.323 second response time [03:31:10] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.307 second response time [03:31:16] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.272 second response time [03:31:54] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1062 MB (11% inode=80%); [03:32:01] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [03:32:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [03:32:44] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [03:32:46] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [03:33:29] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.406 second response time [03:33:33] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [03:33:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [03:33:41] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.499 second response time [03:33:44] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.951 second response time [03:33:48] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.49, 6.89, 6.87 [03:33:50] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.026 second response time [03:33:55] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.273 second response time [03:33:58] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.770 second response time [03:35:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [03:35:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [03:36:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [03:37:08] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:30] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:39:36] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:39:57] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:40:05] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:40:09] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:40:24] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:40:29] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:40:38] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:41:05] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:41:14] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:41:16] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:41:16] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:41:44] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:41:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [03:42:45] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1059 MB (11% inode=80%); [03:43:15] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.942 second response time [03:43:19] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.064 second response time [03:43:39] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:43:44] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.668 second response time [03:43:46] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:02] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.009 second response time [03:44:02] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.290 second response time [03:44:27] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:45] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:08] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:21] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [03:45:22] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.393 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [03:45:22] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [03:45:24] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [03:45:26] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.04065275192 secs [03:45:27] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.79, 1.72 [03:45:32] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:32] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:37] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.797 second response time [03:45:41] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.617 second response time [03:46:30] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.019 second response time [03:46:42] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.768 second response time [03:47:06] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22337 bytes in 1.091 second response time [03:47:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw102 [03:47:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.59, 2.08, 1.84 [03:47:31] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [03:47:31] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.024 second response time [03:47:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw102 mw111 [03:49:04] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:49:10] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:49:15] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:49:18] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:49:33] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:49:38] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [03:50:07] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [03:51:04] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.396 second response time [03:51:08] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.324 second response time [03:51:13] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.339 second response time [03:51:18] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.284 second response time [03:51:21] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.18, 1.91, 1.85 [03:51:30] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [03:51:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:51:33] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.313 second response time [03:53:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [03:54:18] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:55:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [03:55:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.87, 1.84 [03:55:50] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 764 MB (8% inode=80%); [03:57:03] PROBLEM - cp31 Stunnel HTTP for mon111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:57:03] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:57:08] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.85, 1.83 [03:57:23] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:27] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:29] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:57:35] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:48] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [03:58:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [03:58:20] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:58:44] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:58:44] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:58:52] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:58] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:59] Error 503 Backend fetch failed, forwarded for 73.68.166.56, 127.0.0.1 [03:58:59] (Varnish XID 25166821) via cp30 at Mon, 14 Mar 2022 03:58:26 GMT. [03:59:02] RECOVERY - cp31 Stunnel HTTP for mon111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 33961 bytes in 0.339 second response time [03:59:03] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.322 second response time [03:59:04] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:59:06] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:59:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.08, 1.97, 1.88 [03:59:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw101 mw102 mw112 mw121 mw122 [03:59:20] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.231 second response time [03:59:26] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.330 second response time [03:59:29] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:59:29] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:59:31] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.311 second response time [03:59:48] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.793 second response time [04:00:02] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [04:00:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:00:15] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [04:00:16] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.344 second response time [04:00:43] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [04:00:47] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.022 second response time [04:00:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [04:00:56] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.325 second response time [04:01:04] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.334 second response time [04:01:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.84, 1.84 [04:01:23] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:01:31] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.542 second response time [04:01:39] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.024 second response time [04:01:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [04:01:53] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [04:01:59] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 8% [04:02:15] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:03:50] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 15% [04:04:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:04:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:05:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [04:05:38] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.52, 7.18, 7.03 [04:05:44] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:36] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:50] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:52] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:55] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1046 MB (11% inode=80%); [04:07:04] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:08] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:16] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [04:07:27] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.81, 7.14, 7.02 [04:07:52] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [04:07:56] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [04:08:34] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.304 second response time [04:08:46] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.595 second response time [04:08:48] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.359 second response time [04:08:59] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [04:09:07] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.323 second response time [04:09:26] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.014 second response time [04:09:33] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [04:09:34] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.915 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [04:09:38] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.05346053839 secs [04:09:38] I'm getting 503s, with error messages like "(Varnish XID 54232392) via cp30 at Mon, 14 Mar 2022 04:06:51 GMT. " (on behalf of my IP) when I try loading a few of my favorite Miraheze wikis. [04:09:40] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [04:09:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.342 second response time [04:09:46] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [04:11:05] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.13, 1.42, 1.64 [04:11:38] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.06, 7.66, 7.22 [04:12:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:13:25] robla, yeah, same, I opened https://phabricator.miraheze.org/T8930 if you want to add additional reporting information there [04:13:25] [url] ⚓ T8930 Persistent 503s on multiple wikis, including `metawiki` | phabricator.miraheze.org [04:13:37] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.65, 7.49, 7.20 [04:13:59] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:14:28] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:16:21] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:17:05] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.231 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [04:17:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:18:09] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [04:18:23] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1040 MB (11% inode=80%); [04:19:58] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.576 second response time [04:20:07] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.586 second response time [04:22:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.24, 1.48, 1.01 [04:22:26] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 880 MB (9% inode=80%); [04:23:58] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:24:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.35, 1.02 [04:24:16] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:24:27] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:24:28] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:25] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:33] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.64, 7.53, 7.24 [04:25:52] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:26:04] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:26:08] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:27:21] PROBLEM - cp31 Stunnel HTTP for mail121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:27:26] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:27:27] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:27:28] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:27:29] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:27:30] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:27:33] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:27:43] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:27:59] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:28:17] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:10] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 55% [04:29:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [04:29:17] RECOVERY - cp31 Stunnel HTTP for mail121 on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.242 second response time [04:29:25] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.802 second response time [04:29:26] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.309 second response time [04:29:27] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.310 second response time [04:29:29] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.335 second response time [04:29:32] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.49, 6.95, 7.08 [04:29:33] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.312 second response time [04:29:38] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.306 second response time [04:29:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [04:29:58] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [04:30:16] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.404 second response time [04:30:27] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.0348855257 secs [04:30:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:31:07] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [04:31:12] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 43% [04:31:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:31:38] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:32:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:33:03] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 2% [04:33:08] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 8% [04:33:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [04:33:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:33:31] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.39, 5.99, 6.64 [04:33:34] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [04:33:36] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [04:33:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [04:35:44] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [04:35:44] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:36:14] thanks dmehus [04:36:21] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:36:55] np [04:37:00] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [04:37:03] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [04:37:07] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [04:37:31] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:40] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:40] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:47] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:49] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:38:05] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 991 MB (10% inode=80%); [04:38:20] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [04:38:25] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 0.307 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [04:38:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:38:36] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.436 second response time [04:39:07] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 0.287 second response time [04:39:13] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.347 second response time [04:39:28] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.299 second response time [04:39:35] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.305 second response time [04:39:39] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [04:39:45] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.261 second response time [04:39:47] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.307 second response time [04:40:16] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.113 second response time [04:40:19] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.825 second response time [04:50:34] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.388 second response time [04:51:20] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:22] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:31] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.48, 2.97 [04:52:43] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:54:07] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:54:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.89, 3.28, 2.96 [04:54:57] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:05] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.231 second response time [04:55:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw101 mw102 mw111 mw112 mw122 [04:55:27] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: PROCS CRITICAL: 2 processes with args 'redis-server' [04:55:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw101 mw122 [04:56:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:15] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:57:08] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:57:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [04:57:17] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:57:25] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [04:57:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [04:58:00] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.711 second response time [04:58:13] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [04:58:14] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.314 second response time [04:59:07] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [04:59:13] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.014 second response time [04:59:25] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 925 MB (9% inode=80%); [05:00:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.84, 1.88, 1.31 [05:00:23] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.55, 6.93, 6.29 [05:02:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.78, 1.53, 1.25 [05:04:45] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [05:05:17] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [05:05:29] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.311 second response time [05:05:30] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.704 second response time [05:07:08] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:08:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:09:45] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [05:09:47] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:10:58] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:10:59] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:11:42] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:13:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 1.95, 1.68 [05:13:09] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:13:35] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:13:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw111 mw112 [05:15:07] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.85, 1.68 [05:15:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [05:16:04] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:16:06] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:16:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.69, 7.55, 7.09 [05:16:31] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:17:05] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.88, 1.70 [05:17:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:18:08] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 914 MB (9% inode=80%); [05:18:08] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [05:18:10] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [05:18:18] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.72, 7.26, 7.04 [05:18:18] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [05:18:24] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [05:18:40] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset 0.0127953589 secs [05:19:26] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [05:19:26] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.309 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [05:19:59] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [05:21:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.99, 1.81 [05:22:58] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:25:04] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.10, 1.55, 1.67 [05:25:13] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 905 MB (9% inode=80%); [05:27:26] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:28:07] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:29:09] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:29:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 8.06, 6.62 [05:30:17] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.01, 7.02, 6.92 [05:31:02] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:31:04] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:31:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.57, 2.04, 1.82 [05:31:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.32, 7.50, 6.61 [05:32:00] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:07] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:07] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:10] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:17] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.22, 7.33, 7.06 [05:33:02] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 868 MB (9% inode=80%); [05:33:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.79, 1.76 [05:33:12] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [05:34:24] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [05:35:50] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [05:36:41] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:36:43] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [05:37:04] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.57, 1.68 [05:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:37:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [05:37:49] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.372 second response time [05:37:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [05:38:27] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [05:38:43] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.311 second response time [05:38:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [05:39:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [05:39:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [05:40:08] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:40:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw101 mw121 [05:40:19] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.48, 6.52, 6.74 [05:40:27] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 4% [05:40:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:41:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:41:18] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:41:30] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [05:42:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [05:44:17] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.20, 7.49, 7.10 [05:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:48:19] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.63, 7.94, 7.39 [05:48:21] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [05:48:26] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 0.652 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [05:48:35] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [05:48:36] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [05:48:43] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [05:49:11] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [05:49:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.04, 8.29, 7.23 [05:51:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.87, 1.76 [05:51:49] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.33, 7.58, 7.10 [05:53:04] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.46, 1.67, 1.69 [05:53:09] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 899 MB (9% inode=80%); [05:54:35] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:55:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:55:39] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.01363241673 secs [05:57:18] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:58:17] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.04, 7.28, 7.16 [05:58:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:58:55] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:24] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 2.760 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [06:00:17] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.12, 6.98, 7.05 [06:02:56] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [06:02:56] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [06:03:17] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:03:29] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [06:03:39] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [06:03:39] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [06:03:56] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:04:17] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.65, 6.24, 6.78 [06:04:53] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.489 second response time [06:04:54] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.067 second response time [06:05:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw121 mw122 [06:05:18] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:05:28] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [06:05:39] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.320 second response time [06:05:39] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14552 bytes in 0.340 second response time [06:05:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw112 mw121 [06:05:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw101 mw102 mw111 mw112 mw121 [06:06:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.015 second response time [06:06:36] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:07:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.66, 1.63 [06:07:29] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [06:07:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [06:07:48] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [06:07:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [06:10:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.459 second response time [06:11:05] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.63, 1.70, 1.65 [06:11:11] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [06:11:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [06:11:19] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [06:11:25] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 855 MB (9% inode=80%); [06:11:32] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.769 second response time [06:11:32] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 6.484 second response time [06:11:35] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:12:01] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.035 second response time [06:12:33] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.515 second response time [06:13:06] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.631 second response time [06:16:40] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:17:17] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.665 second response time [06:17:37] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:17:57] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:18:00] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:18:16] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:18:48] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:18:57] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 2.10, 1.82 [06:19:16] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 478 MB (5% inode=80%); [06:19:20] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:22] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:43] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [06:20:02] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:30] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:20:35] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:20:52] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.905 second response time [06:21:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.89, 1.78 [06:21:11] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:15] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 773 MB (8% inode=80%); [06:21:22] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.755 second response time [06:21:25] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.537 second response time [06:21:39] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:03] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.741 second response time [06:23:02] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:23:10] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [06:23:38] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:40] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.321 second response time [06:23:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw101 mw102 mw121 mw122 [06:24:23] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:40] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [06:24:49] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:51] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.323 second response time [06:24:52] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [06:26:07] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 543 MB (5% inode=80%); [06:26:11] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [06:26:26] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.693 second response time [06:26:29] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.0004619956017 secs [06:26:33] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [06:26:45] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.428 second response time [06:26:49] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.253 second response time [06:27:03] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.335 second response time [06:27:04] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.64, 1.70 [06:27:23] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 6.025 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [06:27:42] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.627 second response time [06:27:46] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [06:27:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [06:27:55] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [06:28:17] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.81, 6.74, 6.70 [06:30:17] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.29, 6.62, 6.66 [06:32:09] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:52] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:54] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [06:33:10] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:11] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:12] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:23] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [06:33:26] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:33:32] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:54] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [06:34:04] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:15] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:35] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:34:47] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:34:52] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 4.873 second response time [06:34:53] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.967 second response time [06:35:01] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:35:08] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 1.550 second response time [06:35:08] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.319 second response time [06:35:11] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.328 second response time [06:35:16] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:35:19] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [06:35:27] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [06:35:28] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:35:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw122 [06:35:53] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.501 second response time [06:35:55] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:36:03] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.926 second response time [06:36:13] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.021 second response time [06:37:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [06:41:17] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.16, 6.70, 6.57 [06:43:09] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [06:43:10] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [06:43:13] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [06:43:17] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.92, 6.41, 6.48 [06:43:44] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset 0.005151242018 secs [06:44:27] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.437 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [06:44:28] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 33 minutes ago with 0 failures [06:44:32] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [06:47:51] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.89, 1.75 [06:48:39] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [06:49:14] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.58, 6.94, 6.66 [06:49:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.83, 1.74 [06:51:14] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.90, 7.84, 7.02 [06:52:00] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:02] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server' [06:53:13] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.13, 7.01, 6.81 [06:53:58] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [06:54:01] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [06:55:45] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.61, 1.68 [06:57:56] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:59:11] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.87, 8.20, 7.31 [06:59:48] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:00:07] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:00:20] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:00:59] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:01:08] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [07:01:11] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.74, 7.63, 7.21 [07:01:33] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [07:02:01] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [07:02:07] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [07:02:08] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [07:02:09] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [07:02:21] PROBLEM - cp31 Stunnel HTTP for phab121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [07:03:01] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:04] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [07:03:11] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw101 mw102 mw111 mw112 [07:03:29] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:29] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.249 second response time [07:03:35] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.375 second response time [07:03:35] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [07:03:58] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:04:00] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.288 second response time [07:04:05] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [07:04:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw102 mw111 mw121 [07:04:21] RECOVERY - cp31 Stunnel HTTP for phab121 on cp31 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.321 second response time [07:04:30] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:04:31] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.243 second response time [07:04:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [07:04:45] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:00] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:01] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:03] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:03] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:09] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:10] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.49, 5.98, 6.64 [07:05:10] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:20] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:30] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:37] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 4.987 second response time [07:05:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw102 mw121 [07:06:00] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.843 second response time [07:06:01] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.363 second response time [07:06:14] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.322 second response time [07:06:28] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.267 second response time [07:06:30] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.336 second response time [07:06:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.267 second response time [07:06:55] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.016 second response time [07:06:58] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [07:06:59] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [07:07:03] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.336 second response time [07:07:05] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.320 second response time [07:07:08] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.765 second response time [07:07:09] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.333 second response time [07:07:10] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.250 second response time [07:07:11] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [07:07:30] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [07:07:31] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.306 second response time [07:07:32] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.338 second response time [07:08:13] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [07:08:30] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.88, 1.75 [07:09:30] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [07:09:31] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [07:09:33] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.011 second response time [07:09:47] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:57] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:10:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [07:10:28] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.74, 1.71 [07:11:09] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.717 second response time [07:11:25] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.020 second response time [07:11:26] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.014 second response time [07:11:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.362 second response time [07:11:44] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [07:11:54] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:55] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.983 second response time [07:12:05] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:52] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:58] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [07:13:06] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [07:13:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [07:13:57] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 6.941 second response time [07:13:57] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.041 second response time [07:14:05] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 4.942 second response time [07:14:48] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.271 second response time [07:15:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [07:16:22] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.59, 1.67 [07:16:58] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [07:17:03] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset 0.007724195719 secs [07:18:11] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [07:19:31] .op [07:19:31] Attempting to OP... [07:20:05] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.08, 7.14, 6.71 [07:20:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.79, 1.74 [07:21:29] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [07:21:35] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 29 minutes ago with 0 failures [07:21:47] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 8.954 second response time [07:21:47] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 7.583 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [07:22:04] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.76, 7.32, 6.83 [07:22:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.63, 1.69 [07:23:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:25:55] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.341 second response time [07:30:02] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.87, 6.53, 6.65 [07:31:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:35:22] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:51] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:36:17] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.08, 4.58, 2.38 [07:36:30] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:37:31] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:38:25] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:38:54] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:39:35] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:39:35] [02mw-config] 07ugochimobi opened pull request 03#4508: necessary wikibase settings for gp/gd - 13https://github.com/miraheze/mw-config/pull/4508 [07:39:49] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:08] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:22] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:26] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:40:37] miraheze/mw-config - ugochimobi the build passed. [07:40:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:57] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:03] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:20] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [07:41:27] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:41] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [07:41:45] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [07:41:47] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.02715203166 secs [07:41:54] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 9.284 second response time [07:41:58] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:01] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.09, 6.81, 6.64 [07:42:06] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:07] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.116 second response time [07:42:23] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.507 second response time [07:42:36] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [07:42:52] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.306 second response time [07:42:53] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [07:43:01] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.659 second response time [07:43:25] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.278 second response time [07:43:27] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.794 second response time [07:43:54] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.476 second response time [07:44:00] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.50, 6.34, 6.49 [07:44:03] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.125 second response time [07:44:17] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.70, 1.55, 1.73 [07:46:03] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:46:17] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.39, 1.16, 1.56 [07:47:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw101 mw102 mw112 mw121 mw122 [07:48:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [07:48:40] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:20] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:30] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [07:49:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.81, 1.62 [07:50:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [07:50:20] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [07:50:45] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.02549982071 secs [07:51:17] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [07:51:22] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [07:51:30] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [07:51:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [07:51:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.18, 1.59, 1.57 [07:51:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [07:52:20] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [07:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:54:36] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 2.480 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [07:54:59] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 2.317 second response time [07:55:06] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.548 second response time [07:55:13] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.846 second response time [07:55:17] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.268 second response time [07:55:19] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:55:45] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.636 second response time [07:56:28] Reception123: Morning, I assume you're busy :p and hope you're fine? [07:59:34] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:07:53] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.010 second response time [08:09:08] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.251 second response time [08:09:08] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [08:09:12] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.024 second response time [08:09:20] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.231 second response time [08:09:34] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:09:47] ugochimobi: yeah, hopefully I'll be back tomorrow [08:11:09] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.588 second response time [08:11:09] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.536 second response time [08:11:13] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.620 second response time [08:11:20] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.128 second response time [08:11:53] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.645 second response time [08:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:15:37] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 1016 MB (10% inode=80%); [08:18:01] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server' [08:19:59] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [08:20:31] Reception123: ah, nice [08:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:28:12] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 1.24, 0.98, 3.80 [08:30:12] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 0.60, 0.82, 3.40 [08:43:05] [02mw-config] 07ugochimobi opened pull request 03#4509: Popups config for gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4509 [08:44:14] miraheze/mw-config - ugochimobi the build passed. [08:51:17] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:36] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:50] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:52] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:10] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:15] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 32.02, 16.47, 7.74 [08:54:00] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:11] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 565 MB (5% inode=80%); [08:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:54:55] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:55:13] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:55:57] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:56:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.97, 3.51, 1.76 [08:56:50] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 949 MB (10% inode=80%); [08:57:53] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:58:05] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:17] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.14, 1.61, 0.99 [08:58:34] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:38] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [08:58:38] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:38] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:40] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:04] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:06] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:09] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:12] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:46] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.561 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [08:59:47] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [08:59:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.83, 1.51 [08:59:56] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 40 minutes ago with 0 failures [08:59:58] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.022 second response time [09:00:03] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.026 second response time [09:00:17] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.48, 1.15, 0.90 [09:00:33] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.312 second response time [09:00:33] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [09:00:36] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.325 second response time [09:00:37] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.319 second response time [09:01:00] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.038 second response time [09:01:04] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.554 second response time [09:01:05] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.617 second response time [09:01:08] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [09:01:39] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.39, 2.57 [09:02:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.62, 1.62, 1.46 [09:02:12] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [09:02:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.78, 3.45, 2.51 [09:03:41] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:45] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 944 MB (9% inode=80%); [09:03:48] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.61, 1.49 [09:04:16] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:04:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.83, 3.08, 2.48 [09:05:39] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [09:05:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.34, 2.77 [09:10:36] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:12:16] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 2.152 second response time [09:12:31] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 934 MB (9% inode=80%); [09:12:31] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.494 second response time [09:12:55] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.557 second response time [09:13:01] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.630 second response time [09:13:02] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.174 second response time [09:16:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.52, 3.12 [09:16:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.61, 3.12 [09:18:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.55, 3.16 [09:20:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.37, 3.33, 3.11 [09:20:13] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.650 second response time [09:20:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.20, 3.08 [09:20:48] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:15] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:18] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:23] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:54] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:24:00] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:24:11] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:25:03] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:25:08] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:28:49] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:28:53] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:28:54] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:28:54] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [09:29:11] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [09:29:16] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures [09:29:53] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 647 MB (6% inode=80%); [09:30:03] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 7.173 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [09:30:51] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [09:30:55] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [09:30:55] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [09:33:50] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:33:50] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:37:47] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:45] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [09:42:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.70, 3.42, 3.30 [09:42:50] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:44:07] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:44:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 3.26, 3.26 [09:44:26] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:44:55] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:45:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:46:15] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [09:46:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw121 [09:46:43] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:46:49] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:46:53] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [09:46:53] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:46:53] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:46:56] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:47:02] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.01915743947 secs [09:47:04] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:39] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 901 MB (9% inode=80%); [09:48:06] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 2.680 second response time [09:48:09] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [09:48:10] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [09:48:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [09:48:55] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [09:48:59] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [09:49:02] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [09:49:13] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [09:49:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:51:59] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [09:52:33] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:57] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:52:59] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [09:53:05] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [09:53:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw102 mw111 mw122 [09:53:24] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.53, 7.08, 6.29 [09:53:46] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:53:53] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:54:07] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.008 second response time [09:54:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 5 backends are down. mw101 mw102 mw111 mw112 mw121 [09:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:54:35] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:36] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:59] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.327 second response time [09:55:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:28] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:40] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:46] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw102 mw112 mw122 [09:55:49] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:54] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw102 mw112 mw122 [09:55:55] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:58] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.432 second response time [09:56:00] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:13] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:30] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.340 second response time [09:56:50] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.384 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [09:57:18] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [09:57:22] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.835 second response time [09:57:23] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.72, 6.58, 6.31 [09:57:23] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:24] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 7.193 second response time [09:57:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:33] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:33] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [09:57:36] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.282 second response time [09:57:43] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.312 second response time [09:57:43] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.922 second response time [09:57:45] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 898 MB (9% inode=80%); [09:57:47] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [09:57:52] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [09:57:54] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [09:57:55] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [09:58:08] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.312 second response time [09:58:11] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.258 second response time [09:58:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:58:39] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [09:58:43] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [09:58:48] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 4.948 second response time [09:58:55] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.007117718458 secs [09:59:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [09:59:19] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.022 second response time [09:59:19] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [09:59:26] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.511 second response time [09:59:30] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.402 second response time [09:59:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [10:00:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [10:01:52] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:02:12] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:02:51] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [10:02:56] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:03:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:03:42] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:04:44] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:04:56] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:04:59] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:05:00] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:01] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:05] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:05:24] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:05:25] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:25] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:26] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:30] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:32] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [10:05:33] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:05:34] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:05:35] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:37] PROBLEM - cp20 Stunnel HTTP for test101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:38] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:05:41] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:05:43] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [10:05:51] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:51] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:06:13] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [10:06:18] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [10:06:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [10:06:22] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:06:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [10:06:28] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:06:32] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [10:06:38] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:06:43] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [10:06:51] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [10:06:56] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [10:06:56] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [10:06:57] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [10:06:58] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:12] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw101 mw102 mw111 mw121 mw122 [10:07:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [10:07:20] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.028 second response time [10:07:21] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.022 second response time [10:07:23] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [10:07:28] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [10:07:33] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.326 second response time [10:07:34] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.314 second response time [10:07:34] RECOVERY - cp20 Stunnel HTTP for test101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.021 second response time [10:07:36] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [10:07:38] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:48] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.292 second response time [10:07:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [10:08:11] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.325 second response time [10:08:22] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.320 second response time [10:08:26] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.293 second response time [10:08:27] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.312 second response time [10:08:35] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.314 second response time [10:08:56] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.312 second response time [10:09:00] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:09:02] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:09:28] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [10:09:32] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [10:09:35] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [10:10:16] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 33% [10:10:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:11:24] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.236 second response time [10:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:11:39] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:42] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.016 second response time [10:11:42] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:44] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:53] Is anybody receiving a 502 or 503 error? I am, and I have proof of it. [10:11:56] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [10:11:59] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [10:12:01] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 0.687 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [10:12:08] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:08] [discord] https://cdn.discordapp.com/attachments/808001911868489748/952871748133601330/bandicam_2022-03-14_06-11-07-022.jpg [10:12:08] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:10] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:16] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:17] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:25] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.344 second response time [10:12:30] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:34] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.314 second response time [10:12:36] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:40] [discord] When I tried to get onto a page, I got this instead, so I am not sure if this is happening to anyone else. [10:12:49] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [10:13:15] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [10:13:16] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [10:13:19] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:33] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 35 minutes ago with 0 failures [10:13:37] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset 0.03439244628 secs [10:13:38] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.027 second response time [10:13:39] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [10:13:42] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.318 second response time [10:13:42] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.312 second response time [10:13:42] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.240 second response time [10:13:54] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:55] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:58] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [10:14:02] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [10:14:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.329 second response time [10:14:05] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.208 second response time [10:14:12] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.333 second response time [10:14:15] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.328 second response time [10:14:25] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.316 second response time [10:14:33] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.275 second response time [10:14:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [10:15:14] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [10:15:17] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 4.275 second response time [10:15:19] [discord] I was getting them everywhere including Meta and Phabricator [10:15:39] newlivewiki.miraheze.org，目测打不开 [10:15:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [10:16:05] [discord] I put one of the errors in #general [10:16:12] [discord] I put one of the errors in #general earlier [10:16:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 7.036 second response time [10:16:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:16:32] [discord] Oh, so I'm not the only one then? [10:16:47] No ^^' [10:16:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:17:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [10:17:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:17:15] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [10:17:17] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [10:17:25] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [10:17:30] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.71, 1.57 [10:17:40] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.334 second response time [10:17:54] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [10:17:57] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.310 second response time [10:18:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [10:19:16] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.02, 6.74, 5.84 [10:19:28] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.88, 1.65 [10:19:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [10:20:03] [discord] And we’re back up [10:20:14] [discord] We'll just see how long. [10:21:07] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:21:26] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.72, 1.62 [10:23:27] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.67, 1.69, 1.62 [10:23:48] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:57] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:25:14] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.83, 7.64, 6.53 [10:25:42] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:14] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:56] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:56] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:57] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:31:34] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:41] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [10:31:53] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:31:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:02] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:10] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:18] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 1.96, 1.74 [10:32:32] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:12] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.59, 7.79, 7.01 [10:33:29] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.358 second response time [10:33:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:35] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [10:33:52] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.316 second response time [10:33:55] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [10:33:55] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [10:34:04] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [10:34:06] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.677 second response time [10:34:06] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [10:34:13] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 5.706 second response time [10:34:14] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.02153652906 secs [10:34:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.99, 1.79 [10:34:28] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.306 second response time [10:34:46] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.113 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [10:35:11] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.31, 7.39, 6.97 [10:35:25] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [10:35:47] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.532 second response time [10:38:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.20, 1.57, 1.66 [10:38:29] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:36] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:39:42] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:39:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.34, 2.62 [10:39:46] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [10:40:59] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:16] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:41:35] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.020 second response time [10:41:36] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:41:38] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.029 second response time [10:41:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [10:41:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 2.89, 2.54 [10:41:44] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [10:42:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw111 mw121 mw122 [10:42:48] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server' [10:43:03] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset 0.007112979889 secs [10:43:09] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.63, 6.47, 6.74 [10:43:57] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [10:43:58] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 2.020 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [10:44:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [10:44:43] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [10:44:44] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [10:44:46] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [10:44:46] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [10:45:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [10:47:53] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.668 second response time [10:51:07] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:51:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:51:57] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:37] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:02] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:04] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:53:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.25, 6.94, 6.82 [10:54:01] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:57:04] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.50, 7.96, 7.24 [10:58:46] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:59:52] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:01] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:55] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:04:26] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [11:04:37] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 2.423 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [11:04:42] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 16 minutes ago with 0 failures [11:04:50] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [11:04:57] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [11:05:01] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.65, 7.89, 7.53 [11:06:18] PROBLEM - matomo101 NTP time on matomo101 is WARNING: NTP WARNING: Offset -0.1109735072 secs [11:06:44] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [11:08:23] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.03418701887 secs [11:09:47] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [11:11:49] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.924 second response time [11:12:22] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.998 second response time [11:14:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:16:00] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.054 second response time [11:16:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.24, 2.81 [11:16:15] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.273 second response time [11:17:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:18:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.46, 2.94 [11:18:57] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.26, 6.24, 6.75 [11:20:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.56, 3.04 [11:20:19] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:17] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:22:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.06, 3.27, 2.99 [11:22:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.54, 6.73, 6.85 [11:23:40] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:16] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:18] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:54] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.60, 6.33, 6.69 [11:25:22] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:25:33] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:26:49] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:27:06] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.254 second response time [11:27:17] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [11:27:26] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [11:27:34] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.250 second response time [11:27:36] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.303 second response time [11:27:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:42] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:44] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [11:27:50] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:57] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.247 second response time [11:27:58] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.016 second response time [11:27:59] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.012 second response time [11:28:14] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [11:28:18] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:20] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:28] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:28] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:30] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:32] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [11:28:32] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:35] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:39] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:42] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.05581155419 secs [11:28:46] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:56] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [11:29:05] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.332 second response time [11:29:09] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:13] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [11:29:23] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.431 second response time [11:29:30] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.313 second response time [11:29:33] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.331 second response time [11:29:35] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [11:29:39] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.780 second response time [11:29:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.322 second response time [11:29:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [11:29:48] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [11:29:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw102 [11:29:52] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.094 second response time [11:29:52] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.510 second response time [11:29:59] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.014 second response time [11:30:14] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.356 second response time [11:30:17] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [11:30:21] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [11:30:23] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [11:30:24] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [11:30:27] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.373 second response time [11:30:30] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.467 second response time [11:30:30] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.033 second response time [11:30:38] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.317 second response time [11:30:44] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.318 second response time [11:31:08] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [11:31:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw121 [11:31:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [11:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:33:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [11:33:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [11:34:17] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:59] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:05] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [11:36:06] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.040 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [11:36:07] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures [11:36:36] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.003447324038 secs [11:36:41] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.469 second response time [11:38:00] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [11:39:37] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.097 second response time [11:42:47] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.559 second response time [11:43:49] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:49:57] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:51:31] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:00] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:52:34] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:45] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.90, 7.67, 6.69 [11:53:48] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:03] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [11:54:28] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [11:54:29] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:54:35] PROBLEM - cp21 Stunnel HTTP for test101 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [11:54:38] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [11:54:44] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.16, 7.32, 6.67 [11:55:09] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:23] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:55:23] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:37] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [11:55:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 5 backends are down. mw101 mw102 mw111 mw112 mw121 [11:56:02] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [11:56:24] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.791 second response time [11:56:27] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.329 second response time [11:56:31] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw111 mw112 mw121 mw122 [11:56:34] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [11:56:35] RECOVERY - cp21 Stunnel HTTP for test101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.024 second response time [11:57:05] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.877 second response time [11:57:20] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.750 second response time [11:57:35] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [11:58:03] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:58:46] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.247 second response time [11:58:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:22] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:59:52] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:55] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:08] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:20] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:23] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:37] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:41] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:42] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.320 second response time [12:00:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.60, 6.25, 6.45 [12:00:45] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.332 second response time [12:00:55] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [12:00:56] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:01:04] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:06] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [12:01:09] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [12:01:16] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:21] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [12:01:28] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:33] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:37] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:45] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw102 mw122 [12:01:50] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.315 second response time [12:01:50] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time [12:02:04] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [12:02:07] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.763 second response time [12:02:18] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 1.688 second response time [12:02:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [12:02:25] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.853 second response time [12:02:33] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [12:02:39] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.320 second response time [12:03:00] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.334 second response time [12:03:03] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.485 second response time [12:03:04] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.592 second response time [12:03:15] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.318 second response time [12:03:23] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.320 second response time [12:03:29] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.439 second response time [12:03:34] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.602 second response time [12:03:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [12:03:45] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.345 second response time [12:05:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [12:05:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [12:05:57] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [12:06:01] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [12:06:03] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 0.878 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [12:06:16] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [12:09:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:33] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/251e4e2adb89...3e1b25c14cad [12:09:35] [02miraheze/CreateWiki] 07translatewiki 033e1b25c - Localisation updates from https://translatewiki.net. [12:09:36] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/ManageWiki/compare/f86d703c133e...9e32a52e81fc [12:09:38] [02miraheze/ManageWiki] 07translatewiki 039e32a52 - Localisation updates from https://translatewiki.net. [12:09:39] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/9a5bef26fd7f...50296dc1129d [12:09:41] [02miraheze/MirahezeMagic] 07translatewiki 0350296dc - Localisation updates from https://translatewiki.net. [12:11:44] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.01298621297 secs [12:12:22] miraheze/ManageWiki - translatewiki the build has errored. [12:12:33] miraheze/MirahezeMagic - translatewiki the build has errored. [12:12:52] miraheze/CreateWiki - translatewiki the build has errored. [12:13:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:19:35] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.09, 7.43, 6.99 [12:21:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.51, 1.73, 1.15 [12:23:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 4.74, 6.30, 3.78 [12:23:57] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:23:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.62, 1.73, 1.22 [12:25:00] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.90, 4.70, 3.50 [12:25:20] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:34] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.16, 8.16, 7.37 [12:25:40] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:25:46] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:25:59] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.26, 1.61, 1.25 [12:27:33] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.46, 7.91, 7.38 [12:28:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.72, 1.50 [12:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:29:32] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.42, 7.99, 7.46 [12:29:57] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:30:10] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.68, 1.70, 1.52 [12:30:21] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:31:32] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [12:31:34] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.248 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [12:31:56] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:32:25] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.007316797972 secs [12:33:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 5.58, 8.18, 5.58 [12:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:33:31] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.25, 7.74, 7.51 [12:33:38] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [12:33:39] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [12:33:51] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [12:37:00] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.70, 4.69, 4.74 [12:38:37] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 2.765 second response time [12:38:49] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 3.556 second response time [12:42:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.13, 4.84, 3.40 [12:42:48] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.129 second response time [12:42:48] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.503 second response time [12:44:45] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:46:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.11, 3.64, 3.26 [12:47:27] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.95, 7.81, 7.40 [12:47:42] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:48:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.35, 2.85, 3.01 [12:49:28] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.57, 7.63, 7.38 [12:49:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.75, 1.53 [12:51:21] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:46] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.65, 1.52 [12:52:21] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [12:52:35] PROBLEM - cp21 Stunnel HTTP for test101 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [12:52:43] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [12:53:01] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [12:53:40] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw112 mw121 [12:53:51] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:53:53] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:54:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [12:54:22] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.136 second response time [12:54:35] RECOVERY - cp21 Stunnel HTTP for test101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.027 second response time [12:54:43] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.340 second response time [12:55:02] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.987 second response time [12:55:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw101 mw111 mw112 mw122 [12:55:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [12:55:40] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [12:56:01] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:56:14] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:56:43] PROBLEM - matomo101 php-fpm on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:56:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.52, 8.23, 7.58 [12:57:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [12:57:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:58:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [12:58:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.77, 8.14, 7.63 [13:00:36] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 1.273 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [13:01:27] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures [13:01:28] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.044339329 secs [13:01:34] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [13:01:38] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [13:01:40] RECOVERY - matomo101 php-fpm on matomo101 is OK: PROCS OK: 5 processes with command name 'php-fpm7.4' [13:01:44] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [13:02:17] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.210 second response time [13:02:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.48, 3.00 [13:03:22] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [13:04:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.42, 3.03 [13:05:23] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.22, 6.65, 6.85 [13:06:36] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 3.18, 2.99 [13:07:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.19, 6.92, 6.94 [13:07:41] PROBLEM - matomo101 PowerDNS Recursor on matomo101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:08:10] PROBLEM - matomo101 SSH on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:56] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:08:57] PROBLEM - matomo101 conntrack_table_size on matomo101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:09:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [13:09:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.95, 6.34, 6.72 [13:09:36] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:09:44] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:46] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [13:09:48] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:49] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:55] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:09:56] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:02] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:06] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:14] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:46] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:58] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.462 second response time [13:11:15] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [13:11:15] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw101 mw111 mw112 [13:11:24] PROBLEM - matomo101 ferm_active on matomo101 is CRITICAL: connect to address 2a10:6740::6:106 port 5666: No route to hostconnect to host 2a10:6740::6:106 port 5666: No route to host [13:11:25] PROBLEM - matomo101 NTP time on matomo101 is CRITICAL: connect to address 2a10:6740::6:106 port 5666: No route to hostconnect to host 2a10:6740::6:106 port 5666: No route to host [13:11:33] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.328 second response time [13:11:41] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.612 second response time [13:11:41] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.094 second response time [13:11:48] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.006 second response time [13:11:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [13:11:54] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.344 second response time [13:11:54] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [13:11:57] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.334 second response time [13:12:04] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.634 second response time [13:12:10] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.023 second response time [13:12:19] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [13:12:22] PROBLEM - Host matomo101 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:106) [13:12:41] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [13:13:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.34, 8.29, 6.99 [13:13:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [13:13:50] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.343 second response time [13:14:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [13:15:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [13:15:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 8.00, 7.04 [13:19:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 8.37, 7.40 [13:19:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.05, 8.68, 6.69 [13:20:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.75, 8.75, 7.04 [13:21:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 8.18, 7.44 [13:21:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.27, 8.79, 6.97 [13:22:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.74, 8.57, 7.17 [13:25:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.39, 9.16, 7.95 [13:26:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.01, 2.59 [13:27:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.40, 9.23, 8.13 [13:27:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.18, 9.24, 7.71 [13:28:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.14, 2.70 [13:28:01] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [13:28:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 3.02, 2.64 [13:28:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.64, 3.33, 2.58 [13:29:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.71, 9.17, 7.89 [13:30:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.34, 2.83 [13:30:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.12, 2.85, 2.50 [13:31:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.19, 9.37, 8.40 [13:32:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.04, 2.78 [13:32:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.50, 9.00, 7.97 [13:33:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 8.92, 7.76 [13:35:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.51, 9.16, 8.55 [13:35:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.70, 8.39, 7.99 [13:35:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.48, 3.03 [13:35:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.74, 9.14, 8.27 [13:37:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.71, 8.27, 7.78 [13:37:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.89, 8.18, 8.04 [13:38:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.44, 3.08 [13:39:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.88, 3.36, 3.09 [13:40:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.96, 3.12, 3.00 [13:40:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.47, 9.30, 8.72 [13:42:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.53, 9.20, 8.74 [13:44:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.51, 10.09, 9.12 [13:45:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.69, 8.80, 8.24 [13:47:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.64, 8.38, 8.14 [13:48:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.77, 9.48, 9.12 [13:51:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.08, 9.05, 8.46 [13:53:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.51, 8.94, 8.50 [13:53:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.32, 6.91, 5.67 [13:54:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.42, 9.73, 9.23 [13:55:07] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.17, 6.67, 5.74 [13:56:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.11, 9.06, 9.04 [13:56:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.68, 9.99, 8.72 [13:57:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.28, 8.28, 8.33 [13:58:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.70, 3.28 [13:58:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.69, 9.28 [13:58:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.80, 9.33, 8.65 [14:00:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 8.57, 8.24 [14:00:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.02, 8.91, 9.06 [14:00:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.52, 10.12, 9.13 [14:02:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.89, 3.38, 3.25 [14:02:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.54, 9.09, 8.46 [14:04:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.65, 9.09, 8.55 [14:04:35] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:43] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [14:04:45] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [14:04:47] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:47] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [14:04:49] PROBLEM - cp30 Stunnel HTTP for mon111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [14:04:52] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [14:04:56] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [14:05:01] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:12] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:05:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:27] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:29] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [14:05:29] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:34] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [14:05:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.25, 8.64, 7.12 [14:05:43] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [14:05:51] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.62, 2.29, 1.37 [14:05:58] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:08] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.12, 8.91, 7.79 [14:06:14] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:19] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:20] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:26] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:28] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:30] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:31] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.022 second response time [14:06:40] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.318 second response time [14:06:42] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.319 second response time [14:06:43] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.306 second response time [14:06:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.59, 9.69, 9.30 [14:06:50] RECOVERY - cp30 Stunnel HTTP for mon111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 33915 bytes in 0.339 second response time [14:06:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.44, 9.17, 9.13 [14:06:55] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:56] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.231 second response time [14:06:58] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.002 second response time [14:06:58] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.575 second response time [14:07:15] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.913 second response time [14:07:15] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [14:07:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:07:23] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [14:07:28] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.545 second response time [14:07:33] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.320 second response time [14:07:41] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.501 second response time [14:07:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.08, 8.58, 7.27 [14:07:49] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw102 [14:07:50] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.028 second response time [14:07:52] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.306 second response time [14:07:53] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.023 second response time [14:07:57] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.82, 1.30 [14:08:00] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.599 second response time [14:08:12] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw101 mw111 mw112 mw121 mw122 [14:08:12] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [14:08:16] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.266 second response time [14:08:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.68, 8.06, 8.27 [14:08:23] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.328 second response time [14:08:28] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.699 second response time [14:08:31] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.988 second response time [14:08:32] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.805 second response time [14:08:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.30, 9.07, 9.14 [14:08:58] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.734 second response time [14:09:31] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.364 second response time [14:09:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.65, 7.82, 7.16 [14:09:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.40, 1.20 [14:10:07] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [14:10:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.47, 7.88, 7.70 [14:10:45] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.027 second response time [14:10:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 8.10, 8.50 [14:10:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.35, 8.92, 8.96 [14:11:49] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [14:12:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.98, 4.33, 3.49 [14:12:05] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [14:12:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 8.64, 8.85 [14:12:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.74, 9.03, 9.01 [14:14:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 8.80, 8.89 [14:14:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 8.89, 8.75 [14:16:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.95, 9.03, 8.55 [14:16:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.83, 9.20, 9.02 [14:17:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 4.39, 3.61 [14:18:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 9.17, 9.03 [14:20:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.20, 9.37, 9.00 [14:21:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.82, 3.30 [14:22:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.21, 9.44, 9.15 [14:22:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.98, 3.68 [14:23:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.67, 3.31 [14:24:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.08, 8.54, 8.85 [14:26:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 3.97, 3.71 [14:26:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.55, 7.60, 8.48 [14:27:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.37, 4.42, 3.66 [14:28:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.75, 3.66 [14:28:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.27, 8.25, 8.50 [14:28:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.74, 9.75, 9.50 [14:29:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.21, 9.24, 8.95 [14:30:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 4.16, 3.81 [14:30:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.11, 9.69, 9.50 [14:31:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.63, 9.02, 8.91 [14:32:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 9.34, 9.39 [14:34:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.15, 10.86, 9.95 [14:35:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.38, 9.06 [14:36:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.89, 9.41, 8.65 [14:37:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.86, 3.77 [14:37:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.63, 8.49, 8.76 [14:38:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.10, 9.52, 9.61 [14:39:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 4.00, 3.83 [14:39:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.13, 8.96 [14:41:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.89, 3.82 [14:42:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.66, 8.77, 8.73 [14:42:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.24, 10.06, 9.05 [14:43:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.25, 9.52, 9.23 [14:43:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.63, 3.99 [14:44:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.59, 9.31, 8.89 [14:46:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 10.16, 9.76 [14:47:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.70, 3.93 [14:48:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.87, 9.81, 9.67 [14:49:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.64, 3.88 [14:50:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.36, 9.27, 8.94 [14:50:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.44, 10.13, 9.81 [14:51:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 4.12, 4.03 [14:53:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.47, 10.22, 9.56 [14:54:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.77, 3.60 [14:55:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.87, 3.97 [14:56:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.71, 9.54, 9.75 [14:56:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.34, 3.46, 3.50 [14:58:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.43, 9.55, 9.71 [14:58:49] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4277 MB (10% inode=96%); [15:00:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.41, 9.92, 9.58 [15:00:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.94, 8.98, 9.49 [15:00:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.13, 3.36 [15:01:01] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13866 MB (10% inode=99%); [15:01:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.21, 9.66, 9.66 [15:02:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.24, 9.29, 9.52 [15:03:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.96, 4.02, 3.96 [15:04:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 9.39, 9.56 [15:05:11] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4296 MB (10% inode=96%); [15:05:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.80, 3.88 [15:12:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.73, 9.41, 9.29 [15:16:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.02, 8.00, 8.49 [15:16:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.96, 9.91, 9.53 [15:16:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 9.03, 9.07 [15:17:01] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 32713 MB (24% inode=99%); [15:18:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.04, 8.69, 8.95 [15:19:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.64, 8.66, 8.03 [15:19:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.90, 10.28, 9.47 [15:21:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 3.52, 3.18 [15:21:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.35, 9.10, 9.13 [15:21:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.64, 3.60 [15:23:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.92, 3.23, 3.11 [15:23:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.16, 7.86, 7.89 [15:23:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:23:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.02, 2.93, 3.35 [15:24:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.14, 9.55, 9.35 [15:24:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.17, 7.89, 8.48 [15:26:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.30, 3.26, 4.00 [15:26:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.02, 9.35, 9.31 [15:30:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.91, 9.46, 9.31 [15:33:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.99, 9.19, 8.81 [15:35:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.15, 9.66, 9.05 [15:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.06, 9.05, 8.41 [15:37:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.27, 9.28, 9.01 [15:38:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.38, 3.55 [15:39:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.91, 3.88, 3.34 [15:40:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.59, 3.62 [15:40:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.44, 8.25, 8.22 [15:40:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.79, 3.26 [15:41:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.91, 3.42 [15:42:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.66, 3.63 [15:42:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.55, 3.24 [15:43:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.60, 9.39, 9.12 [15:44:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.82, 3.41, 3.55 [15:44:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.76, 8.84, 8.54 [15:44:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.18, 3.14 [15:45:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 9.56, 9.22 [15:45:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.23, 9.17, 8.93 [15:46:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.11, 8.49, 8.44 [15:50:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.63, 9.65, 8.94 [15:51:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 2.90, 3.22 [15:52:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.00, 3.31 [15:53:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.59, 9.98, 9.45 [15:55:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.51, 9.09, 9.05 [15:57:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 9.15, 9.07 [15:57:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.12, 10.15, 9.65 [15:59:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.57, 9.00, 9.30 [15:59:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.44, 3.35 [16:00:32] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [16:00:32] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:00:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.65, 9.74, 9.49 [16:00:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 9.50, 9.90 [16:01:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.72, 9.95, 9.60 [16:02:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.60, 10.02, 10.05 [16:03:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 9.87, 9.31 [16:03:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.81, 3.36 [16:03:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.32, 3.34, 3.34 [16:04:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.92, 10.62, 9.87 [16:04:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.60, 9.28, 9.79 [16:05:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.64, 3.35 [16:05:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.01, 9.06, 9.38 [16:06:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.72, 9.83, 9.67 [16:07:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.99, 3.36, 3.27 [16:09:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.56, 9.77, 9.54 [16:10:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.41, 9.89, 9.90 [16:11:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.76, 9.60, 9.50 [16:12:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.52, 9.61, 9.79 [16:14:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.99, 10.28, 10.00 [16:15:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 9.40, 9.72 [16:18:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 9.66, 9.84 [16:23:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.07, 10.16, 9.95 [16:23:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.75, 9.10, 9.09 [16:24:32] !log [@mwtask111] starting deploy of {'config': True} to all [16:24:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.95, 9.72, 9.68 [16:24:47] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 15s [16:24:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:25:06] That's concerning [16:25:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.33, 8.70, 8.94 [16:26:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.58, 8.81, 9.35 [16:28:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.49, 3.43 [16:28:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.99, 9.83, 9.91 [16:30:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.07, 3.29 [16:30:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.44, 7.69, 8.39 [16:30:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.37, 10.17, 10.03 [16:33:01] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14136 MB (10% inode=99%); [16:36:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.85, 3.50 [16:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.21, 9.26, 8.83 [16:37:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.09, 2.86 [16:38:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.31, 9.32, 9.85 [16:39:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.36, 4.02, 3.21 [16:40:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.49, 4.57, 3.64 [16:40:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.20, 8.07, 8.47 [16:40:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.22, 9.49, 9.16 [16:41:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.74, 7.97, 8.50 [16:42:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.87, 3.67 [16:42:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.79, 3.46 [16:42:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 9.12, 9.06 [16:42:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.11, 3.18, 3.05 [16:44:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.31, 8.87, 8.72 [16:44:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.90, 9.70, 9.27 [16:45:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.69, 8.80, 8.70 [16:46:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.20, 3.30 [16:46:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.13, 9.82, 9.74 [16:47:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 3.11, 3.40 [16:48:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.21, 9.34, 8.93 [16:48:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.50, 9.48, 9.64 [16:50:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.41, 8.94, 8.81 [16:50:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.67, 9.87, 9.76 [16:52:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 9.65, 9.58 [16:53:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.48, 9.03 [16:54:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 9.89, 9.67 [16:54:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.81, 9.75, 9.81 [16:55:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.68, 8.80, 8.84 [16:56:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.41, 9.87, 9.71 [16:58:36] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [16:59:01] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7500 MB (5% inode=99%); [17:00:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.37, 3.31 [17:01:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:01:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.40, 9.52, 9.12 [17:02:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.37, 3.33 [17:02:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.45, 9.70, 9.72 [17:02:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.60, 8.92, 7.84 [17:03:01] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 24027 MB (18% inode=99%); [17:03:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.49, 9.45, 9.14 [17:04:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.02, 9.37, 8.97 [17:04:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.30, 8.39, 7.78 [17:04:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 9.73, 9.64 [17:05:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:05:33] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.33, 5.35, 3.95 [17:06:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 9.32, 9.00 [17:06:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.51, 9.34, 9.62 [17:07:27] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.49, 4.02, 3.63 [17:11:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 9.13, 9.00 [17:13:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.07, 8.82, 8.89 [17:14:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 10.16, 9.85 [17:16:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.07, 1.37, 1.07 [17:17:03] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.11, 6.07, 5.14 [17:17:38] [02mw-config] 07ugochimobi opened pull request 03#4510: T8932: ApprovedRevs configs - 13https://github.com/miraheze/mw-config/pull/4510 [17:17:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.23, 9.23, 9.02 [17:18:19] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.32, 1.35, 1.10 [17:18:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.30, 9.08, 9.53 [17:18:49] miraheze/mw-config - ugochimobi the build passed. [17:19:03] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.62, 5.49, 5.05 [17:19:28] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.50, 1.25 [17:21:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.36, 1.23 [17:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:25:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 9.99, 9.67 [17:26:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.00, 9.06, 9.13 [17:28:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.48, 3.21 [17:28:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:28:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.68, 9.55, 9.94 [17:28:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 8.80, 9.02 [17:30:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.26, 3.16 [17:32:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 9.79, 9.91 [17:34:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.14, 8.96 [17:34:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.31, 9.07, 9.62 [17:34:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.64, 10.42, 9.60 [17:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.33, 8.00, 8.56 [17:36:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.60, 9.08, 9.22 [17:39:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.31, 7.27, 8.31 [17:40:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.76, 9.64, 9.56 [17:42:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.02, 9.69, 9.38 [17:44:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 9.67, 9.41 [17:45:01] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13974 MB (10% inode=99%); [17:46:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.55, 3.33 [17:46:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.65, 9.74, 9.71 [17:48:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.06, 3.37, 3.29 [17:48:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.46, 10.07, 9.16 [17:48:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.25, 9.90, 9.52 [17:50:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.26, 8.60, 8.54 [17:50:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 9.73, 9.15 [17:50:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.58, 9.94, 9.59 [17:52:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.57, 3.41 [17:52:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.22, 9.72, 9.20 [17:53:01] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 20292 MB (15% inode=99%); [17:54:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 3.95, 3.55 [17:54:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.88, 8.40, 8.48 [17:54:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.67, 9.60, 9.21 [17:54:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.51, 9.85, 9.61 [17:58:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 4.16, 3.46 [17:58:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.92, 9.30, 8.88 [17:59:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.94, 4.06, 3.49 [18:00:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.92, 3.46 [18:02:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.24, 9.49, 9.02 [18:02:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.58, 9.37, 9.22 [18:02:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.68, 9.76, 9.48 [18:04:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.64, 9.27, 9.00 [18:04:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.40, 9.39, 9.24 [18:04:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 9.32, 9.35 [18:05:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.37, 3.68, 3.55 [18:06:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 9.47, 9.76 [18:07:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.82, 3.56 [18:09:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.90, 3.97 [18:09:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 3.13, 3.33 [18:10:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.19, 9.48, 9.28 [18:11:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.73, 3.59 [18:12:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.11, 9.57, 9.20 [18:12:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 8.97, 9.12 [18:12:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.20, 3.41 [18:14:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.74, 9.23, 8.96 [18:14:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.89, 9.57, 9.24 [18:14:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.31, 9.49, 9.55 [18:14:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 3.85, 3.62 [18:16:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.57, 8.33, 8.67 [18:16:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.46, 9.23, 9.13 [18:18:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.78, 10.11, 9.26 [18:18:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.82, 3.69 [18:18:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.97, 10.59, 9.72 [18:18:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.95, 8.54, 8.88 [18:18:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.85, 9.49, 9.58 [18:19:01] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13634 MB (10% inode=99%); [18:19:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 3.90, 3.75 [18:20:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 9.79, 9.26 [18:20:11] PROBLEM - mw122 NTP time on mw122 is WARNING: NTP WARNING: Offset 0.1477572918 secs [18:20:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.00, 4.09, 3.81 [18:20:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.14, 9.71, 9.52 [18:20:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.13, 9.69, 9.63 [18:21:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 3.98, 3.51 [18:21:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.89, 3.78 [18:22:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.13, 10.11, 9.69 [18:23:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.68, 3.46 [18:23:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.92, 3.80 [18:23:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 9.44, 9.19 [18:23:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.91, 1.61, 1.22 [18:24:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 3.42, 3.63 [18:24:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.08, 9.49, 9.53 [18:24:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.78, 9.34, 9.56 [18:25:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 3.25, 3.33 [18:25:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.37, 3.61 [18:25:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.97, 9.21, 9.14 [18:25:59] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.45, 1.52, 1.24 [18:26:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.87, 2.87, 3.40 [18:29:01] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 55634 MB (41% inode=99%); [18:30:35] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [18:30:35] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:30:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.03, 7.90, 8.48 [18:34:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.61, 9.46, 9.41 [18:36:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 8.99, 9.23 [18:37:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 3.10, 3.35 [18:38:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.32, 3.35 [18:39:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.74, 9.31, 9.01 [18:41:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.00, 9.46, 9.12 [18:42:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.36, 3.35 [18:44:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.62, 7.98, 8.48 [18:48:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:48:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.77, 9.33, 8.85 [18:49:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 9.73, 9.32 [18:50:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 9.20, 8.86 [18:51:11] RECOVERY - mw122 NTP time on mw122 is OK: NTP OK: Offset 0.01532387733 secs [18:54:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:54:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.09, 9.70, 9.16 [18:55:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.88, 9.94, 9.61 [18:55:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.48, 2.18, 1.70 [18:56:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.18, 9.27, 9.07 [18:58:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.10, 8.95 [18:59:37] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:00:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 4.10, 3.58 [19:00:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.98, 9.20, 8.99 [19:02:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 3.90, 3.24 [19:04:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.68, 9.61, 9.31 [19:06:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.18, 9.13, 9.18 [19:08:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.77, 1.19 [19:08:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.73, 3.22 [19:10:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.45, 1.14 [19:10:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.82, 3.60 [19:10:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.65, 3.24 [19:10:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.10, 8.02, 8.48 [19:12:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.85, 8.92 [19:12:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.80, 7.69, 8.46 [19:13:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 7.11, 6.35, 3.36 [19:14:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.41, 9.35, 8.83 [19:14:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.85, 3.39 [19:14:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 8.47, 8.59 [19:15:00] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.22, 4.98, 3.21 [19:15:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.85, 1.78, 2.00 [19:16:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.92, 1.53, 1.03 [19:16:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.54, 3.47 [19:16:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.95, 8.74, 8.74 [19:18:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.75, 3.24, 3.37 [19:18:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.54, 1.15, 0.95 [19:18:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.59, 3.40 [19:18:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.05, 8.95, 8.73 [19:19:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.29, 9.75, 9.09 [19:19:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.08, 1.91, 2.00 [19:20:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.74, 9.88, 9.22 [19:21:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.15, 3.87, 3.56 [19:21:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.49, 1.77, 1.94 [19:22:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.14, 3.96, 3.55 [19:22:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.53, 9.71, 9.24 [19:22:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.18, 9.12, 8.89 [19:23:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 3.31, 3.39 [19:24:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.83, 3.06, 3.89 [19:24:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.06, 3.38, 3.37 [19:24:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.48, 10.38, 9.54 [19:25:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.19, 9.66, 9.37 [19:25:59] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.35, 1.07, 1.62 [19:26:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.20, 10.08, 9.35 [19:27:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.97, 10.13, 9.57 [19:28:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.99, 9.75, 9.33 [19:28:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 9.61, 9.43 [19:29:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.43, 9.80, 9.52 [19:30:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.97, 10.57, 9.67 [19:30:36] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [19:30:36] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:30:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.32, 9.74, 9.49 [19:31:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.51, 10.07, 9.65 [19:31:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.78, 9.75, 9.50 [19:32:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.87, 9.42, 9.41 [19:33:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.39, 9.36, 9.44 [19:34:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.81, 2.58, 3.27 [19:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:34:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 10.29, 9.72 [19:34:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.41, 10.16, 9.41 [19:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.88, 9.03, 9.34 [19:36:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.86, 9.78, 9.35 [19:37:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 8.81, 9.07 [19:38:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:39:26] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 2.99, 6.24, 4.82 [19:40:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.19, 10.06, 9.57 [19:41:20] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.52, 4.69, 4.42 [19:42:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.05, 9.83, 9.56 [19:44:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.85, 10.45, 9.81 [19:45:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.05, 9.81, 9.61 [19:46:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 3.80, 3.46 [19:46:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.85, 4.07, 3.59 [19:46:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.59, 7.71, 8.46 [19:47:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.31, 7.67, 8.43 [19:48:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.63, 3.43 [19:49:30] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/a52c5ff38550...6c424f8c7621 [19:49:31] [02miraheze/puppet] 07JohnFLewis 036c424f8 - matomo: use db112 [19:49:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.03, 9.53, 9.48 [19:50:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.80, 3.40, 3.38 [19:50:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.73, 3.58 [19:51:50] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.466 second response time [19:52:12] RECOVERY - Host matomo101 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [19:52:12] PROBLEM - matomo101 Redis Process on matomo101 is CRITICAL: connect to address 2a10:6740::6:106 port 5666: No route to hostconnect to host 2a10:6740::6:106 port 5666: No route to host [19:52:12] RECOVERY - matomo101 SSH on matomo101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [19:52:12] PROBLEM - matomo101 Puppet on matomo101 is WARNING: WARNING: Puppet last ran 7 hours ago [19:52:12] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.18, 0.64, 0.22 [19:52:12] RECOVERY - matomo101 ferm_active on matomo101 is OK: OK ferm input default policy is set [19:52:37] RECOVERY - matomo101 conntrack_table_size on matomo101 is OK: OK: nf_conntrack is 0 % full [19:52:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 9.08, 8.77 [19:52:44] RECOVERY - matomo101 PowerDNS Recursor on matomo101 is OK: DNS OK: 0.273 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [19:52:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.78, 9.82, 9.84 [19:52:48] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.484 second response time [19:52:57] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 0.265 second response time [19:53:14] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.554 second response time [19:53:20] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [19:53:36] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.004514217377 secs [19:53:56] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:54:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 4.16, 3.70 [19:54:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 3.18, 3.39 [19:54:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.99, 8.31, 7.58 [19:55:36] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 743 MB (7% inode=80%); [19:55:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.18, 9.26, 8.91 [19:55:50] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.867 second response time [19:56:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.99, 3.71 [19:56:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.15, 7.75, 7.45 [19:56:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.42, 9.32, 8.93 [19:56:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.83, 10.57, 10.10 [19:57:37] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 482 MB (5% inode=80%); [19:57:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 9.24, 8.96 [19:58:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.70, 5.43, 4.25 [19:59:11] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:37] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 734 MB (7% inode=80%); [19:59:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [19:59:58] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:08] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.98, 4.38, 3.70 [20:00:38] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.18, 8.86, 8.89 [20:00:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.39, 9.35, 9.71 [20:01:16] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.612 second response time [20:01:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.27, 7.41, 8.44 [20:01:37] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 475 MB (5% inode=80%); [20:01:57] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.255 second response time [20:02:09] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 5.265 second response time [20:02:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.88, 3.60 [20:02:38] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.901 second response time [20:02:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.37, 9.28, 9.02 [20:03:37] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 737 MB (7% inode=80%); [20:03:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.01, 9.92, 9.28 [20:04:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.90, 3.19, 3.38 [20:04:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.39, 9.96, 9.85 [20:05:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.76, 8.59, 8.74 [20:05:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.96, 9.48, 9.19 [20:06:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.87, 3.98 [20:06:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.32, 9.42, 9.66 [20:08:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 4.43, 4.18 [20:08:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 9.24, 9.18 [20:09:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.81, 7.18, 8.18 [20:09:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.21, 9.33, 9.14 [20:11:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.61, 5.00, 2.70 [20:11:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.28, 8.88, 9.00 [20:11:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.63, 3.45 [20:12:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.30, 4.39, 3.68 [20:17:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.26, 7.83, 8.48 [20:18:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.62, 7.57, 8.38 [20:18:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.80, 9.32, 9.24 [20:19:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.06, 8.76, 8.54 [20:19:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.62, 1.59, 1.90 [20:21:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.63, 1.36 [20:21:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.42, 1.90, 1.97 [20:21:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 8.99, 8.84 [20:21:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.92, 3.88 [20:23:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.81, 8.37, 8.44 [20:23:10] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.47, 1.33 [20:23:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.92, 1.47, 1.80 [20:23:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.67, 3.70 [20:23:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 4.26, 4.01 [20:24:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.58, 8.54, 8.51 [20:25:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 9.84, 9.22 [20:25:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.92, 3.92 [20:26:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.45, 7.99, 8.32 [20:27:17] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.28, 1.27, 1.64 [20:27:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.08, 4.54, 3.98 [20:27:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 4.52, 7.91, 8.60 [20:27:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 4.05, 3.96 [20:28:40] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [20:28:40] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:28:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.93, 9.37, 9.47 [20:29:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.61, 3.82 [20:30:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.02, 8.21, 8.21 [20:32:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.80, 8.33, 8.28 [20:33:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.67, 3.77 [20:34:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.44, 8.70, 8.05 [20:34:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.75, 9.57 [20:36:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.32, 8.34, 8.01 [20:36:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.09, 9.43, 9.47 [20:39:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 8.92, 8.75 [20:39:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 4.18, 3.93 [20:40:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.88, 3.04, 3.40 [20:41:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.20, 9.10, 8.84 [20:41:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.54, 9.12, 8.70 [20:43:36] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 405 MB (4% inode=80%); [20:43:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.60, 9.52, 8.89 [20:44:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.55, 3.51 [20:45:36] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 757 MB (8% inode=80%); [20:46:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.37, 3.44 [20:47:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.87, 9.88, 9.20 [20:50:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.70, 3.27, 3.40 [20:51:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.62, 9.76, 9.28 [20:52:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 9.21, 9.04 [20:53:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.28, 9.62, 9.29 [20:54:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 8.91, 8.94 [20:55:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.19, 9.10, 8.45 [20:55:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.00, 10.41, 9.64 [20:56:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.89, 3.60 [20:57:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 8.93, 8.46 [20:57:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.18, 9.87, 9.53 [20:57:41] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [20:58:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.97, 3.67 [20:59:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.05, 9.25, 8.63 [20:59:30] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.98, 10.02, 9.63 [21:00:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.71, 9.22, 8.98 [21:01:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.32, 9.15, 8.69 [21:01:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.81, 8.02, 7.74 [21:01:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 9.95, 9.65 [21:02:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.95, 9.20, 9.00 [21:02:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.77, 4.54, 3.91 [21:03:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.60, 7.54, 7.60 [21:04:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.29, 9.79, 9.25 [21:04:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.88, 3.74 [21:05:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.38, 10.20, 9.14 [21:06:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.91, 9.16, 9.09 [21:07:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.33, 9.43, 8.99 [21:09:11] PROBLEM - cp31 Disk Space on cp31 is CRITICAL: DISK CRITICAL - free space: / 2327 MB (5% inode=96%); [21:11:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.61, 7.67, 8.40 [21:12:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.21, 3.92, 3.79 [21:14:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.75, 3.74 [21:16:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 4.06, 3.86 [21:17:11] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 2370 MB (6% inode=96%); [21:17:36] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 515 MB (5% inode=80%); [21:18:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.39, 1.06 [21:18:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.21, 9.29, 8.99 [21:18:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.73, 3.76 [21:19:11] PROBLEM - cp31 Disk Space on cp31 is CRITICAL: DISK CRITICAL - free space: / 2301 MB (5% inode=96%); [21:19:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.39, 9.50, 9.20 [21:19:36] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 788 MB (8% inode=80%); [21:20:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.20, 1.03 [21:20:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.54, 1.35 [21:20:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.54, 8.79, 8.84 [21:21:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.28, 9.00, 9.06 [21:22:29] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.17, 1.38, 1.31 [21:22:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.28, 4.29, 3.93 [21:23:36] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 534 MB (5% inode=80%); [21:25:37] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 795 MB (8% inode=80%); [21:26:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.91, 3.84 [21:28:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.45, 9.18, 8.94 [21:29:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.75, 3.99 [21:30:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 4.05, 3.92 [21:31:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.73, 3.94, 4.04 [21:32:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.52, 3.74 [21:34:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 4.11, 3.94 [21:35:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 9.25, 9.12 [21:36:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 4.00, 3.93 [21:36:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.64, 7.69, 8.31 [21:37:59] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.820 second response time [21:39:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.66, 3.99 [21:40:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.73, 3.79 [21:42:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.59, 9.01, 8.62 [21:43:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.05, 9.76, 8.82 [21:43:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.74, 3.93 [21:44:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.27, 8.68, 8.57 [21:45:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.03, 9.37, 8.79 [21:45:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.67, 3.89 [21:46:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.81, 3.89 [21:47:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.92, 3.95 [21:48:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.08, 9.65, 8.94 [21:48:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 9.41, 9.22 [21:48:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.50, 10.19, 9.17 [21:49:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.96, 3.97 [21:49:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.70, 1.38 [21:50:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.94, 9.13, 9.15 [21:50:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 9.02, 8.79 [21:50:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.50, 9.95, 9.21 [21:51:57] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.51, 1.35 [21:54:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 10.16, 9.43 [21:55:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.85, 3.91 [21:56:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 9.19, 9.14 [21:56:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.11, 9.76, 9.17 [21:56:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.41, 9.24 [21:57:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.58, 3.79 [21:58:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.70, 9.02, 9.12 [22:00:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.29, 3.86 [22:00:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 9.61, 9.24 [22:00:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.83, 9.76, 9.38 [22:00:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.98, 2.66, 3.25 [22:02:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.91, 4.21, 2.38 [22:03:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.55, 3.95, 3.81 [22:04:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.36, 3.61, 3.81 [22:04:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.19, 8.99, 9.01 [22:05:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.85, 3.79 [22:06:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.25, 3.65 [22:06:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.94, 8.96, 9.01 [22:08:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.34, 1.55, 1.74 [22:08:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.01, 9.21, 9.51 [22:10:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.20, 1.09, 1.54 [22:10:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.73, 9.71, 9.38 [22:11:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.30, 3.51 [22:12:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.33, 3.52 [22:12:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.05, 7.66, 8.38 [22:13:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.45, 3.54 [22:14:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.74, 3.67 [22:14:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.80, 9.18, 9.30 [22:16:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.34, 9.61, 9.44 [22:17:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.68, 3.61 [22:18:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.37, 8.02, 8.46 [22:18:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 9.76, 9.53 [22:19:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.87, 3.71 [22:20:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.61, 8.85, 8.64 [22:20:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.73, 7.64, 8.46 [22:22:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.02, 3.17, 3.38 [22:24:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.03, 10.37, 9.81 [22:24:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.30, 8.96, 8.77 [22:25:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 2.91, 3.36 [22:26:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 8.58, 8.50 [22:26:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 9.93, 9.70 [22:28:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.14, 9.25, 8.77 [22:28:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.28, 9.53, 9.10 [22:30:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.30, 8.63, 8.61 [22:34:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.57, 9.43, 9.43 [22:34:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.88, 9.68, 9.19 [22:40:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.30, 9.24, 8.90 [22:40:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.22, 9.66, 9.41 [22:42:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.17, 8.29, 8.47 [22:42:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.95, 8.53, 8.67 [22:44:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 9.92, 9.92 [22:46:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.97, 10.06, 9.98 [22:48:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.59, 9.60, 9.83 [22:50:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.91, 9.80, 9.85 [22:50:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.37, 9.84, 8.95 [22:52:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.15, 9.10, 8.81 [22:52:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.87, 9.40, 9.71 [22:52:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 9.43, 8.90 [22:53:53] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: DISK CRITICAL - free space: / 2342 MB (5% inode=96%); [22:54:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.40, 8.48, 8.63 [22:57:32] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [22:57:32] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:58:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.21, 3.43, 3.35 [22:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:58:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.48, 9.37, 8.99 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.26, 3.99, 3.53 [23:00:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.20, 9.23, 9.01 [23:00:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.28, 7.46, 8.28 [23:00:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:04:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.58, 3.49 [23:06:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.77, 7.99, 8.42 [23:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:08:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.61, 7.15, 8.29 [23:10:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.97, 7.73, 8.39 [23:12:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.92, 3.46 [23:12:33] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.81, 1.50 [23:12:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.87, 8.37, 8.58 [23:14:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 3.00, 3.29 [23:14:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.75, 3.45 [23:14:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.63, 7.71, 8.31 [23:16:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.35, 3.71 [23:17:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.64, 3.75, 3.09 [23:18:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.94, 3.63 [23:18:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.51, 3.08 [23:19:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.78, 1.62, 1.17 [23:21:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.87, 1.96, 1.34 [23:22:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.35, 3.14 [23:23:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.07, 2.96, 3.29 [23:26:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 3.75, 3.31 [23:27:32] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [23:27:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.39, 3.40 [23:28:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 4.18, 3.58 [23:28:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.82, 3.40 [23:29:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:29:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 3.12, 3.30 [23:31:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.60, 1.98, 1.74 [23:33:20] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1998s [23:33:55] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 2032s [23:33:59] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.89, 1.62, 1.64 [23:34:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.91, 3.73 [23:34:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.80, 3.34, 3.36 [23:36:09] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.15, 1.75, 1.95 [23:36:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.14, 9.02, 8.48 [23:40:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.81, 4.23, 3.78 [23:42:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.65, 3.63 [23:43:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.40, 8.40, 7.91 [23:44:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.91, 3.92, 3.72 [23:44:01] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.41, 1.68 [23:44:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.33, 8.36, 8.48 [23:45:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.76, 8.06, 7.84 [23:48:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.64, 8.57, 7.77 [23:49:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 9.35, 8.40 [23:49:53] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.73, 1.71 [23:51:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.30, 8.54, 8.23 [23:51:51] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.53, 1.63 [23:52:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.86, 3.83 [23:52:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.68, 9.34, 8.26 [23:52:56] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.92, 1.60, 1.20 [23:53:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.26, 8.68, 8.16 [23:53:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.63, 8.48, 8.25 [23:54:56] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.28, 1.70, 1.27 [23:55:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.31, 8.06, 7.99 [23:56:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.94, 1.36, 1.20 [23:58:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.24, 8.26, 8.10 [23:59:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki