[00:00:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.60, 9.84, 9.43 [00:01:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.30, 10.96, 10.15 [00:02:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.97, 3.17, 3.20 [00:03:04] [02puppet] 07Universal-Omega edited pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:05:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.10, 9.68, 9.95 [00:05:42] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 217 failures. Last run 4 minutes ago with 217 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:06:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.99, 8.94, 9.20 [00:08:31] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:08:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.72, 9.85, 9.93 [00:08:55] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:09:26] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.08, 8.98, 9.13 [00:10:47] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:10:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.01, 8.93, 9.59 [00:11:07] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6ee853b1a089...2e9ab7a97db4 [00:11:08] [02miraheze/puppet] 07JohnFLewis 032e9ab7a - varnish: do not normalise static images (test?) [00:12:11] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:15:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.57, 9.84, 9.58 [00:17:18] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:17:36] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:21:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.62, 3.97, 3.49 [00:22:05] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:22:51] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:23:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.29, 8.96, 9.85 [00:24:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.74, 3.48 [00:27:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.05, 4.59, 3.83 [00:29:01] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:29:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.67, 7.35, 8.31 [00:29:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.25, 2.92 [00:29:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 9.55, 9.93 [00:30:38] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:30:51] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [00:30:53] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:31:47] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:32:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 2.83, 2.81 [00:32:40] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:32:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.40, 10.22, 10.14 [00:33:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.92, 3.83 [00:33:31] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:34:54] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:35:15] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:35:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.69, 9.46, 9.90 [00:36:17] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:37:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.25, 7.86, 8.46 [00:37:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.51, 3.99 [00:39:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageTriage-efbae63 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/85b2ce4adedc [00:39:06] [02miraheze/mediawiki] 07dependabot[bot] 0385b2ce4 - Bump extensions/PageTriage from `0e564dd` to `efbae63` [00:39:08] [02mediawiki] 07dependabot[bot] opened pull request 03#5024: Bump extensions/PageTriage from `0e564dd` to `efbae63` - 13https://github.com/miraheze/mediawiki/pull/5024 [00:39:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageTriage-efbae63 - 13https://github.com/miraheze/mediawiki [00:39:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SocialProfile-a55e1fc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0ea78c750701 [00:39:12] [02miraheze/mediawiki] 07dependabot[bot] 030ea78c7 - Bump extensions/SocialProfile from `f407fdb` to `a55e1fc` [00:39:14] [02mediawiki] 07dependabot[bot] opened pull request 03#5025: Bump extensions/SocialProfile from `f407fdb` to `a55e1fc` - 13https://github.com/miraheze/mediawiki/pull/5025 [00:39:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-a55e1fc - 13https://github.com/miraheze/mediawiki [00:39:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiLove-a3a8629 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/865dc8ba2e93 [00:39:18] [02miraheze/mediawiki] 07dependabot[bot] 03865dc8b - Bump extensions/WikiLove from `a89bf17` to `a3a8629` [00:39:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiLove-a3a8629 - 13https://github.com/miraheze/mediawiki [00:39:21] [02mediawiki] 07dependabot[bot] opened pull request 03#5026: Bump extensions/WikiLove from `a89bf17` to `a3a8629` - 13https://github.com/miraheze/mediawiki/pull/5026 [00:39:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/erudite-909572a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e82fc21fde51 [00:39:24] [02miraheze/mediawiki] 07dependabot[bot] 03e82fc21 - Bump skins/erudite from `6c9913d` to `909572a` [00:39:26] [02mediawiki] 07dependabot[bot] opened pull request 03#5027: Bump skins/erudite from `6c9913d` to `909572a` - 13https://github.com/miraheze/mediawiki/pull/5027 [00:39:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/erudite-909572a - 13https://github.com/miraheze/mediawiki [00:39:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-5d7f8c2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1024b8fc85a5 [00:39:30] [02miraheze/mediawiki] 07dependabot[bot] 031024b8f - Bump extensions/Cargo from `e33a7fd` to `5d7f8c2` [00:39:32] [02mediawiki] 07dependabot[bot] opened pull request 03#5028: Bump extensions/Cargo from `e33a7fd` to `5d7f8c2` - 13https://github.com/miraheze/mediawiki/pull/5028 [00:39:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-5d7f8c2 - 13https://github.com/miraheze/mediawiki [00:39:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Disambiguator-433aaf5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/27c0f934d986 [00:39:36] [02miraheze/mediawiki] 07dependabot[bot] 0327c0f93 - Bump extensions/Disambiguator from `7275829` to `433aaf5` [00:39:38] [02mediawiki] 07dependabot[bot] opened pull request 03#5029: Bump extensions/Disambiguator from `7275829` to `433aaf5` - 13https://github.com/miraheze/mediawiki/pull/5029 [00:39:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-433aaf5 - 13https://github.com/miraheze/mediawiki [00:39:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CiteThisPage-d6cb358 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8ddce6738aa1 [00:39:42] [02miraheze/mediawiki] 07dependabot[bot] 038ddce67 - Bump extensions/CiteThisPage from `0b662a9` to `d6cb358` [00:39:44] [02mediawiki] 07dependabot[bot] opened pull request 03#5030: Bump extensions/CiteThisPage from `0b662a9` to `d6cb358` - 13https://github.com/miraheze/mediawiki/pull/5030 [00:39:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-d6cb358 - 13https://github.com/miraheze/mediawiki [00:39:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Josa-df1f3b8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f7c7cceb5a09 [00:39:48] [02miraheze/mediawiki] 07dependabot[bot] 03f7c7cce - Bump extensions/Josa from `14712b7` to `df1f3b8` [00:39:50] [02mediawiki] 07dependabot[bot] opened pull request 03#5031: Bump extensions/Josa from `14712b7` to `df1f3b8` - 13https://github.com/miraheze/mediawiki/pull/5031 [00:39:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Josa-df1f3b8 - 13https://github.com/miraheze/mediawiki [00:39:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageImages-f2d571d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bf9f6adcf8ea [00:39:54] [02miraheze/mediawiki] 07dependabot[bot] 03bf9f6ad - Bump extensions/PageImages from `c4f13bd` to `f2d571d` [00:39:56] [02mediawiki] 07dependabot[bot] opened pull request 03#5032: Bump extensions/PageImages from `c4f13bd` to `f2d571d` - 13https://github.com/miraheze/mediawiki/pull/5032 [00:39:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageImages-f2d571d - 13https://github.com/miraheze/mediawiki [00:39:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-3a68118 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1f9ce02ded71 [00:40:00] [02miraheze/mediawiki] 07dependabot[bot] 031f9ce02 - Bump skins/Evelution from `d492062` to `3a68118` [00:40:02] [02mediawiki] 07dependabot[bot] opened pull request 03#5033: Bump skins/Evelution from `d492062` to `3a68118` - 13https://github.com/miraheze/mediawiki/pull/5033 [00:40:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5025: Bump extensions/SocialProfile from `f407fdb` to `a55e1fc` - 13https://github.com/miraheze/mediawiki/pull/5025 [00:40:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-3a68118 - 13https://github.com/miraheze/mediawiki [00:40:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#5024: Bump extensions/PageTriage from `0e564dd` to `efbae63` - 13https://github.com/miraheze/mediawiki/pull/5024 [00:40:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#5027: Bump skins/erudite from `6c9913d` to `909572a` - 13https://github.com/miraheze/mediawiki/pull/5027 [00:40:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#5026: Bump extensions/WikiLove from `a89bf17` to `a3a8629` - 13https://github.com/miraheze/mediawiki/pull/5026 [00:40:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-5aa82f2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4d0cbf020be9 [00:40:12] [02miraheze/mediawiki] 07dependabot[bot] 034d0cbf0 - Bump extensions/GlobalWatchlist from `663a1ec` to `5aa82f2` [00:40:14] [02mediawiki] 07dependabot[bot] opened pull request 03#5034: Bump extensions/GlobalWatchlist from `663a1ec` to `5aa82f2` - 13https://github.com/miraheze/mediawiki/pull/5034 [00:40:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-5aa82f2 - 13https://github.com/miraheze/mediawiki [00:40:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Comments-f25de8d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6a3965425042 [00:40:18] [02miraheze/mediawiki] 07dependabot[bot] 036a39654 - Bump extensions/Comments from `c390305` to `f25de8d` [00:40:20] [02mediawiki] 07dependabot[bot] opened pull request 03#5035: Bump extensions/Comments from `c390305` to `f25de8d` - 13https://github.com/miraheze/mediawiki/pull/5035 [00:40:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Comments-f25de8d - 13https://github.com/miraheze/mediawiki [00:40:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#5030: Bump extensions/CiteThisPage from `0b662a9` to `d6cb358` - 13https://github.com/miraheze/mediawiki/pull/5030 [00:40:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#5029: Bump extensions/Disambiguator from `7275829` to `433aaf5` - 13https://github.com/miraheze/mediawiki/pull/5029 [00:40:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#5028: Bump extensions/Cargo from `e33a7fd` to `5d7f8c2` - 13https://github.com/miraheze/mediawiki/pull/5028 [00:40:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-16beb2e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0259cb93e652 [00:40:29] [02miraheze/mediawiki] 07dependabot[bot] 030259cb9 - Bump extensions/TemplateSandbox from `d7ac520` to `16beb2e` [00:40:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#5028: Bump extensions/Cargo from `e33a7fd` to `5d7f8c2` - 13https://github.com/miraheze/mediawiki/pull/5028 [00:40:32] [02mediawiki] 07dependabot[bot] opened pull request 03#5036: Bump extensions/TemplateSandbox from `d7ac520` to `16beb2e` - 13https://github.com/miraheze/mediawiki/pull/5036 [00:40:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5028: Bump extensions/Cargo from `e33a7fd` to `5d7f8c2` - 13https://github.com/miraheze/mediawiki/pull/5028 [00:40:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-16beb2e - 13https://github.com/miraheze/mediawiki [00:40:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TranslationNotifications-2f58a4d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4e0c3a55f578 [00:40:38] [02miraheze/mediawiki] 07dependabot[bot] 034e0c3a5 - Bump extensions/TranslationNotifications from `e0cce2d` to `2f58a4d` [00:40:39] [02mediawiki] 07dependabot[bot] opened pull request 03#5037: Bump extensions/TranslationNotifications from `e0cce2d` to `2f58a4d` - 13https://github.com/miraheze/mediawiki/pull/5037 [00:40:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TranslationNotifications-2f58a4d - 13https://github.com/miraheze/mediawiki [00:40:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralAuth-6cfc2d1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/375ffcec521b [00:40:44] [02miraheze/mediawiki] 07dependabot[bot] 03375ffce - Bump extensions/CentralAuth from `14aebc2` to `6cfc2d1` [00:40:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-6cfc2d1 - 13https://github.com/miraheze/mediawiki [00:40:47] [02mediawiki] 07dependabot[bot] opened pull request 03#5038: Bump extensions/CentralAuth from `14aebc2` to `6cfc2d1` - 13https://github.com/miraheze/mediawiki/pull/5038 [00:40:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#5031: Bump extensions/Josa from `14712b7` to `df1f3b8` - 13https://github.com/miraheze/mediawiki/pull/5031 [00:40:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/OATHAuth-b2806a0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/11901219c9e8 [00:40:51] [02miraheze/mediawiki] 07dependabot[bot] 031190121 - Bump extensions/OATHAuth from `c8db119` to `b2806a0` [00:40:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#5035: Bump extensions/Comments from `c390305` to `f25de8d` - 13https://github.com/miraheze/mediawiki/pull/5035 [00:40:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5034: Bump extensions/GlobalWatchlist from `663a1ec` to `5aa82f2` - 13https://github.com/miraheze/mediawiki/pull/5034 [00:40:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/OATHAuth-b2806a0 - 13https://github.com/miraheze/mediawiki [00:40:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5032: Bump extensions/PageImages from `c4f13bd` to `f2d571d` - 13https://github.com/miraheze/mediawiki/pull/5032 [00:40:59] [02mediawiki] 07dependabot[bot] opened pull request 03#5039: Bump extensions/OATHAuth from `c8db119` to `b2806a0` - 13https://github.com/miraheze/mediawiki/pull/5039 [00:41:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Truglass-28ddecb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ff79e5c4176e [00:41:02] [02miraheze/mediawiki] 07dependabot[bot] 03ff79e5c - Bump skins/Truglass from `41c68ff` to `28ddecb` [00:41:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Truglass-28ddecb - 13https://github.com/miraheze/mediawiki [00:41:05] [02mediawiki] 07dependabot[bot] opened pull request 03#5040: Bump skins/Truglass from `41c68ff` to `28ddecb` - 13https://github.com/miraheze/mediawiki/pull/5040 [00:41:06] [02mediawiki] 07dependabot[bot] opened pull request 03#5041: Bump extensions/UniversalLanguageSelector from `b707511` to `95f27a3` - 13https://github.com/miraheze/mediawiki/pull/5041 [00:41:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-95f27a3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/95a97933b510 [00:41:09] [02miraheze/mediawiki] 07dependabot[bot] 0395a9793 - Bump extensions/UniversalLanguageSelector from `b707511` to `95f27a3` [00:41:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-95f27a3 - 13https://github.com/miraheze/mediawiki [00:41:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#5033: Bump skins/Evelution from `d492062` to `3a68118` - 13https://github.com/miraheze/mediawiki/pull/5033 [00:41:14] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:41:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Thanks-1156b19 - 13https://github.com/miraheze/mediawiki [00:41:17] [02mediawiki] 07dependabot[bot] opened pull request 03#5042: Bump extensions/Thanks from `0701bde` to `1156b19` - 13https://github.com/miraheze/mediawiki/pull/5042 [00:41:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Thanks-1156b19 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8b52ae93a721 [00:41:20] [02miraheze/mediawiki] 07dependabot[bot] 038b52ae9 - Bump extensions/Thanks from `0701bde` to `1156b19` [00:41:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#5036: Bump extensions/TemplateSandbox from `d7ac520` to `16beb2e` - 13https://github.com/miraheze/mediawiki/pull/5036 [00:41:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RevisionSlider-85eaafd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e7e51a797f44 [00:41:24] [02miraheze/mediawiki] 07dependabot[bot] 03e7e51a7 - Bump extensions/RevisionSlider from `2e38ae3` to `85eaafd` [00:41:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RevisionSlider-85eaafd - 13https://github.com/miraheze/mediawiki [00:41:27] [02mediawiki] 07dependabot[bot] opened pull request 03#5043: Bump extensions/RevisionSlider from `2e38ae3` to `85eaafd` - 13https://github.com/miraheze/mediawiki/pull/5043 [00:41:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#5037: Bump extensions/TranslationNotifications from `e0cce2d` to `2f58a4d` - 13https://github.com/miraheze/mediawiki/pull/5037 [00:41:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Timeless-3300eba [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0cd316e2c0d4 [00:41:32] [02miraheze/mediawiki] 07dependabot[bot] 030cd316e - Bump skins/Timeless from `12e3dd1` to `3300eba` [00:41:33] [02mediawiki] 07dependabot[bot] opened pull request 03#5044: Bump skins/Timeless from `12e3dd1` to `3300eba` - 13https://github.com/miraheze/mediawiki/pull/5044 [00:41:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Timeless-3300eba - 13https://github.com/miraheze/mediawiki [00:41:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Vector-d13f23a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9d12729773fd [00:41:38] [02miraheze/mediawiki] 07dependabot[bot] 039d12729 - Bump skins/Vector from `5dea00c` to `d13f23a` [00:41:39] [02mediawiki] 07dependabot[bot] opened pull request 03#5045: Bump skins/Vector from `5dea00c` to `d13f23a` - 13https://github.com/miraheze/mediawiki/pull/5045 [00:41:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Vector-d13f23a - 13https://github.com/miraheze/mediawiki [00:41:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#5039: Bump extensions/OATHAuth from `c8db119` to `b2806a0` - 13https://github.com/miraheze/mediawiki/pull/5039 [00:41:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Capiunto-3431872 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ed314cd3055e [00:41:45] [02miraheze/mediawiki] 07dependabot[bot] 03ed314cd - Bump extensions/Capiunto from `adc464d` to `3431872` [00:41:47] [02mediawiki] 07dependabot[bot] opened pull request 03#5046: Bump extensions/Capiunto from `adc464d` to `3431872` - 13https://github.com/miraheze/mediawiki/pull/5046 [00:41:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Capiunto-3431872 - 13https://github.com/miraheze/mediawiki [00:41:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#5041: Bump extensions/UniversalLanguageSelector from `b707511` to `95f27a3` - 13https://github.com/miraheze/mediawiki/pull/5041 [00:41:51] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.37, 1.81, 1.48 [00:41:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CheckUser-bc47dcc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/af6cfb038c4f [00:41:53] [02miraheze/mediawiki] 07dependabot[bot] 03af6cfb0 - Bump extensions/CheckUser from `610ba68` to `bc47dcc` [00:41:54] [02mediawiki] 07dependabot[bot] opened pull request 03#5047: Bump extensions/CheckUser from `610ba68` to `bc47dcc` - 13https://github.com/miraheze/mediawiki/pull/5047 [00:41:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#5041: Bump extensions/UniversalLanguageSelector from `b707511` to `95f27a3` - 13https://github.com/miraheze/mediawiki/pull/5041 [00:41:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CheckUser-bc47dcc - 13https://github.com/miraheze/mediawiki [00:41:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#5040: Bump skins/Truglass from `41c68ff` to `28ddecb` - 13https://github.com/miraheze/mediawiki/pull/5040 [00:42:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PollNY-199da9d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/102e80b92aba [00:42:02] [02miraheze/mediawiki] 07dependabot[bot] 03102e80b - Bump extensions/PollNY from `f22c8b2` to `199da9d` [00:42:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PollNY-199da9d - 13https://github.com/miraheze/mediawiki [00:42:05] [02mediawiki] 07dependabot[bot] opened pull request 03#5048: Bump extensions/PollNY from `f22c8b2` to `199da9d` - 13https://github.com/miraheze/mediawiki/pull/5048 [00:42:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/StandardDialogs-7bcabe7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/020c0853030e [00:42:08] [02miraheze/mediawiki] 07dependabot[bot] 03020c085 - Bump extensions/StandardDialogs from `73b2d03` to `7bcabe7` [00:42:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/StandardDialogs-7bcabe7 - 13https://github.com/miraheze/mediawiki [00:42:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.24, 8.66, 8.39 [00:42:11] [02mediawiki] 07dependabot[bot] opened pull request 03#5049: Bump extensions/StandardDialogs from `73b2d03` to `7bcabe7` - 13https://github.com/miraheze/mediawiki/pull/5049 [00:42:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5038: Bump extensions/CentralAuth from `14aebc2` to `6cfc2d1` - 13https://github.com/miraheze/mediawiki/pull/5038 [00:42:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TextExtracts-630d242 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ba4834be108d [00:42:15] [02miraheze/mediawiki] 07dependabot[bot] 03ba4834b - Bump extensions/TextExtracts from `3f080bc` to `630d242` [00:42:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#5042: Bump extensions/Thanks from `0701bde` to `1156b19` - 13https://github.com/miraheze/mediawiki/pull/5042 [00:42:18] [02mediawiki] 07dependabot[bot] opened pull request 03#5050: Bump extensions/TextExtracts from `3f080bc` to `630d242` - 13https://github.com/miraheze/mediawiki/pull/5050 [00:42:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TextExtracts-630d242 - 13https://github.com/miraheze/mediawiki [00:42:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Refreshed-2b2530f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/823226eee168 [00:42:23] [02miraheze/mediawiki] 07dependabot[bot] 03823226e - Bump skins/Refreshed from `95e5e93` to `2b2530f` [00:42:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5051: Bump skins/Refreshed from `95e5e93` to `2b2530f` - 13https://github.com/miraheze/mediawiki/pull/5051 [00:42:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Refreshed-2b2530f - 13https://github.com/miraheze/mediawiki [00:42:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#5043: Bump extensions/RevisionSlider from `2e38ae3` to `85eaafd` - 13https://github.com/miraheze/mediawiki/pull/5043 [00:42:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#5044: Bump skins/Timeless from `12e3dd1` to `3300eba` - 13https://github.com/miraheze/mediawiki/pull/5044 [00:42:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-db298a1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cca817ae061a [00:42:32] [02miraheze/mediawiki] 07dependabot[bot] 03cca817a - Bump extensions/WikimediaIncubator from `2f1e105` to `db298a1` [00:42:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-db298a1 - 13https://github.com/miraheze/mediawiki [00:42:35] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:42:37] [02mediawiki] 07dependabot[bot] opened pull request 03#5052: Bump extensions/WikimediaIncubator from `2f1e105` to `db298a1` - 13https://github.com/miraheze/mediawiki/pull/5052 [00:42:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#5046: Bump extensions/Capiunto from `adc464d` to `3431872` - 13https://github.com/miraheze/mediawiki/pull/5046 [00:42:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MassMessage-0e2ee2e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/550af1e7a74d [00:42:41] [02miraheze/mediawiki] 07dependabot[bot] 03550af1e - Bump extensions/MassMessage from `2850498` to `0e2ee2e` [00:42:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#5045: Bump skins/Vector from `5dea00c` to `d13f23a` - 13https://github.com/miraheze/mediawiki/pull/5045 [00:42:44] [02mediawiki] 07dependabot[bot] opened pull request 03#5053: Bump extensions/MassMessage from `2850498` to `0e2ee2e` - 13https://github.com/miraheze/mediawiki/pull/5053 [00:42:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MassMessage-0e2ee2e - 13https://github.com/miraheze/mediawiki [00:42:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/foreground-0d6a685 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1328398f82fc [00:42:48] [02miraheze/mediawiki] 07dependabot[bot] 031328398 - Bump skins/foreground from `002ff5d` to `0d6a685` [00:42:50] [02mediawiki] 07dependabot[bot] opened pull request 03#5054: Bump skins/foreground from `002ff5d` to `0d6a685` - 13https://github.com/miraheze/mediawiki/pull/5054 [00:42:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/foreground-0d6a685 - 13https://github.com/miraheze/mediawiki [00:42:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AntiSpoof-d1e52a7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d1f210958ec4 [00:42:54] [02miraheze/mediawiki] 07dependabot[bot] 03d1f2109 - Bump extensions/AntiSpoof from `37c00a4` to `d1e52a7` [00:42:56] [02mediawiki] 07dependabot[bot] opened pull request 03#5055: Bump extensions/AntiSpoof from `37c00a4` to `d1e52a7` - 13https://github.com/miraheze/mediawiki/pull/5055 [00:42:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-d1e52a7 - 13https://github.com/miraheze/mediawiki [00:42:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Tabber-3cc24cf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/855c3b25cc42 [00:43:01] [02miraheze/mediawiki] 07dependabot[bot] 03855c3b2 - Bump extensions/Tabber from `b5d1d4e` to `3cc24cf` [00:43:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Tabber-3cc24cf - 13https://github.com/miraheze/mediawiki [00:43:03] [02mediawiki] 07dependabot[bot] opened pull request 03#5056: Bump extensions/Tabber from `b5d1d4e` to `3cc24cf` - 13https://github.com/miraheze/mediawiki/pull/5056 [00:43:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#5047: Bump extensions/CheckUser from `610ba68` to `bc47dcc` - 13https://github.com/miraheze/mediawiki/pull/5047 [00:43:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#5048: Bump extensions/PollNY from `f22c8b2` to `199da9d` - 13https://github.com/miraheze/mediawiki/pull/5048 [00:43:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CategoryTree-e1741c3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/43f6cf9d3e7f [00:43:09] [02miraheze/mediawiki] 07dependabot[bot] 0343f6cf9 - Bump extensions/CategoryTree from `e1620b2` to `e1741c3` [00:43:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CategoryTree-e1741c3 - 13https://github.com/miraheze/mediawiki [00:43:12] [02mediawiki] 07dependabot[bot] opened pull request 03#5057: Bump extensions/CategoryTree from `e1620b2` to `e1741c3` - 13https://github.com/miraheze/mediawiki/pull/5057 [00:43:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Babel-22d6bf4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2d8137d45b35 [00:43:15] [02miraheze/mediawiki] 07dependabot[bot] 032d8137d - Bump extensions/Babel from `559a00a` to `22d6bf4` [00:43:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#5050: Bump extensions/TextExtracts from `3f080bc` to `630d242` - 13https://github.com/miraheze/mediawiki/pull/5050 [00:43:18] [02mediawiki] 07dependabot[bot] opened pull request 03#5058: Bump extensions/Babel from `559a00a` to `22d6bf4` - 13https://github.com/miraheze/mediawiki/pull/5058 [00:43:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Babel-22d6bf4 - 13https://github.com/miraheze/mediawiki [00:43:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#5049: Bump extensions/StandardDialogs from `73b2d03` to `7bcabe7` - 13https://github.com/miraheze/mediawiki/pull/5049 [00:43:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Score-8eb0bbc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/85a4c973c59e [00:43:25] [02miraheze/mediawiki] 07dependabot[bot] 0385a4c97 - Bump extensions/Score from `4e642a6` to `8eb0bbc` [00:43:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#5051: Bump skins/Refreshed from `95e5e93` to `2b2530f` - 13https://github.com/miraheze/mediawiki/pull/5051 [00:43:27] [02mediawiki] 07dependabot[bot] opened pull request 03#5059: Bump extensions/Score from `4e642a6` to `8eb0bbc` - 13https://github.com/miraheze/mediawiki/pull/5059 [00:43:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Score-8eb0bbc - 13https://github.com/miraheze/mediawiki [00:43:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#5052: Bump extensions/WikimediaIncubator from `2f1e105` to `db298a1` - 13https://github.com/miraheze/mediawiki/pull/5052 [00:43:32] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Math-a6b7906 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ec4ccd4e6e4e [00:43:33] [02miraheze/mediawiki] 07dependabot[bot] 03ec4ccd4 - Bump extensions/Math from `4cf0a3a` to `a6b7906` [00:43:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Math-a6b7906 - 13https://github.com/miraheze/mediawiki [00:43:36] [02mediawiki] 07dependabot[bot] opened pull request 03#5060: Bump extensions/Math from `4cf0a3a` to `a6b7906` - 13https://github.com/miraheze/mediawiki/pull/5060 [00:43:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#5053: Bump extensions/MassMessage from `2850498` to `0e2ee2e` - 13https://github.com/miraheze/mediawiki/pull/5053 [00:43:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#5055: Bump extensions/AntiSpoof from `37c00a4` to `d1e52a7` - 13https://github.com/miraheze/mediawiki/pull/5055 [00:43:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateWizard-5f3e285 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d251d871dc44 [00:43:42] [02miraheze/mediawiki] 07dependabot[bot] 03d251d87 - Bump extensions/TemplateWizard from `dbe5c12` to `5f3e285` [00:43:44] [02mediawiki] 07dependabot[bot] opened pull request 03#5061: Bump extensions/TemplateWizard from `dbe5c12` to `5f3e285` - 13https://github.com/miraheze/mediawiki/pull/5061 [00:43:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateWizard-5f3e285 - 13https://github.com/miraheze/mediawiki [00:43:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#5056: Bump extensions/Tabber from `b5d1d4e` to `3cc24cf` - 13https://github.com/miraheze/mediawiki/pull/5056 [00:43:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5054: Bump skins/foreground from `002ff5d` to `0d6a685` - 13https://github.com/miraheze/mediawiki/pull/5054 [00:43:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Gadgets-91ea8b7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2a8d9342ad34 [00:43:52] [02miraheze/mediawiki] 07dependabot[bot] 032a8d934 - Bump extensions/Gadgets from `f958b46` to `91ea8b7` [00:43:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Gadgets-91ea8b7 - 13https://github.com/miraheze/mediawiki [00:43:54] [02mediawiki] 07dependabot[bot] opened pull request 03#5062: Bump extensions/Gadgets from `f958b46` to `91ea8b7` - 13https://github.com/miraheze/mediawiki/pull/5062 [00:43:56] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RSS-e76c5a6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6e6deb2fa78c [00:43:57] [02miraheze/mediawiki] 07dependabot[bot] 036e6deb2 - Bump extensions/RSS from `0c817cf` to `e76c5a6` [00:43:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RSS-e76c5a6 - 13https://github.com/miraheze/mediawiki [00:44:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5063: Bump extensions/RSS from `0c817cf` to `e76c5a6` - 13https://github.com/miraheze/mediawiki/pull/5063 [00:44:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/NewUserNotif-171f067 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/769c7170c82e [00:44:04] [02miraheze/mediawiki] 07dependabot[bot] 03769c717 - Bump extensions/NewUserNotif from `81325f1` to `171f067` [00:44:05] [02mediawiki] 07dependabot[bot] opened pull request 03#5064: Bump extensions/NewUserNotif from `81325f1` to `171f067` - 13https://github.com/miraheze/mediawiki/pull/5064 [00:44:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/NewUserNotif-171f067 - 13https://github.com/miraheze/mediawiki [00:44:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#5057: Bump extensions/CategoryTree from `e1620b2` to `e1741c3` - 13https://github.com/miraheze/mediawiki/pull/5057 [00:44:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#5058: Bump extensions/Babel from `559a00a` to `22d6bf4` - 13https://github.com/miraheze/mediawiki/pull/5058 [00:44:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#5059: Bump extensions/Score from `4e642a6` to `8eb0bbc` - 13https://github.com/miraheze/mediawiki/pull/5059 [00:44:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-4c3dcef [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cb8e751d90d8 [00:44:14] [02miraheze/mediawiki] 07dependabot[bot] 03cb8e751 - Bump skins/Cosmos from `c0cbc1a` to `4c3dcef` [00:44:15] [02mediawiki] 07dependabot[bot] opened pull request 03#5065: Bump skins/Cosmos from `c0cbc1a` to `4c3dcef` - 13https://github.com/miraheze/mediawiki/pull/5065 [00:44:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#5060: Bump extensions/Math from `4cf0a3a` to `a6b7906` - 13https://github.com/miraheze/mediawiki/pull/5060 [00:44:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Cosmos-4c3dcef - 13https://github.com/miraheze/mediawiki [00:44:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/chameleon-4b7531c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c3fffbe2430d [00:44:21] [02miraheze/mediawiki] 07dependabot[bot] 03c3fffbe - Bump skins/chameleon from `a923c46` to `4b7531c` [00:44:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.60, 8.98, 8.89 [00:44:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/chameleon-4b7531c - 13https://github.com/miraheze/mediawiki [00:44:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5066: Bump skins/chameleon from `a923c46` to `4b7531c` - 13https://github.com/miraheze/mediawiki/pull/5066 [00:44:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PdfHandler-d178aa8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b636afef6ff8 [00:44:28] [02miraheze/mediawiki] 07dependabot[bot] 03b636afe - Bump extensions/PdfHandler from `ba526c2` to `d178aa8` [00:44:29] [02mediawiki] 07dependabot[bot] opened pull request 03#5067: Bump extensions/PdfHandler from `ba526c2` to `d178aa8` - 13https://github.com/miraheze/mediawiki/pull/5067 [00:44:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PdfHandler-d178aa8 - 13https://github.com/miraheze/mediawiki [00:44:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.72, 9.77, 9.54 [00:44:32] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UploadsLink-80655dc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fd73ac0c157d [00:44:34] [02miraheze/mediawiki] 07dependabot[bot] 03fd73ac0 - Bump extensions/UploadsLink from `3301ba6` to `80655dc` [00:44:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UploadsLink-80655dc - 13https://github.com/miraheze/mediawiki [00:44:37] [02mediawiki] 07dependabot[bot] opened pull request 03#5068: Bump extensions/UploadsLink from `3301ba6` to `80655dc` - 13https://github.com/miraheze/mediawiki/pull/5068 [00:44:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#5061: Bump extensions/TemplateWizard from `dbe5c12` to `5f3e285` - 13https://github.com/miraheze/mediawiki/pull/5061 [00:44:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Mirage-4c00cd9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0beb0fb13df3 [00:44:41] [02miraheze/mediawiki] 07dependabot[bot] 030beb0fb - Bump skins/Mirage from `72722e4` to `4c00cd9` [00:44:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5069: Bump skins/Mirage from `72722e4` to `4c00cd9` - 13https://github.com/miraheze/mediawiki/pull/5069 [00:44:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Mirage-4c00cd9 - 13https://github.com/miraheze/mediawiki [00:44:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5062: Bump extensions/Gadgets from `f958b46` to `91ea8b7` - 13https://github.com/miraheze/mediawiki/pull/5062 [00:44:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#5064: Bump extensions/NewUserNotif from `81325f1` to `171f067` - 13https://github.com/miraheze/mediawiki/pull/5064 [00:44:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Metrolook-ba638aa [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ebc7596d9e7b [00:44:50] [02miraheze/mediawiki] 07dependabot[bot] 03ebc7596 - Bump skins/Metrolook from `eed515c` to `ba638aa` [00:44:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5070: Bump skins/Metrolook from `eed515c` to `ba638aa` - 13https://github.com/miraheze/mediawiki/pull/5070 [00:44:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.33, 1.35 [00:44:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Metrolook-ba638aa - 13https://github.com/miraheze/mediawiki [00:44:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UserMerge-614e434 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/58107e6ea783 [00:44:56] [02miraheze/mediawiki] 07dependabot[bot] 0358107e6 - Bump extensions/UserMerge from `f8306f2` to `614e434` [00:44:58] [02mediawiki] 07dependabot[bot] opened pull request 03#5071: Bump extensions/UserMerge from `f8306f2` to `614e434` - 13https://github.com/miraheze/mediawiki/pull/5071 [00:44:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UserMerge-614e434 - 13https://github.com/miraheze/mediawiki [00:45:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#5063: Bump extensions/RSS from `0c817cf` to `e76c5a6` - 13https://github.com/miraheze/mediawiki/pull/5063 [00:45:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ProofreadPage-48027fd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ddc3679b4783 [00:45:04] [02miraheze/mediawiki] 07dependabot[bot] 03ddc3679 - Bump extensions/ProofreadPage from `8204d15` to `48027fd` [00:45:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-48027fd - 13https://github.com/miraheze/mediawiki [00:45:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5072: Bump extensions/ProofreadPage from `8204d15` to `48027fd` - 13https://github.com/miraheze/mediawiki/pull/5072 [00:45:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/StructuredNavigation-e44eb2e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/460fb7c61765 [00:45:10] [02miraheze/mediawiki] 07dependabot[bot] 03460fb7c - Bump extensions/StructuredNavigation from `5c72149` to `e44eb2e` [00:45:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/StructuredNavigation-e44eb2e - 13https://github.com/miraheze/mediawiki [00:45:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.29, 8.41, 8.34 [00:45:13] [02mediawiki] 07dependabot[bot] opened pull request 03#5073: Bump extensions/StructuredNavigation from `5c72149` to `e44eb2e` - 13https://github.com/miraheze/mediawiki/pull/5073 [00:45:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-3d9c9bc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d3a1c4261b1b [00:45:16] [02miraheze/mediawiki] 07dependabot[bot] 03d3a1c42 - Bump extensions/MirahezeMagic from `55a7813` to `3d9c9bc` [00:45:17] [02mediawiki] 07dependabot[bot] opened pull request 03#5074: Bump extensions/MirahezeMagic from `55a7813` to `3d9c9bc` - 13https://github.com/miraheze/mediawiki/pull/5074 [00:45:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#5065: Bump skins/Cosmos from `c0cbc1a` to `4c3dcef` - 13https://github.com/miraheze/mediawiki/pull/5065 [00:45:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#5065: Bump skins/Cosmos from `c0cbc1a` to `4c3dcef` - 13https://github.com/miraheze/mediawiki/pull/5065 [00:45:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-3d9c9bc - 13https://github.com/miraheze/mediawiki [00:45:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalUsage-6bf0c3f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b8439a52cec7 [00:45:25] [02miraheze/mediawiki] 07dependabot[bot] 03b8439a5 - Bump extensions/GlobalUsage from `70afccc` to `6bf0c3f` [00:45:26] [02mediawiki] 07dependabot[bot] opened pull request 03#5075: Bump extensions/GlobalUsage from `70afccc` to `6bf0c3f` - 13https://github.com/miraheze/mediawiki/pull/5075 [00:45:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-6bf0c3f - 13https://github.com/miraheze/mediawiki [00:45:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#5066: Bump skins/chameleon from `a923c46` to `4b7531c` - 13https://github.com/miraheze/mediawiki/pull/5066 [00:45:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#5066: Bump skins/chameleon from `a923c46` to `4b7531c` - 13https://github.com/miraheze/mediawiki/pull/5066 [00:45:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#5066: Bump skins/chameleon from `a923c46` to `4b7531c` - 13https://github.com/miraheze/mediawiki/pull/5066 [00:45:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5070: Bump skins/Metrolook from `eed515c` to `ba638aa` - 13https://github.com/miraheze/mediawiki/pull/5070 [00:45:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#5067: Bump extensions/PdfHandler from `ba526c2` to `d178aa8` - 13https://github.com/miraheze/mediawiki/pull/5067 [00:45:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Linter-6569219 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e4b8d0daa68e [00:45:38] [02miraheze/mediawiki] 07dependabot[bot] 03e4b8d0d - Bump extensions/Linter from `71d2b3c` to `6569219` [00:45:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5068: Bump extensions/UploadsLink from `3301ba6` to `80655dc` - 13https://github.com/miraheze/mediawiki/pull/5068 [00:45:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Linter-6569219 - 13https://github.com/miraheze/mediawiki [00:45:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5076: Bump extensions/Linter from `71d2b3c` to `6569219` - 13https://github.com/miraheze/mediawiki/pull/5076 [00:45:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SpamBlacklist-57b68a0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/190ef8fa0b6d [00:45:46] [02miraheze/mediawiki] 07dependabot[bot] 03190ef8f - Bump extensions/SpamBlacklist from `1191db0` to `57b68a0` [00:45:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 2.95, 3.37 [00:45:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SpamBlacklist-57b68a0 - 13https://github.com/miraheze/mediawiki [00:45:49] [02mediawiki] 07dependabot[bot] opened pull request 03#5077: Bump extensions/SpamBlacklist from `1191db0` to `57b68a0` - 13https://github.com/miraheze/mediawiki/pull/5077 [00:45:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateData-3f82cf6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e7f45dd11c0c [00:45:52] [02miraheze/mediawiki] 07dependabot[bot] 03e7f45dd - Bump extensions/TemplateData from `13690c5` to `3f82cf6` [00:45:53] [02mediawiki] 07dependabot[bot] opened pull request 03#5078: Bump extensions/TemplateData from `13690c5` to `3f82cf6` - 13https://github.com/miraheze/mediawiki/pull/5078 [00:45:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#5069: Bump skins/Mirage from `72722e4` to `4c00cd9` - 13https://github.com/miraheze/mediawiki/pull/5069 [00:45:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateData-3f82cf6 - 13https://github.com/miraheze/mediawiki [00:45:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalPreferences-3e04f4d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9762ae70359a [00:45:59] [02miraheze/mediawiki] 07dependabot[bot] 039762ae7 - Bump extensions/GlobalPreferences from `ffb96a7` to `3e04f4d` [00:46:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5079: Bump extensions/GlobalPreferences from `ffb96a7` to `3e04f4d` - 13https://github.com/miraheze/mediawiki/pull/5079 [00:46:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#5071: Bump extensions/UserMerge from `f8306f2` to `614e434` - 13https://github.com/miraheze/mediawiki/pull/5071 [00:46:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalPreferences-3e04f4d - 13https://github.com/miraheze/mediawiki [00:46:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#5072: Bump extensions/ProofreadPage from `8204d15` to `48027fd` - 13https://github.com/miraheze/mediawiki/pull/5072 [00:46:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-610c03f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d692c21402b9 [00:46:08] [02miraheze/mediawiki] 07dependabot[bot] 03d692c21 - Bump extensions/Translate from `12e795e` to `610c03f` [00:46:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5080: Bump extensions/Translate from `12e795e` to `610c03f` - 13https://github.com/miraheze/mediawiki/pull/5080 [00:46:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-610c03f - 13https://github.com/miraheze/mediawiki [00:46:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5073: Bump extensions/StructuredNavigation from `5c72149` to `e44eb2e` - 13https://github.com/miraheze/mediawiki/pull/5073 [00:46:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#5074: Bump extensions/MirahezeMagic from `55a7813` to `3d9c9bc` - 13https://github.com/miraheze/mediawiki/pull/5074 [00:46:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/CologneBlue-4493647 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/41d38e7f29d9 [00:46:17] [02miraheze/mediawiki] 07dependabot[bot] 0341d38e7 - Bump skins/CologneBlue from `ca2b35e` to `4493647` [00:46:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/CologneBlue-4493647 - 13https://github.com/miraheze/mediawiki [00:46:20] [02mediawiki] 07dependabot[bot] opened pull request 03#5081: Bump skins/CologneBlue from `ca2b35e` to `4493647` - 13https://github.com/miraheze/mediawiki/pull/5081 [00:46:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GeoData-df83a41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/25d9d284815d [00:46:23] [02miraheze/mediawiki] 07dependabot[bot] 0325d9d28 - Bump extensions/GeoData from `9a7861c` to `df83a41` [00:46:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GeoData-df83a41 - 13https://github.com/miraheze/mediawiki [00:46:26] [02mediawiki] 07dependabot[bot] opened pull request 03#5082: Bump extensions/GeoData from `9a7861c` to `df83a41` - 13https://github.com/miraheze/mediawiki/pull/5082 [00:46:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#5075: Bump extensions/GlobalUsage from `70afccc` to `6bf0c3f` - 13https://github.com/miraheze/mediawiki/pull/5075 [00:46:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#5077: Bump extensions/SpamBlacklist from `1191db0` to `57b68a0` - 13https://github.com/miraheze/mediawiki/pull/5077 [00:46:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalBlocking-b231ea7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fa49373c7bbb [00:46:32] [02miraheze/mediawiki] 07dependabot[bot] 03fa49373 - Bump extensions/GlobalBlocking from `b7441ba` to `b231ea7` [00:46:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalBlocking-b231ea7 - 13https://github.com/miraheze/mediawiki [00:46:35] [02mediawiki] 07dependabot[bot] opened pull request 03#5083: Bump extensions/GlobalBlocking from `b7441ba` to `b231ea7` - 13https://github.com/miraheze/mediawiki/pull/5083 [00:46:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MassEditRegex-b285c73 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/eca4993186c1 [00:46:38] [02miraheze/mediawiki] 07dependabot[bot] 03eca4993 - Bump extensions/MassEditRegex from `27b4608` to `b285c73` [00:46:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5076: Bump extensions/Linter from `71d2b3c` to `6569219` - 13https://github.com/miraheze/mediawiki/pull/5076 [00:46:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MassEditRegex-b285c73 - 13https://github.com/miraheze/mediawiki [00:46:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5084: Bump extensions/MassEditRegex from `27b4608` to `b285c73` - 13https://github.com/miraheze/mediawiki/pull/5084 [00:46:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cite-61b83bb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cf5bf1a82281 [00:46:46] [02miraheze/mediawiki] 07dependabot[bot] 03cf5bf1a - Bump extensions/Cite from `976ac42` to `61b83bb` [00:46:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cite-61b83bb - 13https://github.com/miraheze/mediawiki [00:46:49] [02mediawiki] 07dependabot[bot] opened pull request 03#5085: Bump extensions/Cite from `976ac42` to `61b83bb` - 13https://github.com/miraheze/mediawiki/pull/5085 [00:46:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#5078: Bump extensions/TemplateData from `13690c5` to `3f82cf6` - 13https://github.com/miraheze/mediawiki/pull/5078 [00:46:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-dbf441c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/edd3ae8f81ac [00:46:53] [02miraheze/mediawiki] 07dependabot[bot] 03edd3ae8 - Bump extensions/MultimediaViewer from `fae4afc` to `dbf441c` [00:46:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-dbf441c - 13https://github.com/miraheze/mediawiki [00:46:56] [02mediawiki] 07dependabot[bot] opened pull request 03#5086: Bump extensions/MultimediaViewer from `fae4afc` to `dbf441c` - 13https://github.com/miraheze/mediawiki/pull/5086 [00:46:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#5079: Bump extensions/GlobalPreferences from `ffb96a7` to `3e04f4d` - 13https://github.com/miraheze/mediawiki/pull/5079 [00:46:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageSchemas-7129e95 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e82657af85bd [00:47:01] [02miraheze/mediawiki] 07dependabot[bot] 03e82657a - Bump extensions/PageSchemas from `7795965` to `7129e95` [00:47:02] [02mediawiki] 07dependabot[bot] opened pull request 03#5087: Bump extensions/PageSchemas from `7795965` to `7129e95` - 13https://github.com/miraheze/mediawiki/pull/5087 [00:47:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-7129e95 - 13https://github.com/miraheze/mediawiki [00:47:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/InputBox-97da829 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ccfb831998f3 [00:47:07] [02miraheze/mediawiki] 07dependabot[bot] 03ccfb831 - Bump extensions/InputBox from `e0a1c32` to `97da829` [00:47:08] [02mediawiki] 07dependabot[bot] opened pull request 03#5088: Bump extensions/InputBox from `e0a1c32` to `97da829` - 13https://github.com/miraheze/mediawiki/pull/5088 [00:47:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/InputBox-97da829 - 13https://github.com/miraheze/mediawiki [00:47:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ReplaceText-dbd95b2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ea2ddd36f4ef [00:47:13] [02miraheze/mediawiki] 07dependabot[bot] 03ea2ddd3 - Bump extensions/ReplaceText from `62a18e8` to `dbd95b2` [00:47:14] [02mediawiki] 07dependabot[bot] opened pull request 03#5089: Bump extensions/ReplaceText from `62a18e8` to `dbd95b2` - 13https://github.com/miraheze/mediawiki/pull/5089 [00:47:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ReplaceText-dbd95b2 - 13https://github.com/miraheze/mediawiki [00:47:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#5081: Bump skins/CologneBlue from `ca2b35e` to `4493647` - 13https://github.com/miraheze/mediawiki/pull/5081 [00:47:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/OAuth-9176f5d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/102b2a69f1bc [00:47:20] [02miraheze/mediawiki] 07dependabot[bot] 03102b2a6 - Bump extensions/OAuth from `75168c1` to `9176f5d` [00:47:22] [02mediawiki] 07dependabot[bot] opened pull request 03#5090: Bump extensions/OAuth from `75168c1` to `9176f5d` - 13https://github.com/miraheze/mediawiki/pull/5090 [00:47:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/OAuth-9176f5d - 13https://github.com/miraheze/mediawiki [00:47:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#5084: Bump extensions/MassEditRegex from `27b4608` to `b285c73` - 13https://github.com/miraheze/mediawiki/pull/5084 [00:47:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CreateWiki-9b3267d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a0cd5c5748fc [00:47:28] [02miraheze/mediawiki] 07dependabot[bot] 03a0cd5c5 - Bump extensions/CreateWiki from `2b0f27d` to `9b3267d` [00:47:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.49, 7.84, 8.45 [00:47:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#5083: Bump extensions/GlobalBlocking from `b7441ba` to `b231ea7` - 13https://github.com/miraheze/mediawiki/pull/5083 [00:47:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-9b3267d - 13https://github.com/miraheze/mediawiki [00:47:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#5080: Bump extensions/Translate from `12e795e` to `610c03f` - 13https://github.com/miraheze/mediawiki/pull/5080 [00:47:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5080: Bump extensions/Translate from `12e795e` to `610c03f` - 13https://github.com/miraheze/mediawiki/pull/5080 [00:47:35] [02mediawiki] 07dependabot[bot] opened pull request 03#5091: Bump extensions/CreateWiki from `2b0f27d` to `9b3267d` - 13https://github.com/miraheze/mediawiki/pull/5091 [00:47:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.67, 9.13, 9.33 [00:47:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5086: Bump extensions/MultimediaViewer from `fae4afc` to `dbf441c` - 13https://github.com/miraheze/mediawiki/pull/5086 [00:47:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/EventLogging-3ec3f22 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/161448df4e00 [00:47:40] [02miraheze/mediawiki] 07dependabot[bot] 03161448d - Bump extensions/EventLogging from `b3a9222` to `3ec3f22` [00:47:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#5082: Bump extensions/GeoData from `9a7861c` to `df83a41` - 13https://github.com/miraheze/mediawiki/pull/5082 [00:47:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/EventLogging-3ec3f22 - 13https://github.com/miraheze/mediawiki [00:47:44] [02mediawiki] 07dependabot[bot] opened pull request 03#5092: Bump extensions/EventLogging from `b3a9222` to `3ec3f22` - 13https://github.com/miraheze/mediawiki/pull/5092 [00:47:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5085: Bump extensions/Cite from `976ac42` to `61b83bb` - 13https://github.com/miraheze/mediawiki/pull/5085 [00:47:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#5088: Bump extensions/InputBox from `e0a1c32` to `97da829` - 13https://github.com/miraheze/mediawiki/pull/5088 [00:47:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Wikibase-0b54839 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/686128c245e8 [00:47:50] [02miraheze/mediawiki] 07dependabot[bot] 03686128c - Bump extensions/Wikibase from `5007351` to `0b54839` [00:47:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5093: Bump extensions/Wikibase from `5007351` to `0b54839` - 13https://github.com/miraheze/mediawiki/pull/5093 [00:47:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Wikibase-0b54839 - 13https://github.com/miraheze/mediawiki [00:47:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/VisualEditor-2a9889a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/33dd20614b92 [00:47:56] [02miraheze/mediawiki] 07dependabot[bot] 0333dd206 - Bump extensions/VisualEditor from `b0b884f` to `2a9889a` [00:47:58] [02mediawiki] 07dependabot[bot] opened pull request 03#5094: Bump extensions/VisualEditor from `b0b884f` to `2a9889a` - 13https://github.com/miraheze/mediawiki/pull/5094 [00:47:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-2a9889a - 13https://github.com/miraheze/mediawiki [00:48:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#5087: Bump extensions/PageSchemas from `7795965` to `7129e95` - 13https://github.com/miraheze/mediawiki/pull/5087 [00:48:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ProtectSite-afa02d0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f67a8b7b7fcb [00:48:04] [02miraheze/mediawiki] 07dependabot[bot] 03f67a8b7 - Bump extensions/ProtectSite from `6e5558f` to `afa02d0` [00:48:05] [02mediawiki] 07dependabot[bot] opened pull request 03#5095: Bump extensions/ProtectSite from `6e5558f` to `afa02d0` - 13https://github.com/miraheze/mediawiki/pull/5095 [00:48:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ProtectSite-afa02d0 - 13https://github.com/miraheze/mediawiki [00:48:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Nostalgia-433e1fa [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/973602770e25 [00:48:10] [02miraheze/mediawiki] 07dependabot[bot] 039736027 - Bump skins/Nostalgia from `a5c96a3` to `433e1fa` [00:48:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Nostalgia-433e1fa - 13https://github.com/miraheze/mediawiki [00:48:13] [02mediawiki] 07dependabot[bot] opened pull request 03#5096: Bump skins/Nostalgia from `a5c96a3` to `433e1fa` - 13https://github.com/miraheze/mediawiki/pull/5096 [00:48:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#5089: Bump extensions/ReplaceText from `62a18e8` to `dbd95b2` - 13https://github.com/miraheze/mediawiki/pull/5089 [00:48:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CommentStreams-9fb9689 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e02d11babcd2 [00:48:17] [02miraheze/mediawiki] 07dependabot[bot] 03e02d11b - Bump extensions/CommentStreams from `fee8267` to `9fb9689` [00:48:19] [02mediawiki] 07dependabot[bot] opened pull request 03#5097: Bump extensions/CommentStreams from `fee8267` to `9fb9689` - 13https://github.com/miraheze/mediawiki/pull/5097 [00:48:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CommentStreams-9fb9689 - 13https://github.com/miraheze/mediawiki [00:48:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#5090: Bump extensions/OAuth from `75168c1` to `9176f5d` - 13https://github.com/miraheze/mediawiki/pull/5090 [00:48:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-73a5b97 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ec49c21bbed1 [00:48:25] [02miraheze/mediawiki] 07dependabot[bot] 03ec49c21 - Bump extensions/ArticlePlaceholder from `6700354` to `73a5b97` [00:48:26] [02mediawiki] 07dependabot[bot] opened pull request 03#5098: Bump extensions/ArticlePlaceholder from `6700354` to `73a5b97` - 13https://github.com/miraheze/mediawiki/pull/5098 [00:48:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-73a5b97 - 13https://github.com/miraheze/mediawiki [00:48:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Popups-e9767da [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f83abbd10301 [00:48:31] [02miraheze/mediawiki] 07dependabot[bot] 03f83abbd - Bump extensions/Popups from `6f30daa` to `e9767da` [00:48:32] [02mediawiki] 07dependabot[bot] opened pull request 03#5099: Bump extensions/Popups from `6f30daa` to `e9767da` - 13https://github.com/miraheze/mediawiki/pull/5099 [00:48:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Popups-e9767da - 13https://github.com/miraheze/mediawiki [00:48:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#5091: Bump extensions/CreateWiki from `2b0f27d` to `9b3267d` - 13https://github.com/miraheze/mediawiki/pull/5091 [00:48:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-4f84cf3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/55a20445a228 [00:48:38] [02miraheze/mediawiki] 07dependabot[bot] 0355a2044 - Bump extensions/ConfirmEdit from `95640eb` to `4f84cf3` [00:48:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-4f84cf3 - 13https://github.com/miraheze/mediawiki [00:48:41] [02mediawiki] 07dependabot[bot] opened pull request 03#5100: Bump extensions/ConfirmEdit from `95640eb` to `4f84cf3` - 13https://github.com/miraheze/mediawiki/pull/5100 [00:48:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#5092: Bump extensions/EventLogging from `b3a9222` to `3ec3f22` - 13https://github.com/miraheze/mediawiki/pull/5092 [00:48:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/VEForAll-ae706ad [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/82cd6baaeca9 [00:48:46] [02miraheze/mediawiki] 07dependabot[bot] 0382cd6ba - Bump extensions/VEForAll from `7942bab` to `ae706ad` [00:48:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/VEForAll-ae706ad - 13https://github.com/miraheze/mediawiki [00:48:49] [02mediawiki] 07dependabot[bot] opened pull request 03#5101: Bump extensions/VEForAll from `7942bab` to `ae706ad` - 13https://github.com/miraheze/mediawiki/pull/5101 [00:48:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/MinervaNeue-adc20dd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/19f6293d6a27 [00:48:52] [02miraheze/mediawiki] 07dependabot[bot] 0319f6293 - Bump skins/MinervaNeue from `9b6acac` to `adc20dd` [00:48:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-adc20dd - 13https://github.com/miraheze/mediawiki [00:48:55] [02mediawiki] 07dependabot[bot] opened pull request 03#5102: Bump skins/MinervaNeue from `9b6acac` to `adc20dd` - 13https://github.com/miraheze/mediawiki/pull/5102 [00:48:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#5097: Bump extensions/CommentStreams from `fee8267` to `9fb9689` - 13https://github.com/miraheze/mediawiki/pull/5097 [00:48:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Mask-0b93fa7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e196e5486fa2 [00:48:59] [02miraheze/mediawiki] 07dependabot[bot] 03e196e54 - Bump skins/Mask from `82f5011` to `0b93fa7` [00:49:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5103: Bump skins/Mask from `82f5011` to `0b93fa7` - 13https://github.com/miraheze/mediawiki/pull/5103 [00:49:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Mask-0b93fa7 - 13https://github.com/miraheze/mediawiki [00:49:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#5095: Bump extensions/ProtectSite from `6e5558f` to `afa02d0` - 13https://github.com/miraheze/mediawiki/pull/5095 [00:49:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MsCatSelect-8d8a6c5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/665f517f5441 [00:49:07] [02miraheze/mediawiki] 07dependabot[bot] 03665f517 - Bump extensions/MsCatSelect from `6eaf689` to `8d8a6c5` [00:49:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MsCatSelect-8d8a6c5 - 13https://github.com/miraheze/mediawiki [00:49:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5104: Bump extensions/MsCatSelect from `6eaf689` to `8d8a6c5` - 13https://github.com/miraheze/mediawiki/pull/5104 [00:49:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#5094: Bump extensions/VisualEditor from `b0b884f` to `2a9889a` - 13https://github.com/miraheze/mediawiki/pull/5094 [00:49:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5096: Bump skins/Nostalgia from `a5c96a3` to `433e1fa` - 13https://github.com/miraheze/mediawiki/pull/5096 [00:49:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Medik-3a10ae1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c9602569c511 [00:49:16] [02miraheze/mediawiki] 07dependabot[bot] 03c960256 - Bump skins/Medik from `1840eb1` to `3a10ae1` [00:49:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Medik-3a10ae1 - 13https://github.com/miraheze/mediawiki [00:49:19] [02mediawiki] 07dependabot[bot] opened pull request 03#5105: Bump skins/Medik from `1840eb1` to `3a10ae1` - 13https://github.com/miraheze/mediawiki/pull/5105 [00:49:20] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:49:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#5093: Bump extensions/Wikibase from `5007351` to `0b54839` - 13https://github.com/miraheze/mediawiki/pull/5093 [00:49:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ShortDescription-bf568ed [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c7064dcfa6cf [00:49:25] [02miraheze/mediawiki] 07dependabot[bot] 03c7064dc - Bump extensions/ShortDescription from `7c86644` to `bf568ed` [00:49:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#5098: Bump extensions/ArticlePlaceholder from `6700354` to `73a5b97` - 13https://github.com/miraheze/mediawiki/pull/5098 [00:49:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ShortDescription-bf568ed - 13https://github.com/miraheze/mediawiki [00:49:29] [02mediawiki] 07dependabot[bot] opened pull request 03#5106: Bump extensions/ShortDescription from `7c86644` to `bf568ed` - 13https://github.com/miraheze/mediawiki/pull/5106 [00:49:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AbuseFilter-918688c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fefc86489bda [00:49:32] [02miraheze/mediawiki] 07dependabot[bot] 03fefc864 - Bump extensions/AbuseFilter from `7d92a02` to `918688c` [00:49:33] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [00:49:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-918688c - 13https://github.com/miraheze/mediawiki [00:49:35] [02mediawiki] 07dependabot[bot] opened pull request 03#5107: Bump extensions/AbuseFilter from `7d92a02` to `918688c` - 13https://github.com/miraheze/mediawiki/pull/5107 [00:49:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5102: Bump skins/MinervaNeue from `9b6acac` to `adc20dd` - 13https://github.com/miraheze/mediawiki/pull/5102 [00:49:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#5099: Bump extensions/Popups from `6f30daa` to `e9767da` - 13https://github.com/miraheze/mediawiki/pull/5099 [00:49:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5100: Bump extensions/ConfirmEdit from `95640eb` to `4f84cf3` - 13https://github.com/miraheze/mediawiki/pull/5100 [00:49:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-8149d41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b06ce489b2f6 [00:49:43] [02miraheze/mediawiki] 07dependabot[bot] 03b06ce48 - Bump extensions/FlaggedRevs from `7aaa50a` to `8149d41` [00:49:44] [02mediawiki] 07dependabot[bot] opened pull request 03#5108: Bump extensions/FlaggedRevs from `7aaa50a` to `8149d41` - 13https://github.com/miraheze/mediawiki/pull/5108 [00:49:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5101: Bump extensions/VEForAll from `7942bab` to `ae706ad` - 13https://github.com/miraheze/mediawiki/pull/5101 [00:49:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-8149d41 - 13https://github.com/miraheze/mediawiki [00:49:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SandboxLink-26d8d66 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e77d6317bb57 [00:49:50] [02miraheze/mediawiki] 07dependabot[bot] 03e77d631 - Bump extensions/SandboxLink from `567d9c0` to `26d8d66` [00:49:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5109: Bump extensions/SandboxLink from `567d9c0` to `26d8d66` - 13https://github.com/miraheze/mediawiki/pull/5109 [00:49:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SandboxLink-26d8d66 - 13https://github.com/miraheze/mediawiki [00:49:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/EmbedVideo-78455be [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d116e0f909a3 [00:49:56] [02miraheze/mediawiki] 07dependabot[bot] 03d116e0f - Bump extensions/EmbedVideo from `f98a408` to `78455be` [00:49:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/EmbedVideo-78455be - 13https://github.com/miraheze/mediawiki [00:49:59] [02mediawiki] 07dependabot[bot] opened pull request 03#5110: Bump extensions/EmbedVideo from `f98a408` to `78455be` - 13https://github.com/miraheze/mediawiki/pull/5110 [00:50:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MobileFrontend-a0a8881 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1c414d1c214f [00:50:01] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [00:50:02] [02miraheze/mediawiki] 07dependabot[bot] 031c414d1 - Bump extensions/MobileFrontend from `82da625` to `a0a8881` [00:50:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-a0a8881 - 13https://github.com/miraheze/mediawiki [00:50:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#5103: Bump skins/Mask from `82f5011` to `0b93fa7` - 13https://github.com/miraheze/mediawiki/pull/5103 [00:50:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5111: Bump extensions/MobileFrontend from `82da625` to `a0a8881` - 13https://github.com/miraheze/mediawiki/pull/5111 [00:50:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#5106: Bump extensions/ShortDescription from `7c86644` to `bf568ed` - 13https://github.com/miraheze/mediawiki/pull/5106 [00:50:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/QuizGame-3ade24c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0b0bd1b396ac [00:50:11] [02miraheze/mediawiki] 07dependabot[bot] 030b0bd1b - Bump extensions/QuizGame from `797bcc0` to `3ade24c` [00:50:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5104: Bump extensions/MsCatSelect from `6eaf689` to `8d8a6c5` - 13https://github.com/miraheze/mediawiki/pull/5104 [00:50:14] [02mediawiki] 07dependabot[bot] opened pull request 03#5112: Bump extensions/QuizGame from `797bcc0` to `3ade24c` - 13https://github.com/miraheze/mediawiki/pull/5112 [00:50:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/QuizGame-3ade24c - 13https://github.com/miraheze/mediawiki [00:50:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#5105: Bump skins/Medik from `1840eb1` to `3a10ae1` - 13https://github.com/miraheze/mediawiki/pull/5105 [00:50:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiEditor-1594337 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/88a5d580a8e8 [00:50:20] [02miraheze/mediawiki] 07dependabot[bot] 0388a5d58 - Bump extensions/WikiEditor from `f99f482` to `1594337` [00:50:22] [02mediawiki] 07dependabot[bot] opened pull request 03#5113: Bump extensions/WikiEditor from `f99f482` to `1594337` - 13https://github.com/miraheze/mediawiki/pull/5113 [00:50:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiEditor-1594337 - 13https://github.com/miraheze/mediawiki [00:50:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BetaFeatures-2eecf69 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7c2d822899ca [00:50:26] [02miraheze/mediawiki] 07dependabot[bot] 037c2d822 - Bump extensions/BetaFeatures from `6dbf978` to `2eecf69` [00:50:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-2eecf69 - 13https://github.com/miraheze/mediawiki [00:50:29] [02mediawiki] 07dependabot[bot] opened pull request 03#5114: Bump extensions/BetaFeatures from `6dbf978` to `2eecf69` - 13https://github.com/miraheze/mediawiki/pull/5114 [00:50:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-3cab311 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e679f34431e3 [00:50:32] [02miraheze/mediawiki] 07dependabot[bot] 03e679f34 - Bump extensions/TimedMediaHandler from `c92b7bc` to `3cab311` [00:50:34] [02mediawiki] 07dependabot[bot] opened pull request 03#5115: Bump extensions/TimedMediaHandler from `c92b7bc` to `3cab311` - 13https://github.com/miraheze/mediawiki/pull/5115 [00:50:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-3cab311 - 13https://github.com/miraheze/mediawiki [00:50:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5107: Bump extensions/AbuseFilter from `7d92a02` to `918688c` - 13https://github.com/miraheze/mediawiki/pull/5107 [00:50:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MsUpload-96d15eb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b77176e5e68e [00:50:40] [02miraheze/mediawiki] 07dependabot[bot] 03b77176e - Bump extensions/MsUpload from `be41c8d` to `96d15eb` [00:50:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MsUpload-96d15eb - 13https://github.com/miraheze/mediawiki [00:50:42] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [00:50:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5116: Bump extensions/MsUpload from `be41c8d` to `96d15eb` - 13https://github.com/miraheze/mediawiki/pull/5116 [00:50:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#5109: Bump extensions/SandboxLink from `567d9c0` to `26d8d66` - 13https://github.com/miraheze/mediawiki/pull/5109 [00:50:46] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:50:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Echo-e58dea5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2b6b0e995381 [00:50:49] [02miraheze/mediawiki] 07dependabot[bot] 032b6b0e9 - Bump extensions/Echo from `ad5125d` to `e58dea5` [00:50:50] [02mediawiki] 07dependabot[bot] opened pull request 03#5117: Bump extensions/Echo from `ad5125d` to `e58dea5` - 13https://github.com/miraheze/mediawiki/pull/5117 [00:50:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Echo-e58dea5 - 13https://github.com/miraheze/mediawiki [00:50:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralNotice-c919fed [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/97401bd131a6 [00:50:55] [02miraheze/mediawiki] 07dependabot[bot] 0397401bd - Bump extensions/CentralNotice from `e206d09` to `c919fed` [00:50:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#5112: Bump extensions/QuizGame from `797bcc0` to `3ade24c` - 13https://github.com/miraheze/mediawiki/pull/5112 [00:50:58] [02mediawiki] 07dependabot[bot] opened pull request 03#5118: Bump extensions/CentralNotice from `e206d09` to `c919fed` - 13https://github.com/miraheze/mediawiki/pull/5118 [00:50:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-c919fed - 13https://github.com/miraheze/mediawiki [00:51:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#5108: Bump extensions/FlaggedRevs from `7aaa50a` to `8149d41` - 13https://github.com/miraheze/mediawiki/pull/5108 [00:51:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#5110: Bump extensions/EmbedVideo from `f98a408` to `78455be` - 13https://github.com/miraheze/mediawiki/pull/5110 [00:51:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#5110: Bump extensions/EmbedVideo from `f98a408` to `78455be` - 13https://github.com/miraheze/mediawiki/pull/5110 [00:51:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#5110: Bump extensions/EmbedVideo from `f98a408` to `78455be` - 13https://github.com/miraheze/mediawiki/pull/5110 [00:51:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Flow-dbfb30c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/14136610dab9 [00:51:08] [02miraheze/mediawiki] 07dependabot[bot] 031413661 - Bump extensions/Flow from `bd32d47` to `dbfb30c` [00:51:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5119: Bump extensions/Flow from `bd32d47` to `dbfb30c` - 13https://github.com/miraheze/mediawiki/pull/5119 [00:51:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Flow-dbfb30c - 13https://github.com/miraheze/mediawiki [00:51:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5114: Bump extensions/BetaFeatures from `6dbf978` to `2eecf69` - 13https://github.com/miraheze/mediawiki/pull/5114 [00:51:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#5113: Bump extensions/WikiEditor from `f99f482` to `1594337` - 13https://github.com/miraheze/mediawiki/pull/5113 [00:51:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateStyles-7c07529 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e0137cb68560 [00:51:17] [02miraheze/mediawiki] 07dependabot[bot] 03e0137cb - Bump extensions/TemplateStyles from `a9475a0` to `7c07529` [00:51:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateStyles-7c07529 - 13https://github.com/miraheze/mediawiki [00:51:20] [02mediawiki] 07dependabot[bot] opened pull request 03#5120: Bump extensions/TemplateStyles from `a9475a0` to `7c07529` - 13https://github.com/miraheze/mediawiki/pull/5120 [00:51:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SecureLinkFixer-9fa973f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/85f80a2d448c [00:51:23] [02miraheze/mediawiki] 07dependabot[bot] 0385f80a2 - Bump extensions/SecureLinkFixer from `0936952` to `9fa973f` [00:51:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SecureLinkFixer-9fa973f - 13https://github.com/miraheze/mediawiki [00:51:26] [02mediawiki] 07dependabot[bot] opened pull request 03#5121: Bump extensions/SecureLinkFixer from `0936952` to `9fa973f` - 13https://github.com/miraheze/mediawiki/pull/5121 [00:51:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#5111: Bump extensions/MobileFrontend from `82da625` to `a0a8881` - 13https://github.com/miraheze/mediawiki/pull/5111 [00:51:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TwoColConflict-9a2232f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c017bca8a31a [00:51:31] [02miraheze/mediawiki] 07dependabot[bot] 03c017bca - Bump extensions/TwoColConflict from `eff532d` to `9a2232f` [00:51:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TwoColConflict-9a2232f - 13https://github.com/miraheze/mediawiki [00:51:34] [02mediawiki] 07dependabot[bot] opened pull request 03#5122: Bump extensions/TwoColConflict from `eff532d` to `9a2232f` - 13https://github.com/miraheze/mediawiki/pull/5122 [00:51:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/MonoBook-4e82430 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1a3580ff28a3 [00:51:37] [02miraheze/mediawiki] 07dependabot[bot] 031a3580f - Bump skins/MonoBook from `4a1f56b` to `4e82430` [00:51:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/MonoBook-4e82430 - 13https://github.com/miraheze/mediawiki [00:51:40] [02mediawiki] 07dependabot[bot] opened pull request 03#5123: Bump skins/MonoBook from `4a1f56b` to `4e82430` - 13https://github.com/miraheze/mediawiki/pull/5123 [00:51:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#5115: Bump extensions/TimedMediaHandler from `c92b7bc` to `3cab311` - 13https://github.com/miraheze/mediawiki/pull/5115 [00:51:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Nimbus-701a419 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/87cbbe77aded [00:51:44] [02miraheze/mediawiki] 07dependabot[bot] 0387cbbe7 - Bump skins/Nimbus from `8669188` to `701a419` [00:51:46] [02mediawiki] 07dependabot[bot] opened pull request 03#5124: Bump skins/Nimbus from `8669188` to `701a419` - 13https://github.com/miraheze/mediawiki/pull/5124 [00:51:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Nimbus-701a419 - 13https://github.com/miraheze/mediawiki [00:51:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5116: Bump extensions/MsUpload from `be41c8d` to `96d15eb` - 13https://github.com/miraheze/mediawiki/pull/5116 [00:51:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SyntaxHighlight_GeSHi-48797f7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/addb2b8600af [00:51:52] [02miraheze/mediawiki] 07dependabot[bot] 03addb2b8 - Bump extensions/SyntaxHighlight_GeSHi from `0ca322d` to `48797f7` [00:51:53] [02mediawiki] 07dependabot[bot] opened pull request 03#5125: Bump extensions/SyntaxHighlight_GeSHi from `0ca322d` to `48797f7` - 13https://github.com/miraheze/mediawiki/pull/5125 [00:51:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SyntaxHighlight_GeSHi-48797f7 - 13https://github.com/miraheze/mediawiki [00:51:56] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-b0b1746 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e58f9d152350 [00:51:58] [02miraheze/mediawiki] 07dependabot[bot] 03e58f9d1 - Bump extensions/PageForms from `86d9007` to `b0b1746` [00:51:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-b0b1746 - 13https://github.com/miraheze/mediawiki [00:52:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5126: Bump extensions/PageForms from `86d9007` to `b0b1746` - 13https://github.com/miraheze/mediawiki/pull/5126 [00:52:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#5117: Bump extensions/Echo from `ad5125d` to `e58dea5` - 13https://github.com/miraheze/mediawiki/pull/5117 [00:52:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#5118: Bump extensions/CentralNotice from `e206d09` to `c919fed` - 13https://github.com/miraheze/mediawiki/pull/5118 [00:52:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DynamicPageList-0b19c30 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3b19342ae55e [00:52:07] [02miraheze/mediawiki] 07dependabot[bot] 033b19342 - Bump extensions/DynamicPageList from `866ecb4` to `0b19c30` [00:52:08] [02mediawiki] 07dependabot[bot] opened pull request 03#5127: Bump extensions/DynamicPageList from `866ecb4` to `0b19c30` - 13https://github.com/miraheze/mediawiki/pull/5127 [00:52:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#5120: Bump extensions/TemplateStyles from `a9475a0` to `7c07529` - 13https://github.com/miraheze/mediawiki/pull/5120 [00:52:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList-0b19c30 - 13https://github.com/miraheze/mediawiki [00:52:13] [02mediawiki] 07dependabot[bot] opened pull request 03#5128: Bump extensions/DiscussionTools from `397361d` to `d71eb71` - 13https://github.com/miraheze/mediawiki/pull/5128 [00:52:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DiscussionTools-d71eb71 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d302303d96bc [00:52:16] PROBLEM - cp21 Stunnel Http for phab121 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_phab121' not defined [00:52:16] [02miraheze/mediawiki] 07dependabot[bot] 03d302303 - Bump extensions/DiscussionTools from `397361d` to `d71eb71` [00:52:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#5119: Bump extensions/Flow from `bd32d47` to `dbfb30c` - 13https://github.com/miraheze/mediawiki/pull/5119 [00:52:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-d71eb71 - 13https://github.com/miraheze/mediawiki [00:52:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#5124: Bump skins/Nimbus from `8669188` to `701a419` - 13https://github.com/miraheze/mediawiki/pull/5124 [00:52:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#5121: Bump extensions/SecureLinkFixer from `0936952` to `9fa973f` - 13https://github.com/miraheze/mediawiki/pull/5121 [00:52:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#5122: Bump extensions/TwoColConflict from `eff532d` to `9a2232f` - 13https://github.com/miraheze/mediawiki/pull/5122 [00:52:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#5123: Bump skins/MonoBook from `4a1f56b` to `4e82430` - 13https://github.com/miraheze/mediawiki/pull/5123 [00:52:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#5125: Bump extensions/SyntaxHighlight_GeSHi from `0ca322d` to `48797f7` - 13https://github.com/miraheze/mediawiki/pull/5125 [00:52:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#5128: Bump extensions/DiscussionTools from `397361d` to `d71eb71` - 13https://github.com/miraheze/mediawiki/pull/5128 [00:52:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#5126: Bump extensions/PageForms from `86d9007` to `b0b1746` - 13https://github.com/miraheze/mediawiki/pull/5126 [00:52:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#5127: Bump extensions/DynamicPageList from `866ecb4` to `0b19c30` - 13https://github.com/miraheze/mediawiki/pull/5127 [00:52:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:52:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#5126: Bump extensions/PageForms from `86d9007` to `b0b1746` - 13https://github.com/miraheze/mediawiki/pull/5126 [00:52:34] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:52:35] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:52:37] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:52:38] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:52:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.98, 4.14, 3.94 [00:53:22] PROBLEM - cp30 Stunnel Http for phab121 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_phab121' not defined [00:53:34] PROBLEM - cp21 Stunnel Http for mw112 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw112' not defined [00:53:44] PROBLEM - cp21 Stunnel Http for mw122 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw122' not defined [00:54:14] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:54:27] PROBLEM - cp21 Stunnel Http for mw121 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw121' not defined [00:54:28] PROBLEM - cp21 Stunnel Http for mw101 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw101' not defined [00:54:55] PROBLEM - cp30 Stunnel Http for mw121 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw121' not defined [00:54:56] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:55:03] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.17, 1.45, 1.94 [00:55:11] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw111' not defined [00:55:16] PROBLEM - cp21 Stunnel Http for mw102 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mw102' not defined [00:55:18] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:55:20] PROBLEM - cp21 Stunnel Http for mon111 on cp21 is UNKNOWN: NRPE: Command 'check_stunnel_mon111' not defined [00:55:34] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw111' not defined [00:55:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.86, 3.87 [00:55:38] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:55:44] PROBLEM - cp30 Stunnel Http for mon111 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mon111' not defined [00:55:59] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:56:13] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw112' not defined [00:56:17] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [00:56:25] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw122' not defined [00:56:46] PROBLEM - cp30 Stunnel Http for mw101 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw101' not defined [00:56:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.21, 9.37, 9.11 [00:57:08] PROBLEM - cp30 Stunnel Http for mw102 on cp30 is UNKNOWN: NRPE: Command 'check_stunnel_mw102' not defined [00:57:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:02:20] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [01:02:51] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.06, 1.19, 1.61 [01:03:01] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [01:03:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.90, 3.81 [01:04:48] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2855 bytes in 0.068 second response time [01:05:56] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2855 bytes in 0.283 second response time [01:08:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.79, 9.22, 8.66 [01:11:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 9.04, 8.69 [01:11:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.39, 9.69, 9.60 [01:12:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.14, 9.98, 9.34 [01:13:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.60, 3.62, 3.84 [01:14:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.82, 9.86, 9.65 [01:14:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.35, 9.80, 9.08 [01:15:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.34, 9.64, 9.33 [01:16:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.26, 4.04, 4.00 [01:17:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 9.93, 9.76 [01:19:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.67, 4.32, 2.64 [01:19:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.73, 3.88 [01:20:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.06, 9.91, 9.75 [01:22:29] PROBLEM - cp31 Stunnel Http for phab121 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_phab121' not defined [01:23:02] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw122' not defined [01:23:07] PROBLEM - cp31 Stunnel Http for mw101 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw101' not defined [01:23:28] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw111' not defined [01:23:39] PROBLEM - cp31 Stunnel Http for mon111 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mon111' not defined [01:23:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.21, 8.58, 8.71 [01:24:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.63, 1.29 [01:24:57] PROBLEM - cp31 Stunnel Http for mw102 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw102' not defined [01:25:25] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw112' not defined [01:25:35] PROBLEM - cp31 Stunnel Http for mw121 on cp31 is UNKNOWN: NRPE: Command 'check_stunnel_mw121' not defined [01:26:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.00, 9.28, 9.63 [01:26:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.21, 9.60, 9.07 [01:27:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.56, 1.08, 1.13 [01:28:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.27, 10.00, 9.45 [01:28:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.54, 1.32, 1.80 [01:31:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.25, 8.35, 8.67 [01:32:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.70, 0.98, 1.55 [01:33:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.06, 8.81, 9.16 [01:33:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.13, 9.39, 9.17 [01:33:59] PROBLEM - cp31 Stunnel HTTP for phab121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2855 bytes in 0.280 second response time [01:34:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.15, 9.82, 9.54 [01:34:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.43, 7.94, 8.46 [01:36:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.19, 2.99 [01:36:32] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 0.76, 3.76, 3.13 [01:39:23] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 2.73, 2.86 [01:40:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.62, 10.45, 9.66 [01:42:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.09, 3.33 [01:44:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.15, 9.53, 8.97 [01:45:59] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.55, 1.07, 1.96 [01:48:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.90, 8.97, 8.88 [01:49:05] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.58, 0.81, 1.69 [01:54:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.83, 9.28, 8.91 [01:55:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.20, 9.08, 9.00 [01:55:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 9.42, 9.78 [01:58:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.85, 8.61, 8.82 [01:58:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.11, 9.85, 9.85 [01:59:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.20, 9.54, 9.92 [02:00:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.71, 9.26, 9.00 [02:01:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.64, 9.67, 9.78 [02:01:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.21, 9.69, 9.90 [02:01:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 2.55, 1.81 [02:06:11] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.56, 1.86, 1.46 [02:07:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:07:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.50, 9.77, 9.72 [02:08:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.97, 2.00, 1.85 [02:11:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.45, 1.65 [02:12:28] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.16, 9.12, 8.89 [02:12:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:13:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.59, 3.42 [02:13:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.42, 9.70, 9.71 [02:16:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.90, 3.55 [02:16:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.31, 10.07, 9.83 [02:18:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.83, 8.72, 9.94 [02:18:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.38, 8.71, 8.82 [02:19:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 3.14, 3.31 [02:19:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.29, 9.83, 9.80 [02:21:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.40, 9.41, 10.03 [02:21:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.32, 9.01, 8.84 [02:24:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.57, 9.12, 9.86 [02:24:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.41, 1.82, 1.59 [02:24:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.33, 8.62, 8.70 [02:27:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.46, 10.04, 10.07 [02:27:35] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.40, 1.46 [02:27:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.65, 3.89, 3.07 [02:29:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.28, 10.02, 9.88 [02:29:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 4.00, 3.60 [02:30:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.83, 3.19 [02:30:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.23, 9.15, 8.85 [02:31:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.04, 9.50, 9.72 [02:32:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.67, 8.86, 9.73 [02:32:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.66, 3.54 [02:33:25] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.35, 3.33, 3.09 [02:33:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.99, 7.75, 8.30 [02:36:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.11, 9.47, 9.15 [02:38:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 9.90, 9.68 [02:38:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.85, 10.40, 10.13 [02:41:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.37, 9.06, 8.68 [02:42:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 2.91, 3.31 [02:44:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.63, 9.42, 9.81 [02:49:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.72, 3.50 [02:50:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.82, 9.70, 9.25 [02:50:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.99, 9.29, 9.80 [02:54:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.20, 8.64, 9.97 [02:55:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.28, 10.47, 9.71 [02:56:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.30, 3.42 [02:57:27] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.23, 1.90, 1.20 [02:57:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.31, 9.98 [02:58:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.82, 9.91, 9.60 [02:58:23] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.10, 1.49, 1.92 [02:59:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.58, 3.26 [02:59:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.93, 9.66, 9.38 [03:00:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.00, 10.25, 10.03 [03:00:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.18, 10.52, 9.86 [03:00:35] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.42, 1.21, 1.06 [03:01:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.54, 10.42, 9.83 [03:02:13] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw112 [03:02:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.01, 3.27, 3.20 [03:04:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.62, 9.90, 9.73 [03:04:23] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.36, 0.99, 1.57 [03:05:08] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [03:06:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 9.81, 9.93 [03:07:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.41, 9.82, 9.71 [03:08:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.56, 3.44 [03:09:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.26, 10.89, 10.34 [03:12:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.43, 9.10, 9.75 [03:12:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.96, 9.12, 9.62 [03:13:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.76, 8.87, 9.68 [03:13:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.75, 9.43, 9.78 [03:14:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 3.07, 3.27 [03:15:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 5.77, 9.10, 9.83 [03:16:08] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw101 mw112 mw122 [03:16:32] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw101 mw102 mw121 mw122 [03:16:46] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw101 mw102 mw121 mw122 [03:17:43] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw112 mw122 [03:17:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:18:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.77, 7.96, 9.55 [03:18:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:19:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.85, 6.72, 8.37 [03:19:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.25, 6.89, 8.50 [03:19:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.85, 6.54, 8.31 [03:22:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.12, 6.72, 8.35 [03:23:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.44, 6.83, 8.13 [03:25:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 3.69, 5.86, 8.09 [03:53:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.25, 2.97, 2.61 [03:56:28] SRE; persistent 503s and 502s on multiple wikis [04:01:40] dmehus: I don't think anyone is around right now. The issue is known but I'm only available on mobile and not much I can do even if I wasn't. [04:02:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.61, 1.09, 1.79 [04:04:44] dmehus: oh, I just looked at Icinga, this isn't just a localised outage. It looks like 3 MediaWiki servers are down completely. Have been for 50 minutes. It was 4 servers a little bit ago. [04:05:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.82, 1.01, 1.63 [04:06:54] paladox: around? [04:08:42] Different mw servers are down from different cache proxies it looks like... [04:10:33] Reception123: ^ [04:12:45] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.94, 2.11, 1.40 [04:12:51] Now ongoing for an hour [04:15:58] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.96, 1.57, 1.32 [04:19:15] CosmicAlpha, yeah [04:19:20] worthy of an !sre ping? [04:20:32] @Void ^^ [04:20:54] It is. Yes. [04:21:33] Created a task [04:21:40] Void went offline, MacFan4000 [04:21:49] Likely will need paladox or Reception123 when he's up [04:22:31] Yeah for everyone else it’s super early in the morning and so I suspect they are asleep [04:23:47] Well just my luck the one time I'm 1,000 miles away from my PC is when it's actually needed at this time... [04:24:38] Well things still sort of work since half of the mw servers are still working [04:25:10] But yeah this is the worst time possible, as Void and I are the only ones not usually asleep at this time, and Void usually leaves around now. MacFan4000: only 2 servers are constantly running, the rest are fluctuating. [04:27:03] oh, how many MW servers do we have now? [04:27:09] didn't we used to only have like 4-5? [04:27:12] 6 total [04:27:15] oh [04:27:35] mw101, 102, 111, 112, 121, 122 [04:27:38] Yeah only 1 was up a couple seconds ago. [04:28:45] Though I guess there is a 7th if you proxy through test101 [04:28:58] ah [04:42:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:49:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:35:18] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:12] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 3.151 second response time [06:03:54] PROBLEM - duepedia.uk.to - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'duepedia.uk.to' expires in 15 day(s) (Wed 09 Feb 2022 05:51:34 GMT +0000). [06:07:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0d6213579764...e07fd4ddce97 [06:08:00] [02miraheze/ssl] 07MirahezeSSLBot 03e07fd4d - Bot: Update SSL cert for duepedia.uk.to [06:12:09] PROBLEM - staratlaswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'staratlaswiki.org' expires in 15 day(s) (Wed 09 Feb 2022 05:50:28 GMT +0000). [06:14:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e07fd4ddce97...bdfa239e1dd5 [06:14:54] [02miraheze/ssl] 07MirahezeSSLBot 03bdfa239 - Bot: Update SSL cert for staratlaswiki.org [06:19:46] !log rebooted mw102 [06:21:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:30:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [06:30:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [06:31:01] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:31:06] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [06:31:13] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [06:31:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:32:56] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [06:32:57] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [06:33:34] RECOVERY - duepedia.uk.to - LetsEncrypt on sslhost is OK: OK - Certificate 'duepedia.uk.to' will expire on Sun 24 Apr 2022 05:07:52 GMT +0000. [06:34:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 8.44, 7.10 [06:34:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.82, 8.79, 7.39 [06:36:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.98, 10.54, 8.15 [06:36:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.02, 10.20, 8.14 [06:38:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 9.58, 8.15 [06:38:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.15, 9.31, 8.13 [06:39:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.49, 8.43, 7.68 [06:39:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.02, 9.81, 8.11 [06:42:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 9.74, 8.41 [06:42:47] RECOVERY - staratlaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'staratlaswiki.org' will expire on Sun 24 Apr 2022 05:14:45 GMT +0000. [06:44:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.90, 10.24, 8.87 [06:44:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.34, 9.86, 8.66 [06:45:02] Reception123, CosmicAlpha, any idea why Reception123's log entries don't say "log details removed" (https://polcompball.miraheze.org/wiki/Special:Log?type=&user=Reception123&page=&wpdate=&tagfilter=&wpfilters%5B%5D=newusers)? [06:45:04] [url] All public logs - Polcompball Wiki | polcompball.miraheze.org [06:45:10] Agent thinks i18n is broken :( [06:45:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.21, 9.84, 8.66 [06:46:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.18, 8.94, 8.22 [06:47:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.17, 8.49, 6.50 [06:48:22] dmehus: I don't see a specific log entry that should say that. [06:48:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.75, 9.57, 8.78 [06:49:23] CosmicAlpha, yeah, look closer. Normally when comments log entries are revdel'd, you still see the log entries by they say "log details removed" [06:49:31] they just say "delete" and don't finish [06:49:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:49:39] even Agent confirmed it as a problem [06:50:35] dmehus: I don't see a comment that should be revdel [06:50:53] there were tonnes, CosmicAlpha [06:51:12] https://polcompball.miraheze.org/w/index.php?title=Special:Log&logid=50589 [06:51:13] [url] All public logs - Polcompball Wiki | polcompball.miraheze.org [06:51:22] https://polcompball.miraheze.org/w/index.php?title=Special:Log&logid=54594 [06:51:23] [url] All public logs - Polcompball Wiki | polcompball.miraheze.org [06:51:48] then look at the intervening log entries between those [06:51:57] those are all the revdel'd ones [06:53:25] On the other hand, TheGhostofInky did disable the Comments extension [06:53:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.39, 9.53, 9.22 [06:53:44] but if that's the case, shouldn't the comments log entries be removed entirely, not broken? [06:54:08] dmehus: I see that, but I don't see any of those actions period. I don't see the revisions that should've been hidden whether they say it or not. [06:54:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.15, 10.36, 9.31 [06:54:42] well, they're log actions, not revisions [06:54:56] I don't see the actual log entries that should say log summary removed. [06:55:23] you don't see "Reception123 delete "? [06:56:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.74, 7.64, 7.00 [06:56:32] dmehus: that's not the hidden entry, the delete has no i18n associated with it period. That's another log entry on itself. [06:57:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 9.75, 9.26 [06:57:32] Well I'm not sure what you mean there, but let me show you [06:57:36] https://polcompball.miraheze.org/wiki/Special:Log?type=&user=Reception123&page=&wpdate=&tagfilter=&wpfilters%5B%5D=newusers&uselang=qqx&safemode=1 [06:57:37] [url] (pagetitle: (all-logs-page)) | polcompball.miraheze.org [06:58:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.57, 7.98, 8.23 [06:58:11] https://usercontent.irccloud-cdn.com/file/uZHn3sr7/2022-01-23%2022.56.09%20polcompball.miraheze.org%20beb379024840.jpg [06:58:16] There is no i18n associated with "delete" meaning "delete" was an individual log summary. [06:58:17] CosmicAlpha: ^ [06:58:55] CosmicAlpha, anyway I think Agent's theory about it being i18n related is not correct. I think it's the Comments extension being a bug-infested crappy extension :P [06:59:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.27, 10.21, 9.63 [06:59:31] because when the extension was disabled, it removed the comments log, but left remnants of the comments log log entries [06:59:35] which is bad practice [06:59:59] so in short, I suspect it's not an i18n issue after all :) [07:00:07] was just about to comment that I didn't know Comments registered comment deletions on Special:Log/delete. If so then yes, just remnants of the old log entries [07:00:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.92, 10.05, 9.43 [07:00:23] Agent, oh you didn't? [07:00:25] yeah it does [07:00:39] There is no i18n to have the issue for. The delete was the Overriding intentionally selected log entry it seems. And it does that because the other logs aren't even comments related. So yeah it leavs them. Your looking at the revdelete logs, not the hidden log entries themselves. [07:00:44] since Comments comments are actually log entries [07:01:07] CosmicAlpha, yeah, we established that :P [07:01:19] it's the Comments extension being super fuckery :P [07:01:27] anyway, I have to go to bed now :) [07:01:30] No, its it doing what it should. [07:01:46] It shouldn't touch log entries unrelated to it. [07:01:49] CosmicAlpha, agree to disagree, Comments has so many unresolved bugs and fatals [07:01:57] Those are revdelete logs not comments logs. [07:02:07] it's a lost cause, I give up on the Comments extension being useful :P [07:02:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.38, 9.77, 9.59 [07:02:58] well, the Comments extension should NOT remove the Comments log log entries when the extension is removed [07:03:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.15, 9.08, 9.16 [07:03:28] they should stay in the relevant DB table, imho, so the deletion log can pull the details from the DB [07:03:50] I'm going to sleep. It's 2AM anyway. Sorry if I'm not being helpful or patient here I'm just tired. I can look more into what you mean tomorrow. [07:03:53] It's reliant on a system message inside Comments. Once Comments is disabled, that system message goes along with it. [07:04:17] CosmicAlpha, it's fine, we're both just tired [07:04:24] anyway I'm going to bed also :) [07:07:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.75, 9.44, 9.41 [07:10:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.83, 9.76, 9.54 [07:10:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.82, 9.12, 8.73 [07:11:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 9.70, 9.18 [07:12:39] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790461399 [07:12:40] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790461531 [07:12:42] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790461600 [07:12:43] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790461850 [07:12:45] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790461697 [07:12:46] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790462129 [07:13:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.96, 10.25, 9.77 [07:13:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.18, 8.85, 7.85 [07:16:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.20, 9.39, 8.97 [07:16:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.80, 8.20, 7.76 [07:17:24] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790464555 [07:18:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.08, 8.39, 7.64 [07:20:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.46, 9.71, 9.49 [07:21:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:21:37] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790466617 [07:22:55] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790467205 [07:23:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.22, 8.68, 8.18 [07:23:52] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790467663 [07:24:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.87, 8.03, 7.79 [07:24:58] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:25:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.54, 9.57, 9.36 [07:26:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.83, 10.05, 9.69 [07:28:42] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:28:57] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790470284 [07:29:04] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790470324 [07:29:22] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790470535 [07:29:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.72, 9.81, 9.64 [07:29:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.14, 7.57, 7.93 [07:30:12] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790470940 [07:30:20] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790470985 [07:31:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:31:55] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790471871 [07:32:48] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790472344 [07:33:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 9.18, 9.71 [07:36:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 9.36, 9.96 [07:36:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.06, 9.94, 9.90 [07:38:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:39:07] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:39:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.63, 9.87, 10.05 [07:39:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.22, 10.09, 9.30 [07:43:05] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:43:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 10.31, 9.67 [07:44:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.03, 9.85, 9.55 [07:45:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.16, 9.51, 9.77 [07:45:32] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:46:05] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:46:06] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:46:33] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:46:52] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:47:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.85, 10.70, 10.16 [07:48:03] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790480414 [07:49:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.07, 9.25, 9.42 [07:51:50] [02puppet] 07Universal-Omega edited pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:53:33] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:53:34] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [07:53:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.68, 9.20, 9.77 [07:56:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.38, 7.15, 8.46 [07:56:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.54, 9.56, 9.88 [08:04:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.04, 9.01, 8.91 [08:05:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.26, 10.15, 9.83 [08:05:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.18, 9.17, 9.46 [08:06:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.06, 8.60, 8.77 [08:07:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 9.35, 9.54 [08:08:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.43, 9.71, 9.65 [08:08:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.94, 9.30, 8.92 [08:09:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 9.94, 9.23 [08:13:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.03, 10.71, 10.04 [08:15:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 9.59, 9.35 [08:16:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.80, 8.96, 9.48 [08:19:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.29, 9.88, 9.78 [08:19:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.72, 9.87, 9.46 [08:21:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.02, 10.10, 9.53 [08:25:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 9.70, 9.33 [08:25:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.42, 9.97, 9.66 [08:27:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.46, 9.18, 9.16 [08:28:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 10.07, 9.73 [08:30:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.85, 9.58, 9.29 [08:30:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 9.52, 9.56 [08:33:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.66, 9.53, 9.32 [08:33:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.47, 10.18, 9.79 [08:35:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.19, 9.88, 9.89 [08:36:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.84, 9.97, 9.52 [08:39:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.94, 9.60, 9.71 [08:41:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.31, 10.18, 10.01 [08:42:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.67, 9.33, 9.39 [08:42:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.65, 9.43, 9.59 [08:43:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.83, 8.56, 8.01 [08:44:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.78, 9.98, 9.97 [08:45:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.22, 9.68, 9.51 [08:45:59] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.70, 8.08, 7.91 [08:47:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.84, 10.32, 10.10 [08:47:25] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 0.99, 3.53, 2.43 [08:48:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:50:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.93, 8.59, 9.17 [08:53:13] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.31, 1.64, 1.94 [08:55:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:56:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 9.78, 9.97 [08:56:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.00, 10.40, 9.77 [08:57:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.34, 3.03 [08:57:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.21, 8.60, 8.31 [08:59:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.07, 9.94, 10.01 [08:59:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.12, 9.36, 9.47 [09:00:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.94, 2.74, 2.86 [09:00:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 8.26, 8.26 [09:01:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.41, 9.68, 9.92 [09:01:54] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.48, 0.97, 1.50 [09:04:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 10.33, 10.14 [09:11:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.40, 9.31, 9.83 [09:12:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.39, 8.78, 8.39 [09:14:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.34, 10.00, 9.99 [09:15:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.87, 8.36, 8.29 [09:15:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.16, 9.16, 9.76 [09:16:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.40, 9.80, 9.92 [09:18:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.28, 9.47, 9.05 [09:19:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.85, 10.58, 10.18 [09:20:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:21:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.53, 8.45, 8.72 [09:22:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.08, 8.73, 9.74 [09:22:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.10, 9.51, 9.80 [09:24:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:27:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.11, 8.81, 9.10 [09:29:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.30, 9.14, 8.88 [09:30:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.29, 8.67, 9.01 [09:32:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.90, 9.35, 9.03 [09:33:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.83, 9.55, 9.57 [09:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:36:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.45, 9.48, 9.55 [09:40:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.81, 7.56, 8.26 [09:41:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.75, 10.39, 9.58 [09:42:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.53, 9.08, 9.14 [09:44:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 9.29, 9.27 [09:44:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:45:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.14, 8.39, 8.88 [09:51:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.76, 8.95, 8.71 [09:53:37] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.47, 10.47, 9.66 [09:59:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.14, 9.18, 9.46 [09:59:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:02:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 8.17, 8.46 [10:03:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:04:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.24, 10.13, 9.31 [10:07:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.43, 7.82, 8.37 [10:07:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.63, 9.55, 9.23 [10:08:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:12:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 9.45, 9.24 [10:15:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:15:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 9.28, 9.21 [10:18:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 9.54, 9.32 [10:18:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.18, 7.81, 8.36 [10:22:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.65, 8.70, 8.11 [10:25:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.33, 9.28, 8.41 [10:28:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.42, 8.84, 8.39 [10:30:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:31:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.26, 8.05, 8.14 [10:31:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.92, 9.26, 8.84 [10:36:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:36:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.78, 9.11, 8.59 [10:37:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.80, 10.10, 9.33 [10:39:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.56, 9.44, 8.54 [10:40:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.28, 9.07, 9.66 [10:43:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.92, 9.90, 9.87 [10:46:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 9.25, 9.63 [10:48:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.55, 9.38, 8.78 [10:49:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.69, 9.73 [10:52:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.85, 9.28, 9.56 [10:56:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.93, 9.66, 9.35 [10:59:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.09, 10.18, 9.55 [11:01:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:02:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.55, 9.93, 9.61 [11:03:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.36, 9.92, 9.49 [11:06:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.86, 9.31, 9.30 [11:08:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:09:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.96, 9.90, 9.52 [11:15:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 9.76, 9.62 [11:19:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.56, 9.41, 9.24 [11:20:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.60, 9.18, 9.85 [11:20:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.64, 10.02, 9.70 [11:22:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.83, 9.14, 9.17 [11:23:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:23:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.09, 8.73, 8.10 [11:26:24] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790651318 [11:26:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:26:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.21, 8.40, 8.10 [11:27:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.47, 9.50, 9.26 [11:29:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.07, 10.33, 10.07 [11:32:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 9.64, 9.85 [11:32:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.64, 9.63, 9.76 [11:34:25] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790652468 [11:34:26] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790652715 [11:34:28] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790653473 [11:34:29] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790652464 [11:34:31] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790656487 [11:34:32] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790655673 [11:34:34] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790656163 [11:34:35] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r790652836 [11:35:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.34, 10.04, 9.94 [11:36:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.99, 9.24, 9.41 [11:37:18] PROBLEM - mw111 APT on mw111 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (15 critical updates). [11:37:23] [02mediawiki] 07HM100 commented on pull request 03#5033: Bump skins/Evelution from `d492062` to `3a68118` - 13https://github.com/miraheze/mediawiki/pull/5033#issuecomment-1020006451 [11:38:11] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (15 critical updates). [11:38:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 9.76, 9.83 [11:39:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 8.85, 8.35 [11:41:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.90, 8.15, 8.18 [11:43:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.88, 9.99, 9.76 [11:44:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.90, 10.19, 9.89 [11:44:30] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (15 critical updates). [11:44:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.88, 9.72, 9.50 [11:45:35] PROBLEM - mw102 APT on mw102 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (15 critical updates). [11:47:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.84, 9.91, 9.61 [11:49:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.41, 9.27, 9.53 [11:50:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.51, 10.16, 9.76 [11:52:23] PROBLEM - test101 APT on test101 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (15 critical updates). [11:52:33] PROBLEM - mw112 APT on mw112 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (15 critical updates). [11:52:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.75, 9.58, 9.59 [11:53:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.87, 9.30, 8.75 [11:56:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.46, 9.47, 9.61 [11:56:57] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (15 critical updates). [11:58:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.44, 9.46, 9.53 [11:59:17] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.04, 9.07, 9.38 [12:01:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.43, 10.63, 9.96 [12:04:41] PROBLEM - mw101 APT on mw101 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (15 critical updates). [12:05:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.24, 10.08, 9.19 [12:08:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.51, 9.62, 9.14 [12:09:38] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/e9282bef4932...f94e8541b2c9 [12:09:39] [02miraheze/GlobalNewFiles] 07translatewiki 03f94e854 - Localisation updates from https://translatewiki.net. [12:09:41] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/7bb72ad61979...1074b5d3da5d [12:09:42] [02miraheze/MatomoAnalytics] 07translatewiki 031074b5d - Localisation updates from https://translatewiki.net. [12:09:44] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/2495d9a95b54...d28e51f3ee06 [12:09:45] [02miraheze/ManageWiki] 07translatewiki 03d28e51f - Localisation updates from https://translatewiki.net. [12:09:47] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±4] 13https://github.com/miraheze/MirahezeMagic/compare/3d9c9bcacc7e...8f6e4dfa4855 [12:09:48] [02miraheze/MirahezeMagic] 07translatewiki 038f6e4df - Localisation updates from https://translatewiki.net. [12:10:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.65, 9.34, 9.82 [12:14:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.57, 8.56, 8.08 [12:17:53] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.83, 8.48, 8.14 [12:20:11] miraheze/GlobalNewFiles - translatewiki the build passed. [12:20:44] miraheze/MatomoAnalytics - translatewiki the build passed. [12:21:26] miraheze/ManageWiki - translatewiki the build passed. [12:22:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 9.75, 9.72 [12:25:03] miraheze/MirahezeMagic - translatewiki the build passed. [12:26:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.44, 9.55, 9.31 [12:28:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.83, 9.33, 9.90 [12:31:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.99, 9.61, 9.78 [12:34:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.23, 9.22, 9.58 [12:37:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.94, 9.14, 9.40 [12:42:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.16, 8.47, 7.90 [12:45:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.69, 7.74, 7.73 [13:00:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.11, 9.96, 9.57 [13:01:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.11, 7.70, 8.49 [13:04:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.04, 9.25, 9.85 [13:07:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.38, 9.71, 9.92 [13:08:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.01, 8.74, 8.76 [13:12:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.75, 9.72, 9.71 [13:13:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.80, 8.58, 8.14 [13:18:10] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.76, 11.00, 10.23 [13:19:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.09, 8.34, 8.20 [13:28:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.67, 9.26, 9.89 [13:31:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:31:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.69, 9.94 [13:33:57] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.83, 1.84, 1.55 [13:34:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.52, 9.53, 9.08 [13:35:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:37:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.43, 9.56, 9.19 [13:39:49] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.03, 1.59, 1.49 [13:41:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 9.08, 8.45 [13:42:47] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.27, 1.35, 1.41 [13:43:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.48, 10.10, 9.55 [13:50:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.36, 9.92, 9.11 [13:53:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.54, 9.57, 9.12 [13:57:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.54, 9.51, 9.14 [13:59:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.72, 9.11, 9.60 [14:00:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw101 mw102 mw111 mw112 mw121 [14:00:36] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw102 mw111 mw112 mw122 [14:00:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 5.45, 8.06, 8.68 [14:01:17] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw102 mw121 mw122 [14:01:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:02:17] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw111 mw112 mw121 mw122 [14:03:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.55, 7.97, 9.42 [14:03:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.02, 6.90, 8.09 [14:04:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.64, 7.89, 9.52 [14:05:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.51, 7.91, 9.76 [14:05:03] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:05:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.10, 6.33, 8.20 [14:06:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 5.76, 7.45, 9.35 [14:12:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.02, 6.51, 8.10 [14:13:08] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.79, 1.89, 1.57 [14:14:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.88, 6.87, 8.27 [14:14:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.40, 6.84, 8.44 [14:15:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.49, 7.29, 8.34 [14:16:07] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.71, 1.44, 1.45 [15:21:17] !log restart php on mw 102, 111,112,121,122 [15:22:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:33:55] !log disable jobrunner on mw101 [15:34:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:19] !log mw101: stop nginx, restart php, start nginx [15:36:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:37:19] PROBLEM - mw101 JobRunner Service on mw101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [15:39:45] PROBLEM - mw101 Puppet on mw101 is WARNING: WARNING: Puppet is currently disabled, message: RF1, last run 9 minutes ago with 0 failures [16:01:08] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [16:01:36] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [16:01:42] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [16:01:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:01:52] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [16:03:42] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:07:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.30, 7.74 [16:07:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 9.04, 7.72 [16:07:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 8.98, 7.53 [16:10:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 9.87, 8.27 [16:10:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.42, 8.68, 7.57 [16:10:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.10, 10.57, 8.52 [16:10:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.70, 9.41, 7.93 [16:13:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.44, 9.94, 8.56 [16:13:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.18, 9.70, 8.61 [16:13:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.31, 9.22, 8.14 [16:16:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.37, 9.41, 8.61 [16:16:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.65, 11.02, 9.28 [16:19:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.47, 9.83, 8.87 [16:19:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.76, 9.98, 8.52 [16:19:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.86, 9.83, 8.71 [16:20:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.04, 8.71, 8.11 [16:22:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.40, 9.81, 9.09 [16:22:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.67, 9.39, 8.58 [16:22:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.06, 9.37, 8.73 [16:24:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.25, 8.01, 7.94 [16:25:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 9.68, 9.48 [16:28:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.44, 10.12, 9.70 [16:28:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.47, 8.19, 8.40 [16:34:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.41, 9.87, 9.06 [16:34:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.65, 9.76, 9.17 [16:34:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.78, 9.75, 9.72 [16:37:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 9.24, 9.06 [16:39:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.26, 9.83, 8.95 [16:40:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.39, 9.43, 9.14 [16:40:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.88, 8.95, 8.17 [16:40:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.46, 9.94, 9.37 [16:41:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.89, 10.46, 9.96 [16:43:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 9.77, 9.32 [16:43:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.92, 9.52, 9.35 [16:46:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 9.93, 9.47 [16:46:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.06, 10.16, 9.65 [16:46:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.56, 8.86, 8.49 [16:48:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.19, 1.98, 1.67 [16:48:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.79, 9.27, 9.30 [16:49:49] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.64, 7.94, 8.17 [16:52:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.85, 9.69, 9.41 [16:52:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.83, 9.38, 9.55 [16:54:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.44, 10.25, 9.71 [16:57:47] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.61, 1.89, 1.80 [17:00:52] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.19, 2.01, 1.86 [17:01:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.61, 10.17, 9.80 [17:04:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.46, 8.59, 8.51 [17:05:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.19, 9.91, 9.76 [17:07:00] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.46, 1.66, 1.76 [17:07:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 9.77, 9.86 [17:07:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.59, 9.43, 9.74 [17:08:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.58, 10.59, 10.08 [17:09:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 4.15, 3.57 [17:10:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.73, 1.25, 1.58 [17:10:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.86, 10.09, 9.96 [17:10:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.04, 9.35, 8.82 [17:10:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.65, 10.53, 10.12 [17:11:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.49, 9.68, 9.83 [17:12:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.92, 3.58 [17:14:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.00, 10.51, 10.14 [17:15:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 3.31, 3.40 [17:16:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.13, 9.80, 9.93 [17:16:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.85, 9.95, 9.29 [17:23:11] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.51, 7.45, 8.43 [17:23:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.00, 8.47, 9.48 [17:23:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.57, 8.56, 9.49 [17:24:06] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw102 mw121 mw122 [17:24:07] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw102 mw121 mw122 [17:24:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:24:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.44, 8.55, 9.74 [17:24:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw101 mw102 mw121 mw122 [17:24:33] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:24:42] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw111 mw112 mw122 [17:26:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.68, 7.06, 8.49 [17:27:11] PROBLEM - gluster111 APT on gluster111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:30:31] PROBLEM - gluster111 APT on gluster111 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (13 critical updates). [17:30:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.34, 7.05, 8.40 [17:32:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.88, 6.91, 8.23 [17:34:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.76, 6.41, 8.01 [17:35:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:40:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:47:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:50:18] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.30, 2.21, 1.71 [17:51:01] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [17:51:13] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [17:51:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:51:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:51:40] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [17:51:47] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [17:53:14] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.16, 1.90, 1.70 [17:57:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.18, 9.85, 8.74 [17:59:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.38, 8.88, 7.98 [17:59:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.65, 8.73, 8.19 [18:00:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.79, 8.92, 8.19 [18:02:16] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.34, 1.48, 1.59 [18:02:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.83, 9.33, 8.49 [18:03:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 9.12, 8.35 [18:06:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 9.77, 8.79 [18:06:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.09, 10.67, 9.50 [18:09:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.79, 10.13, 9.13 [18:11:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.09, 9.49, 8.79 [18:14:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.23, 9.71, 9.63 [18:15:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.61, 9.97, 9.44 [18:17:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.57, 9.83, 9.68 [18:17:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.45, 9.29, 9.01 [18:27:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.45, 7.73, 8.32 [18:27:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:29:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 9.88, 9.98 [18:30:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.90, 8.90, 9.06 [18:30:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:30:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.44, 9.28, 9.83 [18:32:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.05, 9.96, 10.01 [18:36:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.57, 9.18, 9.21 [18:38:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.32, 9.69, 9.97 [18:39:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.95, 9.82, 9.74 [18:40:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:41:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.37, 10.03, 10.02 [18:43:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.02, 8.91, 9.41 [18:44:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:45:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.26, 10.30, 9.73 [18:46:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.18, 9.65, 9.63 [18:46:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.42, 9.55, 8.75 [18:47:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 9.38, 9.75 [18:49:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.99, 9.27, 9.48 [18:50:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.18, 9.04, 8.25 [18:50:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.90, 10.08, 9.96 [18:53:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.03, 9.21, 9.65 [18:58:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.09, 9.81, 9.48 [18:59:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.05, 9.25, 8.82 [19:01:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.67, 9.97, 9.63 [19:02:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.58, 9.54, 8.99 [19:03:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.35, 9.69, 9.54 [19:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.29, 9.82, 9.71 [19:05:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.03, 8.98, 8.84 [19:06:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.17, 9.05, 9.33 [19:07:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.22, 9.02, 9.43 [19:15:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.91, 3.29 [19:16:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.66, 9.31, 9.30 [19:17:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.04, 9.90, 9.36 [19:18:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.53, 3.24 [19:18:24] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±3] 13https://github.com/miraheze/puppet/compare/2e9ab7a97db4...a9d72049d4e2 [19:18:26] [02miraheze/puppet] 07JohnFLewis 03a9d7204 - implement new health checking methods in varnish/nginx [19:19:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.65, 9.78, 9.43 [19:22:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 9.86, 9.56 [19:23:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.85, 9.11, 9.32 [19:23:11] RECOVERY - cp31 APT on cp31 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:23:27] RECOVERY - db121 APT on db121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:23:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.29, 9.24, 9.29 [19:23:53] RECOVERY - bast121 APT on bast121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:07] RECOVERY - mem121 APT on mem121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:11] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:12] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:13] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:25] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:26] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:36] RECOVERY - bast101 APT on bast101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.56, 8.12, 9.62 [19:24:57] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.57, 2.21, 1.57 [19:24:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.77, 4.57, 3.68 [19:25:05] RECOVERY - cp21 APT on cp21 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:25:09] RECOVERY - mem101 APT on mem101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:25:32] RECOVERY - ldap111 APT on ldap111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:25:38] RECOVERY - test101 APT on test101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:08] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:10] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:16] RECOVERY - db101 APT on db101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:23] RECOVERY - db111 APT on db111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:36] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:38] RECOVERY - cp30 APT on cp30 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.70, 5.84, 3.89 [19:26:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.21, 4.42, 3.26 [19:26:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.93, 6.90, 8.88 [19:27:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.92, 6.86, 8.36 [19:27:04] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:27:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:27:09] RECOVERY - mw111 APT on mw111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:27:18] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:27:20] RECOVERY - cp20 APT on cp20 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:27:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.73, 6.10, 7.91 [19:27:52] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw101 mw102 mw112 mw121 mw122 [19:28:31] hmm lot of errors today [19:28:35] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:28:43] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.91, 1.66, 1.49 [19:28:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.22, 3.73, 3.56 [19:29:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:29:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:29:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 3.94, 5.93, 7.85 [19:29:46] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 5 backends are down. mw101 mw102 mw111 mw112 mw122 [19:29:56] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:29:58] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:30:00] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:30:01] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw101 mw111 mw112 mw121 [19:30:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.68, 6.50, 8.30 [19:30:22] RECOVERY - mon111 APT on mon111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:30:22] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.97, 1.68, 0.96 [19:30:47] RECOVERY - es101 APT on es101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:30:54] RECOVERY - gluster111 APT on gluster111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:11] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:17] RECOVERY - mail121 APT on mail121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:20] RECOVERY - es121 APT on es121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:45] RECOVERY - phab121 APT on phab121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.19, 3.39 [19:32:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:32:06] RECOVERY - es111 APT on es111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:32:44] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [19:32:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.16, 3.79, 3.71 [19:32:58] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [19:33:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.57, 3.82, 3.51 [19:33:16] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.49, 1.16, 0.89 [19:33:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:34:02] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [19:34:13] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:34:14] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:34:28] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 6 minutes ago with 15 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/gluster.gpg],File[wildcard.miraheze.org-2020-2],File[wildcard.miraheze.org-2020-2_private],File[glusterfs.pem] [19:34:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.50, 8.24, 8.62 [19:35:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.63, 2.76, 3.15 [19:36:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:36:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.23, 8.37, 8.54 [19:37:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 8.81, 8.79 [19:40:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.21, 11.16, 9.70 [19:40:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.51, 8.12, 7.86 [19:41:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.86, 2.74, 3.25 [19:41:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.40, 9.45, 8.77 [19:42:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.81, 10.15, 9.28 [19:43:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.35, 9.99, 9.52 [19:43:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.17, 8.42, 8.05 [19:44:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.01, 9.34, 8.86 [19:46:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.37, 10.11, 9.61 [19:47:03] PROBLEM - es121 APT on es121 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (8 critical updates). [19:47:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.92, 10.00, 8.92 [19:49:19] PROBLEM - es101 APT on es101 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [19:50:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.51, 8.84, 8.43 [19:51:35] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a9d72049d4e2...be3616ddf133 [19:51:36] [02miraheze/puppet] 07JohnFLewis 03be3616d - varnish: retain unhealthy backends 503 for health checks [19:51:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 9.59, 9.52 [19:52:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 9.54, 9.51 [19:53:16] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.79, 7.99, 8.17 [19:53:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.33, 9.95 [19:54:35] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [19:57:30] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:58:11] RECOVERY - gluster111 Puppet on gluster111 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:58:21] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/be3616ddf133...bc2fae6c2067 [19:58:23] [02miraheze/puppet] 07JohnFLewis 03bc2fae6 - varnish: use new health check [19:58:27] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/fbcad943a49f...6b77e749d2ad [19:58:28] [02miraheze/dns] 07JohnFLewis 036b77e74 - dns: use new varnish health check [19:58:32] paladox: ^^ [19:58:42] thanks! [19:59:16] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [20:00:21] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[retry-load-new-vcl-file] [20:00:23] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:00:42] JohnLewis: ^ [20:00:42] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [20:02:00] paladox: can you take a look for me and try and fix? [20:02:04] sure [20:02:09] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [20:04:05] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:04:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.54, 9.46 [20:04:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.54, 9.39, 9.79 [20:05:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:06:31] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:07:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.74, 9.72, 9.56 [20:07:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:07:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bc2fae6c2067...941853abf464 [20:07:46] [02miraheze/puppet] 07paladox 03941853a - varnish: Fix typo [20:07:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.96, 10.10, 9.99 [20:10:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/941853abf464...b825193bd8e8 [20:10:49] [02miraheze/puppet] 07paladox 03b825193 - varnish: Fix another typo [20:13:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.28, 9.69, 9.59 [20:14:43] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:15:10] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.42, 2.44, 1.85 [20:15:24] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:15:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.67, 9.38, 9.02 [20:15:58] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:16:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.93, 9.77, 9.62 [20:17:11] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:18:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.56, 1.95, 1.76 [20:18:47] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [20:18:50] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [20:19:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.31, 11.02, 10.10 [20:20:18] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw101 [20:20:41] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [20:21:11] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.11, 1.58, 1.65 [20:22:00] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:22:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.94, 9.74, 9.78 [20:22:55] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791122124 [20:27:04] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.53, 1.96, 1.41 [20:28:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.44, 8.73, 9.26 [20:29:18] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6b77e749d2ad...d4bb1d2719fe [20:29:19] [02miraheze/dns] 07paladox 03d4bb1d2 - 80 -> 443 [20:29:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.24, 8.21, 9.61 [20:31:17] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [20:31:30] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791127902 [20:31:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.28, 6.84, 8.41 [20:32:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.36, 7.68, 9.43 [20:33:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/3a1a72de6427 [20:33:39] [02miraheze/puppet] 07paladox 033a1a72d - Allow health.miraheze.org/check over 80 [20:33:40] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [20:33:42] [02puppet] 07paladox opened pull request 03#2388: Allow health.miraheze.org/check over 80 - 13https://github.com/miraheze/puppet/pull/2388 [20:34:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3a1a72de6427...81587cd8f742 [20:34:48] [02miraheze/puppet] 07paladox 0381587cd - Update mediawiki.conf [20:34:50] [02puppet] 07paladox synchronize pull request 03#2388: Allow health.miraheze.org/check over 80 - 13https://github.com/miraheze/puppet/pull/2388 [20:34:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b825193bd8e8...c65870f57b2c [20:34:58] [02miraheze/puppet] 07paladox 03c65870f - Allow health.miraheze.org/check over 80 (#2388) [20:35:00] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [20:35:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [20:35:03] [02puppet] 07paladox closed pull request 03#2388: Allow health.miraheze.org/check over 80 - 13https://github.com/miraheze/puppet/pull/2388 [20:35:23] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d4bb1d2719fe...b81baa965f02 [20:35:25] [02miraheze/dns] 07paladox 03b81baa9 - Revert "80 -> 443" [20:35:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.60, 6.74, 8.41 [20:37:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.03, 6.95, 8.15 [20:38:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.78, 6.44, 8.33 [20:38:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:39:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:42:22] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [20:42:54] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791135262 [20:43:07] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791135429 [20:43:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b7b7e9c83f56 [20:43:22] [02miraheze/puppet] 07paladox 03b7b7e9c - blackbox: Increase timeout for https_mediawiki_cp to 60s [20:43:24] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [20:43:25] [02puppet] 07paladox opened pull request 03#2389: blackbox: Increase timeout for https_mediawiki_cp to 60s - 13https://github.com/miraheze/puppet/pull/2389 [20:44:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/971100bc1a1a [20:44:02] [02miraheze/puppet] 07paladox 03971100b - cache proxies: Increase varnish::interval-timeout to 60s [20:44:04] [02puppet] 07paladox created branch 03paladox-patch-5-1 - 13https://github.com/miraheze/puppet [20:44:05] [02puppet] 07paladox opened pull request 03#2390: cache proxies: Increase varnish::interval-timeout to 60s - 13https://github.com/miraheze/puppet/pull/2390 [20:44:23] [02puppet] 07paladox closed pull request 03#2389: blackbox: Increase timeout for https_mediawiki_cp to 60s - 13https://github.com/miraheze/puppet/pull/2389 [20:44:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c65870f57b2c...36e6567caad1 [20:44:26] [02miraheze/puppet] 07paladox 0336e6567 - blackbox: Increase timeout for https_mediawiki_cp to 60s (#2389) [20:44:28] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [20:44:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [20:45:07] [02puppet] 07RhinosF1 reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791136751 [20:47:40] [02puppet] 07paladox synchronize pull request 03#2390: cache proxies: Increase varnish::interval-timeout to 60s - 13https://github.com/miraheze/puppet/pull/2390 [20:47:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/971100bc1a1a...1c1c389c4adf [20:47:43] [02miraheze/puppet] 07paladox 031c1c389 - Update common.yaml [20:47:50] [02puppet] 07paladox edited pull request 03#2390: cache proxies: Increase varnish::interval-check & varnish::interval-timeout to 60s - 13https://github.com/miraheze/puppet/pull/2390 [20:48:11] [02puppet] 07paladox closed pull request 03#2390: cache proxies: Increase varnish::interval-check & varnish::interval-timeout to 60s - 13https://github.com/miraheze/puppet/pull/2390 [20:48:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/36e6567caad1...c321e0ebea3b [20:48:14] [02miraheze/puppet] 07paladox 03c321e0e - cache proxies: Increase varnish::interval-check & varnish::interval-timeout to 60s (#2390) [20:48:15] [02puppet] 07paladox deleted branch 03paladox-patch-5-1 - 13https://github.com/miraheze/puppet [20:48:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5-1 [20:52:51] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791141935 [20:53:00] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791142038 [20:53:17] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791142246 [20:53:32] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791142468 [20:57:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.48, 9.06, 8.14 [20:57:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.38, 8.76, 8.10 [20:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.57, 9.71, 8.64 [20:58:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.07, 11.62, 9.55 [20:59:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.12, 2.40 [20:59:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.63, 9.56, 8.48 [21:00:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.07, 9.18, 8.38 [21:02:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 3.04, 2.50 [21:03:53] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.01, 1.39, 1.95 [21:06:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 9.94, 9.03 [21:08:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.23, 9.80, 9.20 [21:09:41] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.04, 1.20, 1.68 [21:15:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.20, 10.26, 9.43 [21:18:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.10, 9.86, 9.46 [21:19:26] !log restart php-fpm & nginx on mw101 [21:19:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:55] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.26, 2.15, 1.42 [21:30:49] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.82, 1.74, 1.37 [21:30:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.81, 3.32 [21:33:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.02, 9.32, 9.13 [21:33:18] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 11 backends are healthy [21:33:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.41, 1.30 [21:33:58] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 11 backends are healthy [21:34:08] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 11 backends are healthy [21:34:41] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 11 backends are healthy [21:36:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.53, 7.55, 8.47 [21:36:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.34, 4.15, 3.51 [21:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:38:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.84, 8.04, 8.49 [21:38:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.19, 9.00, 9.77 [21:39:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c321e0ebea3b...42ce311019ce [21:39:40] [02miraheze/puppet] 07paladox 0342ce311 - Only install es_exporter on es* [21:40:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/42ce311019ce...7f31aa192eb3 [21:40:13] [02miraheze/puppet] 07paladox 037f31aa1 - es_exporter: Fix package for bullseye [21:41:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/836f80ba1489 [21:41:07] [02miraheze/puppet] 07paladox 03836f80b - es_exporter: fix systemd unit [21:41:09] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [21:41:10] [02puppet] 07paladox opened pull request 03#2391: es_exporter: fix systemd unit - 13https://github.com/miraheze/puppet/pull/2391 [21:41:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/836f80ba1489...21c775d35700 [21:41:42] [02miraheze/puppet] 07paladox 0321c775d - Rename prometheus-es-exporter.systemd_override.erb to prometheus-elasticsearch-exporter.systemd_override.erb [21:41:44] [02puppet] 07paladox synchronize pull request 03#2391: es_exporter: fix systemd unit - 13https://github.com/miraheze/puppet/pull/2391 [21:42:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/21c775d35700...f977c95d8a6a [21:42:14] [02miraheze/puppet] 07paladox 03f977c95 - Update prometheus-elasticsearch-exporter.systemd_override.erb [21:42:16] [02puppet] 07paladox synchronize pull request 03#2391: es_exporter: fix systemd unit - 13https://github.com/miraheze/puppet/pull/2391 [21:42:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.78, 3.40 [21:42:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f977c95d8a6a...4f554b815206 [21:42:35] [02miraheze/puppet] 07paladox 034f554b8 - Update es_exporter.pp [21:42:36] [02puppet] 07paladox synchronize pull request 03#2391: es_exporter: fix systemd unit - 13https://github.com/miraheze/puppet/pull/2391 [21:42:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.98, 3.72 [21:42:48] [02puppet] 07paladox closed pull request 03#2391: es_exporter: fix systemd unit - 13https://github.com/miraheze/puppet/pull/2391 [21:42:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/miraheze/puppet/compare/7f31aa192eb3...38450bd92ab0 [21:42:51] [02miraheze/puppet] 07paladox 0338450bd - es_exporter: fix systemd unit (#2391) [21:42:53] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [21:42:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [21:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:45:12] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.37, 3.31 [21:45:41] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:45:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 4.42, 3.94 [21:46:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.19, 8.69, 8.64 [21:47:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 9.31, 8.92 [21:48:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.90, 3.82 [21:49:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.41, 7.90, 8.38 [21:51:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.49, 9.23, 9.87 [21:51:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.30, 4.34, 3.99 [21:51:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.74, 3.82, 3.50 [21:54:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 10.20, 10.16 [21:54:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.96, 3.90 [21:56:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-1/±1] 13https://github.com/miraheze/puppet/compare/38450bd92ab0...a32bfd592f78 [21:56:27] [02miraheze/puppet] 07paladox 03a32bfd5 - Use prometheus-es-exporter deb from wikimedia [21:56:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.39, 7.87, 8.41 [21:57:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 9.73, 9.99 [21:58:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.04, 9.80, 9.41 [21:59:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a32bfd592f78...ffc19dbae445 [21:59:37] [02miraheze/puppet] 07paladox 03ffc19db - es_exporter: ensure some python3 packages are installed [22:00:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.93, 3.86 [22:00:55] Spookreeeno: wipe your test3 af backup as it has PII in [22:01:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.51, 9.38, 9.30 [22:01:46] MirahezeBot: test3 is deleted anyway [22:06:42] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [22:07:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ffc19dbae445...35067f247159 [22:07:10] [02miraheze/puppet] 07paladox 0335067f2 - es_exporter: use https://es101.miraheze.org:9200 [22:08:35] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.56, 4.29, 2.83 [22:09:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 8.80, 9.24 [22:09:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.57, 3.71 [22:09:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.81, 3.50 [22:10:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.64, 9.21, 9.07 [22:11:36] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [22:12:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.13, 8.43, 9.05 [22:13:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.46, 8.80, 8.94 [22:14:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 8.22, 7.93 [22:14:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.96, 8.56, 7.64 [22:15:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.89, 9.35, 9.28 [22:15:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.16, 3.30 [22:16:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 9.21, 9.06 [22:18:21] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.30, 2.95, 3.27 [22:18:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.16, 8.90, 8.41 [22:19:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.08, 9.27, 9.11 [22:20:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.10, 8.34, 8.20 [22:20:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.48, 8.37, 7.92 [22:23:19] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 1.83, 3.72, 3.63 [22:25:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.47, 10.02, 9.42 [22:26:08] [02puppet] 07Universal-Omega reviewed pull request 03#2387 commit - 13https://github.com/miraheze/puppet/pull/2387#discussion_r791201104 [22:26:08] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 0.73, 2.44, 3.14 [22:27:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.21, 9.81, 8.93 [22:29:32] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [22:29:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 10.27, 9.10 [22:30:58] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [22:32:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 9.78, 9.19 [22:33:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 9.53, 9.58 [22:35:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.76, 9.57, 9.15 [22:35:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.53, 10.16, 9.46 [22:36:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.96, 9.13, 8.49 [22:37:56] !log deleted dupe RC entries on mwcosmos,cosmos and betawiki [22:38:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.66, 9.75, 9.43 [22:39:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.16, 3.82 [22:39:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.55, 10.35, 9.90 [22:39:19] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [22:41:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.91, 9.90, 9.38 [22:41:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.05, 9.38, 8.77 [22:42:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 9.81, 9.77 [22:42:57] !log and on meta [22:43:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:44:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.73, 9.63, 9.42 [22:44:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.16, 9.09, 8.75 [22:44:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.81, 9.77, 9.72 [22:45:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.91, 4.60, 4.14 [22:47:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.98, 2.46, 1.74 [22:47:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.22, 9.44, 9.63 [22:48:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.71, 3.89 [22:48:07] !log [rhinos@mwtask111] START fixRC on all wikis [22:48:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:48:46] !log [rhinos@mwtask111] START fixRC on all wikis [22:49:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:50:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.04, 9.11, 9.31 [22:50:36] !log [rhinos@mwtask111] START fixRC on all wikis [22:50:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.35, 9.89, 9.78 [22:50:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:51:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.58, 3.77 [22:51:43] !log [rhinos@mwtask111] START fixRC on all wikis [22:51:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 10.27, 9.70 [22:53:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.53, 9.76, 9.62 [22:53:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.76, 1.68 [22:53:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.79, 9.89, 9.10 [22:54:45] Phabricator isn't sending emails to me :/ [22:55:19] [02puppet] 07Universal-Omega synchronize pull request 03#2387: Cleanup and reorganise MediaWiki module - 13https://github.com/miraheze/puppet/pull/2387 [22:56:17] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.60, 1.63 [22:56:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.47, 9.42, 9.05 [22:56:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.30, 3.06, 3.50 [22:59:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.88, 9.68 [22:59:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.77, 9.73, 9.81 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:04] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.37, 8.94, 9.38 [23:02:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.37, 9.34, 9.05 [23:02:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.75, 10.40, 10.06 [23:05:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.97, 9.58, 9.19 [23:10:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.46, 9.82, 9.50 [23:11:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 2.91, 3.22 [23:13:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.97, 9.76, 9.98 [23:13:44] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 822s [23:13:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.35, 9.55, 9.24 [23:14:37] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for mw101 [23:14:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:39] paladox: what did you do with mw101 [23:16:33] Copied https://github.com/miraheze/puppet/commit/a9d72049d4e2314d2c0d11685011f355e8a28e40 php file and nginx config [23:16:34] [url] implement new health checking methods in varnish/nginx · miraheze/puppet@a9d7204 · GitHub | github.com [23:16:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.83, 9.42, 9.22 [23:17:55] paladox: it still claiming down [23:18:09] Varnish shows it’s up [23:18:19] And even the check recovered [23:18:26] weird [23:18:31] deploy disagrees [23:18:39] i can force in a minute [23:18:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/25279866ba62 [23:19:00] [02miraheze/mw-config] 07RhinosF1 032527986 - ext-list: register ES extensions [23:19:02] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [23:19:05] [02mw-config] 07RhinosF1 opened pull request 03#4435: ext-list: register ES extensions - 13https://github.com/miraheze/mw-config/pull/4435 [23:19:23] CosmicAlpha: ^ [23:19:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.84, 9.30, 9.48 [23:20:08] [02miraheze/mediawiki] 07RhinosF1 pushed 033 commits to 03REL1_37 [+9/-0/±189] 13https://github.com/miraheze/mediawiki/compare/ef1a891a7cf0...d3380237aa9f [23:20:09] [02miraheze/mediawiki] 07RhinosF1 031f0f67b - Merge branch 'wikimedia-REL1_37' into REL1_37 [23:20:11] [02miraheze/mediawiki] 07RhinosF1 0310ae164 - Merge branch 'REL1_37' of https://github.com/miraheze/mediawiki into REL1_37 [23:20:12] [02miraheze/mediawiki] 07RhinosF1 03d338023 - install ES extensions [23:20:15] miraheze/mw-config - RhinosF1 the build has errored. [23:21:24] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/25279866ba62...e371b392b6a3 [23:21:25] [02miraheze/mw-config] 07RhinosF1 03e371b39 - Update extension-list [23:21:27] [02mw-config] 07RhinosF1 synchronize pull request 03#4435: ext-list: register ES extensions - 13https://github.com/miraheze/mw-config/pull/4435 [23:22:25] miraheze/mw-config - RhinosF1 the build passed. [23:23:00] [02mw-config] 07RhinosF1 closed pull request 03#4435: ext-list: register ES extensions - 13https://github.com/miraheze/mw-config/pull/4435 [23:23:01] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [23:23:03] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [23:23:04] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3dc52a4ddb06...f9d63fb01031 [23:23:06] [02miraheze/mw-config] 07RhinosF1 03f9d63fb - ext-list: register ES extensions (#4435) [23:23:33] !log [rhinos@mwtask111] starting deploy of {'world': True} to scsvg [23:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:08] miraheze/mw-config - RhinosF1 the build passed. [23:24:40] !log [rhinos@test101] starting deploy of {'config': True, 'world': True, 'l10n': True, 'extensionlist': True} to skip [23:24:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.85, 9.65, 9.47 [23:28:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.10, 8.56, 9.09 [23:28:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.30, 7.04, 8.25 [23:28:57] CosmicAlpha: Elastica needs composer so do you want to add that bit so I don't conflict with you [23:30:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 0.87, 2.26, 1.61 [23:31:28] paladox: what port does ElasticSearch run on [23:31:42] 9200 [23:32:50] RhinosF1: I can add to my Pupprt PR but is that PR ready to deploy? [23:33:01] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.62, 1.58, 1.46 [23:33:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.42, 8.82, 8.85 [23:35:12] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/34c3fb13cd96 [23:35:13] [02miraheze/mw-config] 07RhinosF1 0334c3fb1 - Betawiki: configure guarded ES cluster [23:35:15] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [23:35:43] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/34c3fb13cd96...ca1fa92fdc07 [23:35:45] [02miraheze/mw-config] 07github-actions 03ca1fa92 - CI: lint code to MediaWiki standards [23:36:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 9.34, 9.08 [23:36:41] CosmicAlpha: will review in morning [23:36:48] [02mw-config] 07RhinosF1 opened pull request 03#4436: Betawiki: configure guarded ES cluster - 13https://github.com/miraheze/mw-config/pull/4436 [23:37:19] RhinosF1: OK [23:37:58] miraheze/mw-config - RhinosF1 the build passed. [23:38:23] !log [rhinos@test101] starting deploy of {'config': True, 'world': True, 'l10n': True, 'extensionlist': True} to skip [23:38:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:38:44] CosmicAlpha: fyi, i killed deploys because it was spewing strange error [23:39:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.02, 7.84, 8.45 [23:42:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.89, 8.60, 9.88 [23:42:39] RhinosF1: what errors? [23:42:53] CosmicAlpha: duplicate entries in config [23:43:01] a manual pull looks ok [23:43:09] !log [rhinos@test101] starting deploy of {'config': True, 'world': True, 'l10n': True, 'extensionlist': True} to skip [23:43:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:21] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: Strange gitconfig errors - rf1, last run 21 minutes ago with 0 failures [23:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:45:20] CosmicAlpha: i approved your puppet pr [23:45:31] !log [rhinos@mwtask111] starting deploy of {'world': True} to scsvg [23:45:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:18] RhinosF1: thanks! [23:46:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.46, 8.28, 8.53 [23:47:31] RhinosF1: I mean exact errors, what duplicate entries? Or is that all it says? [23:47:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.53, 7.54, 8.36 [23:48:08] CosmicAlpha: warning: 6f1ca9619d75aeda087ec87b79d50f65b607e35e:.gitmodules, multiple configurations found for 'submodule.extensions/.branch'. Skipping second one! [23:48:16] like one for each extension [23:49:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.89, 8.09, 8.44 [23:49:29] RhinosF1: oh that would be a merge conflict in world. Would happen if not rebased or reset. [23:49:36] !log [rhinos@mwtask111] DEPLOY ABORTED: Canary check failed for mw101 [23:49:41] CosmicAlpha: hmm [23:49:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:50] !log [rhinos@mwtask111] starting deploy of {'world': True, 'force': True} to scsvg [23:49:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:38] RhinosF1: I actually encountered that alot during my building of the labeler workflow. [23:50:46] ah [23:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:52:17] !log [rhinos@mw101] enabled puppet [23:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:54] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:54:17] RECOVERY - mw101 JobRunner Service on mw101 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:54:28] !log [rhinos@mwtask111] finished deploy of {'world': True, 'force': True} to scsvg - SUCCESS in 278s [23:54:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:15] !log [rhinos@mwtask111] starting deploy of {'config': True, 'l10n': True, 'extensionlist': True} to scsvg [23:55:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:56:11] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/d28e51f3ee06...669c93c43089 [23:56:12] [02miraheze/ManageWiki] 07Universal-Omega 03669c93c - Update config name to `$wgEnableUserEmailMuteList` [23:56:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.77, 9.40, 9.92 [23:58:52] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/669c93c43089...a0b7168b5c98 [23:58:53] [02miraheze/ManageWiki] 07Universal-Omega 03a0b7168 - Blacklist -> Exclude in comments