[00:00:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.72, 6.17, 6.54 [00:05:04] !log [@test3] starting deploy of {'config': True} to skip [00:05:06] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:05:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:31] !log [@mw11] starting deploy of {'config': True} to all [00:05:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:44] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 13s [00:05:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:14] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 6 minutes ago with 8 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d],File[/etc/logrotate.d/nginx],File[/home/johnflewis],File[/home/paladox] [00:06:20] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/collection.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php collection --disable (END - exit=0) [00:06:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.93, 27.80, 23.85 [00:08:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Arrays-75ff84f [+0/-0/±1] 13https://git.io/JKXpM [00:08:46] [02miraheze/mediawiki] 07dependabot[bot] 039c55d19 - Bump extensions/Arrays from `0a50da3` to `75ff84f` [00:08:48] [02mediawiki] 07dependabot[bot] opened pull request 03#3721: Bump extensions/Arrays from `0a50da3` to `75ff84f` - 13https://git.io/JKXpD [00:08:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Arrays-75ff84f - 13https://git.io/vbL5b [00:08:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#3721: Bump extensions/Arrays from `0a50da3` to `75ff84f` - 13https://git.io/JKXpD [00:08:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#3721: Bump extensions/Arrays from `0a50da3` to `75ff84f` - 13https://git.io/JKXpD [00:08:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Translate-a0a6505 [+0/-0/±1] 13https://git.io/JKXpH [00:08:55] [02miraheze/mediawiki] 07dependabot[bot] 039212b63 - Bump extensions/Translate from `ebc6560` to `a0a6505` [00:08:57] [02mediawiki] 07dependabot[bot] opened pull request 03#3722: Bump extensions/Translate from `ebc6560` to `a0a6505` - 13https://git.io/JKXpQ [00:08:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Translate-a0a6505 - 13https://git.io/vbL5b [00:09:00] [02mediawiki] 07dependabot[bot] labeled pull request 03#3722: Bump extensions/Translate from `ebc6560` to `a0a6505` - 13https://git.io/JKXpQ [00:09:01] [02mediawiki] 07dependabot[bot] labeled pull request 03#3722: Bump extensions/Translate from `ebc6560` to `a0a6505` - 13https://git.io/JKXpQ [00:09:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/SnapProjectEmbed-3085cdb [+0/-0/±1] 13https://git.io/JKXpd [00:09:04] [02miraheze/mediawiki] 07dependabot[bot] 0347b4774 - Bump extensions/SnapProjectEmbed from `d7916d9` to `3085cdb` [00:09:06] [02mediawiki] 07dependabot[bot] opened pull request 03#3723: Bump extensions/SnapProjectEmbed from `d7916d9` to `3085cdb` - 13https://git.io/JKXpF [00:09:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/SnapProjectEmbed-3085cdb - 13https://git.io/vbL5b [00:09:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#3723: Bump extensions/SnapProjectEmbed from `d7916d9` to `3085cdb` - 13https://git.io/JKXpF [00:09:10] [02mediawiki] 07dependabot[bot] labeled pull request 03#3723: Bump extensions/SnapProjectEmbed from `d7916d9` to `3085cdb` - 13https://git.io/JKXpF [00:09:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/vendor-17951d4 [+0/-0/±1] 13https://git.io/JKXpA [00:09:13] [02miraheze/mediawiki] 07dependabot[bot] 030a6ffec - Bump vendor from `8740687` to `17951d4` [00:09:15] [02mediawiki] 07dependabot[bot] opened pull request 03#3724: Bump vendor from `8740687` to `17951d4` - 13https://git.io/JKXpx [00:09:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/vendor-17951d4 - 13https://git.io/vbL5b [00:09:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#3724: Bump vendor from `8740687` to `17951d4` - 13https://git.io/JKXpx [00:09:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3724: Bump vendor from `8740687` to `17951d4` - 13https://git.io/JKXpx [00:09:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Variables-9dd1552 [+0/-0/±1] 13https://git.io/JKXpp [00:09:22] [02miraheze/mediawiki] 07dependabot[bot] 034cb8578 - Bump extensions/Variables from `3326474` to `9dd1552` [00:09:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Variables-9dd1552 - 13https://git.io/vbL5b [00:09:25] [02mediawiki] 07dependabot[bot] opened pull request 03#3725: Bump extensions/Variables from `3326474` to `9dd1552` - 13https://git.io/JKXph [00:09:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#3725: Bump extensions/Variables from `3326474` to `9dd1552` - 13https://git.io/JKXph [00:09:28] [02mediawiki] 07dependabot[bot] labeled pull request 03#3725: Bump extensions/Variables from `3326474` to `9dd1552` - 13https://git.io/JKXph [00:09:30] [02mediawiki] 07dependabot[bot] commented on pull request 03#3334: Bump vendor from `8740687` to `3528080` - 13https://git.io/JKXhe [00:09:31] [02mediawiki] 07dependabot[bot] closed pull request 03#3334: Bump vendor from `8740687` to `3528080` - 13https://git.io/JoSTi [00:09:33] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/vendor-3528080 - 13https://git.io/vbL5b [00:09:34] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/vendor-3528080 [00:09:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Editcount-1a6a63d [+0/-0/±1] 13https://git.io/JKXhf [00:09:37] [02miraheze/mediawiki] 07dependabot[bot] 0341dfa3d - Bump extensions/Editcount from `5598a47` to `1a6a63d` [00:09:39] [02mediawiki] 07dependabot[bot] opened pull request 03#3726: Bump extensions/Editcount from `5598a47` to `1a6a63d` - 13https://git.io/JKXhJ [00:09:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Editcount-1a6a63d - 13https://git.io/vbL5b [00:09:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#3726: Bump extensions/Editcount from `5598a47` to `1a6a63d` - 13https://git.io/JKXhJ [00:09:43] [02mediawiki] 07dependabot[bot] labeled pull request 03#3726: Bump extensions/Editcount from `5598a47` to `1a6a63d` - 13https://git.io/JKXhJ [00:09:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/CodeEditor-f0f99f4 [+0/-0/±1] 13https://git.io/JKXhT [00:09:46] [02miraheze/mediawiki] 07dependabot[bot] 039db3c61 - Bump extensions/CodeEditor from `9cd86af` to `f0f99f4` [00:09:48] [02mediawiki] 07dependabot[bot] opened pull request 03#3727: Bump extensions/CodeEditor from `9cd86af` to `f0f99f4` - 13https://git.io/JKXhk [00:09:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/CodeEditor-f0f99f4 - 13https://git.io/vbL5b [00:09:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#3727: Bump extensions/CodeEditor from `9cd86af` to `f0f99f4` - 13https://git.io/JKXhk [00:09:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#3727: Bump extensions/CodeEditor from `9cd86af` to `f0f99f4` - 13https://git.io/JKXhk [00:09:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Thanks-1c071f4 [+0/-0/±1] 13https://git.io/JKXhq [00:09:55] [02miraheze/mediawiki] 07dependabot[bot] 03bce6be7 - Bump extensions/Thanks from `8444017` to `1c071f4` [00:09:57] [02mediawiki] 07dependabot[bot] opened pull request 03#3728: Bump extensions/Thanks from `8444017` to `1c071f4` - 13https://git.io/JKXhm [00:09:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Thanks-1c071f4 - 13https://git.io/vbL5b [00:10:00] [02mediawiki] 07dependabot[bot] labeled pull request 03#3728: Bump extensions/Thanks from `8444017` to `1c071f4` - 13https://git.io/JKXhm [00:10:01] [02mediawiki] 07dependabot[bot] labeled pull request 03#3728: Bump extensions/Thanks from `8444017` to `1c071f4` - 13https://git.io/JKXhm [00:10:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/CategorySortHeaders-2afa8fd [+0/-0/±1] 13https://git.io/JKXh3 [00:10:04] [02miraheze/mediawiki] 07dependabot[bot] 0329f1bfd - Bump extensions/CategorySortHeaders from `be813c0` to `2afa8fd` [00:10:06] [02mediawiki] 07dependabot[bot] opened pull request 03#3729: Bump extensions/CategorySortHeaders from `be813c0` to `2afa8fd` - 13https://git.io/JKXhs [00:10:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/CategorySortHeaders-2afa8fd - 13https://git.io/vbL5b [00:10:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#3729: Bump extensions/CategorySortHeaders from `be813c0` to `2afa8fd` - 13https://git.io/JKXhs [00:10:10] [02mediawiki] 07dependabot[bot] labeled pull request 03#3729: Bump extensions/CategorySortHeaders from `be813c0` to `2afa8fd` - 13https://git.io/JKXhs [00:10:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/UserMerge-f1dffed [+0/-0/±1] 13https://git.io/JKXhZ [00:10:13] [02miraheze/mediawiki] 07dependabot[bot] 034707364 - Bump extensions/UserMerge from `a285dbc` to `f1dffed` [00:10:15] [02mediawiki] 07dependabot[bot] opened pull request 03#3730: Bump extensions/UserMerge from `a285dbc` to `f1dffed` - 13https://git.io/JKXhn [00:10:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/UserMerge-f1dffed - 13https://git.io/vbL5b [00:10:17] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.76, 6.86, 6.52 [00:10:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#3730: Bump extensions/UserMerge from `a285dbc` to `f1dffed` - 13https://git.io/JKXhn [00:10:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3730: Bump extensions/UserMerge from `a285dbc` to `f1dffed` - 13https://git.io/JKXhn [00:10:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ManageWiki-45bebc8 [+0/-0/±1] 13https://git.io/JKXhC [00:10:22] [02miraheze/mediawiki] 07dependabot[bot] 03a3cc6b6 - Bump extensions/ManageWiki from `7726a60` to `45bebc8` [00:10:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ManageWiki-45bebc8 - 13https://git.io/vbL5b [00:10:25] [02mediawiki] 07dependabot[bot] opened pull request 03#3731: Bump extensions/ManageWiki from `7726a60` to `45bebc8` - 13https://git.io/JKXhW [00:10:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#3731: Bump extensions/ManageWiki from `7726a60` to `45bebc8` - 13https://git.io/JKXhW [00:10:28] [02mediawiki] 07dependabot[bot] labeled pull request 03#3731: Bump extensions/ManageWiki from `7726a60` to `45bebc8` - 13https://git.io/JKXhW [00:10:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/GroupsSidebar-6e9cd86 [+0/-0/±1] 13https://git.io/JKXh4 [00:10:31] [02miraheze/mediawiki] 07dependabot[bot] 03add6cad - Bump extensions/GroupsSidebar from `caaec5a` to `6e9cd86` [00:10:33] [02mediawiki] 07dependabot[bot] opened pull request 03#3732: Bump extensions/GroupsSidebar from `caaec5a` to `6e9cd86` - 13https://git.io/JKXhB [00:10:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/GroupsSidebar-6e9cd86 - 13https://git.io/vbL5b [00:10:36] [02mediawiki] 07dependabot[bot] commented on pull request 03#3350: Bump extensions/ManageWiki from `7726a60` to `8925cab` - 13https://git.io/JKXhR [00:10:37] [02mediawiki] 07dependabot[bot] labeled pull request 03#3732: Bump extensions/GroupsSidebar from `caaec5a` to `6e9cd86` - 13https://git.io/JKXhB [00:10:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#3732: Bump extensions/GroupsSidebar from `caaec5a` to `6e9cd86` - 13https://git.io/JKXhB [00:10:40] [02mediawiki] 07dependabot[bot] closed pull request 03#3350: Bump extensions/ManageWiki from `7726a60` to `8925cab` - 13https://git.io/JoSkE [00:10:42] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/ManageWiki-8925cab - 13https://git.io/vbL5b [00:10:43] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/ManageWiki-8925cab [00:10:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/FlaggedRevs-29e3c8f [+0/-0/±1] 13https://git.io/JKXhE [00:10:46] [02miraheze/mediawiki] 07dependabot[bot] 0399ceee9 - Bump extensions/FlaggedRevs from `0169903` to `29e3c8f` [00:10:48] [02mediawiki] 07dependabot[bot] opened pull request 03#3733: Bump extensions/FlaggedRevs from `0169903` to `29e3c8f` - 13https://git.io/JKXhu [00:10:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/FlaggedRevs-29e3c8f - 13https://git.io/vbL5b [00:10:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#3733: Bump extensions/FlaggedRevs from `0169903` to `29e3c8f` - 13https://git.io/JKXhu [00:10:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#3733: Bump extensions/FlaggedRevs from `0169903` to `29e3c8f` - 13https://git.io/JKXhu [00:10:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Moderation-2b0bd8f [+0/-0/±1] 13https://git.io/JKXhz [00:10:55] [02miraheze/mediawiki] 07dependabot[bot] 033c9210c - Bump extensions/Moderation from `cbf109b` to `2b0bd8f` [00:10:57] [02mediawiki] 07dependabot[bot] opened pull request 03#3734: Bump extensions/Moderation from `cbf109b` to `2b0bd8f` - 13https://git.io/JKXh2 [00:10:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Moderation-2b0bd8f - 13https://git.io/vbL5b [00:11:00] [02mediawiki] 07dependabot[bot] labeled pull request 03#3734: Bump extensions/Moderation from `cbf109b` to `2b0bd8f` - 13https://git.io/JKXh2 [00:11:01] [02mediawiki] 07dependabot[bot] labeled pull request 03#3734: Bump extensions/Moderation from `cbf109b` to `2b0bd8f` - 13https://git.io/JKXh2 [00:11:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/cldr-de27f07 [+0/-0/±1] 13https://git.io/JKXho [00:11:04] [02miraheze/mediawiki] 07dependabot[bot] 034dc39ad - Bump extensions/cldr from `27dbc42` to `de27f07` [00:11:06] [02mediawiki] 07dependabot[bot] opened pull request 03#3735: Bump extensions/cldr from `27dbc42` to `de27f07` - 13https://git.io/JKXhK [00:11:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/cldr-de27f07 - 13https://git.io/vbL5b [00:11:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#3735: Bump extensions/cldr from `27dbc42` to `de27f07` - 13https://git.io/JKXhK [00:11:10] [02mediawiki] 07dependabot[bot] labeled pull request 03#3735: Bump extensions/cldr from `27dbc42` to `de27f07` - 13https://git.io/JKXhK [00:11:12] [02mediawiki] 07dependabot[bot] commented on pull request 03#3624: Bump extensions/cldr from `27dbc42` to `045ad57` - 13https://git.io/JKXhX [00:11:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/SpriteSheet-d62639c [+0/-0/±1] 13https://git.io/JKXh1 [00:11:15] [02miraheze/mediawiki] 07dependabot[bot] 037acdecf - Bump extensions/SpriteSheet from `d62d68a` to `d62639c` [00:11:16] [02mediawiki] 07dependabot[bot] closed pull request 03#3624: Bump extensions/cldr from `27dbc42` to `045ad57` - 13https://git.io/Jo94p [00:11:18] [02mediawiki] 07dependabot[bot] opened pull request 03#3736: Bump extensions/SpriteSheet from `d62d68a` to `d62639c` - 13https://git.io/JKXhM [00:11:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/SpriteSheet-d62639c - 13https://git.io/vbL5b [00:11:21] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/cldr-045ad57 - 13https://git.io/vbL5b [00:11:22] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/cldr-045ad57 [00:11:24] [02mediawiki] 07dependabot[bot] labeled pull request 03#3736: Bump extensions/SpriteSheet from `d62d68a` to `d62639c` - 13https://git.io/JKXhM [00:11:25] [02mediawiki] 07dependabot[bot] labeled pull request 03#3736: Bump extensions/SpriteSheet from `d62d68a` to `d62639c` - 13https://git.io/JKXhM [00:11:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Lingo-e4e6dcb [+0/-0/±1] 13https://git.io/JKXhS [00:11:28] [02miraheze/mediawiki] 07dependabot[bot] 038ef2aeb - Bump extensions/Lingo from `907f4ee` to `e4e6dcb` [00:11:30] [02mediawiki] 07dependabot[bot] opened pull request 03#3737: Bump extensions/Lingo from `907f4ee` to `e4e6dcb` - 13https://git.io/JKXh9 [00:11:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Lingo-e4e6dcb - 13https://git.io/vbL5b [00:11:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#3737: Bump extensions/Lingo from `907f4ee` to `e4e6dcb` - 13https://git.io/JKXh9 [00:11:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#3737: Bump extensions/Lingo from `907f4ee` to `e4e6dcb` - 13https://git.io/JKXh9 [00:11:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/DataDump-971284c [+0/-0/±1] 13https://git.io/JKXhQ [00:11:37] [02miraheze/mediawiki] 07dependabot[bot] 03386a0dc - Bump extensions/DataDump from `5ab51f3` to `971284c` [00:11:39] [02mediawiki] 07dependabot[bot] opened pull request 03#3738: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JKXh7 [00:11:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/DataDump-971284c - 13https://git.io/vbL5b [00:11:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageSchemas-410b5a6 [+0/-0/±1] 13https://git.io/JKXhd [00:11:43] [02miraheze/mediawiki] 07dependabot[bot] 03210f88a - Bump extensions/PageSchemas from `a698ba2` to `410b5a6` [00:11:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageSchemas-410b5a6 - 13https://git.io/vbL5b [00:11:46] [02mediawiki] 07dependabot[bot] labeled pull request 03#3738: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JKXh7 [00:11:48] [02mediawiki] 07dependabot[bot] opened pull request 03#3739: Bump extensions/PageSchemas from `a698ba2` to `410b5a6` - 13https://git.io/JKXhF [00:11:49] [02mediawiki] 07dependabot[bot] labeled pull request 03#3738: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JKXh7 [00:11:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#3739: Bump extensions/PageSchemas from `a698ba2` to `410b5a6` - 13https://git.io/JKXhF [00:11:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#3739: Bump extensions/PageSchemas from `a698ba2` to `410b5a6` - 13https://git.io/JKXhF [00:11:54] [02mediawiki] 07dependabot[bot] commented on pull request 03#3379: Bump extensions/DataDump from `5ab51f3` to `45180cd` - 13https://git.io/JKXhN [00:11:55] [02mediawiki] 07dependabot[bot] closed pull request 03#3379: Bump extensions/DataDump from `5ab51f3` to `45180cd` - 13https://git.io/JoSIj [00:11:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/DataDump-45180cd - 13https://git.io/vbL5b [00:11:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MintyDocs-b9d49f6 [+0/-0/±1] 13https://git.io/JKXhA [00:12:00] [02miraheze/mediawiki] 07dependabot[bot] 03e59e0bc - Bump extensions/MintyDocs from `b923810` to `b9d49f6` [00:12:01] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/DataDump-45180cd [00:12:03] [02mediawiki] 07dependabot[bot] opened pull request 03#3740: Bump extensions/MintyDocs from `b923810` to `b9d49f6` - 13https://git.io/JKXhp [00:12:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MintyDocs-b9d49f6 - 13https://git.io/vbL5b [00:12:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#3740: Bump extensions/MintyDocs from `b923810` to `b9d49f6` - 13https://git.io/JKXhp [00:12:07] [02mediawiki] 07dependabot[bot] labeled pull request 03#3740: Bump extensions/MintyDocs from `b923810` to `b9d49f6` - 13https://git.io/JKXhp [00:12:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/GlobalWatchlist-d4d6b47 [+0/-0/±1] 13https://git.io/JKXhj [00:12:10] [02miraheze/mediawiki] 07dependabot[bot] 037f64a76 - Bump extensions/GlobalWatchlist from `0aca229` to `d4d6b47` [00:12:12] [02mediawiki] 07dependabot[bot] opened pull request 03#3741: Bump extensions/GlobalWatchlist from `0aca229` to `d4d6b47` - 13https://git.io/JKXje [00:12:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/GlobalWatchlist-d4d6b47 - 13https://git.io/vbL5b [00:12:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#3741: Bump extensions/GlobalWatchlist from `0aca229` to `d4d6b47` - 13https://git.io/JKXje [00:12:16] [02mediawiki] 07dependabot[bot] labeled pull request 03#3741: Bump extensions/GlobalWatchlist from `0aca229` to `d4d6b47` - 13https://git.io/JKXje [00:12:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Nostalgia-18dc88f [+0/-0/±1] 13https://git.io/JKXjv [00:12:19] [02miraheze/mediawiki] 07dependabot[bot] 0341a9fbb - Bump skins/Nostalgia from `677d208` to `18dc88f` [00:12:21] [02mediawiki] 07dependabot[bot] opened pull request 03#3742: Bump skins/Nostalgia from `677d208` to `18dc88f` - 13https://git.io/JKXjf [00:12:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Nostalgia-18dc88f - 13https://git.io/vbL5b [00:12:24] [02mediawiki] 07dependabot[bot] labeled pull request 03#3742: Bump skins/Nostalgia from `677d208` to `18dc88f` - 13https://git.io/JKXjf [00:12:25] [02mediawiki] 07dependabot[bot] labeled pull request 03#3742: Bump skins/Nostalgia from `677d208` to `18dc88f` - 13https://git.io/JKXjf [00:12:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PortableInfobox-359ba27 [+0/-0/±1] 13https://git.io/JKXjT [00:12:28] [02miraheze/mediawiki] 07dependabot[bot] 03af95efe - Bump extensions/PortableInfobox from `06f22b2` to `359ba27` [00:12:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PortableInfobox-359ba27 - 13https://git.io/vbL5b [00:12:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [00:12:31] [02mediawiki] 07dependabot[bot] opened pull request 03#3743: Bump extensions/PortableInfobox from `06f22b2` to `359ba27` - 13https://git.io/JKXjk [00:12:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#3743: Bump extensions/PortableInfobox from `06f22b2` to `359ba27` - 13https://git.io/JKXjk [00:12:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#3743: Bump extensions/PortableInfobox from `06f22b2` to `359ba27` - 13https://git.io/JKXjk [00:12:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Newsletter-ed69f55 [+0/-0/±1] 13https://git.io/JKXjt [00:12:37] [02miraheze/mediawiki] 07dependabot[bot] 039ad3411 - Bump extensions/Newsletter from `8cd91bf` to `ed69f55` [00:12:39] [02mediawiki] 07dependabot[bot] opened pull request 03#3744: Bump extensions/Newsletter from `8cd91bf` to `ed69f55` - 13https://git.io/JKXjq [00:12:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Newsletter-ed69f55 - 13https://git.io/vbL5b [00:12:42] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKXjm [00:12:43] [02miraheze/services] 07MirahezeSSLBot 037e4483b - BOT: Updating services config for wikis [00:12:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#3744: Bump extensions/Newsletter from `8cd91bf` to `ed69f55` - 13https://git.io/JKXjq [00:12:46] [02mediawiki] 07dependabot[bot] labeled pull request 03#3744: Bump extensions/Newsletter from `8cd91bf` to `ed69f55` - 13https://git.io/JKXjq [00:12:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-23396f2 [+0/-0/±1] 13https://git.io/JKXjY [00:12:49] [02miraheze/mediawiki] 07dependabot[bot] 038819c36 - Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` [00:12:51] [02mediawiki] 07dependabot[bot] opened pull request 03#3745: Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` - 13https://git.io/JKXjO [00:12:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-23396f2 - 13https://git.io/vbL5b [00:12:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#3745: Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` - 13https://git.io/JKXjO [00:12:55] [02mediawiki] 07dependabot[bot] labeled pull request 03#3745: Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` - 13https://git.io/JKXjO [00:12:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/InputBox-aa70764 [+0/-0/±1] 13https://git.io/JKXjs [00:12:58] [02miraheze/mediawiki] 07dependabot[bot] 0393ebdd2 - Bump extensions/InputBox from `1b80260` to `aa70764` [00:13:00] [02mediawiki] 07dependabot[bot] opened pull request 03#3746: Bump extensions/InputBox from `1b80260` to `aa70764` - 13https://git.io/JKXjG [00:13:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/InputBox-aa70764 - 13https://git.io/vbL5b [00:13:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#3746: Bump extensions/InputBox from `1b80260` to `aa70764` - 13https://git.io/JKXjG [00:13:04] [02mediawiki] 07dependabot[bot] labeled pull request 03#3746: Bump extensions/InputBox from `1b80260` to `aa70764` - 13https://git.io/JKXjG [00:13:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Timeline-ef49888 [+0/-0/±1] 13https://git.io/JKXjW [00:13:07] [02miraheze/mediawiki] 07dependabot[bot] 03e568b7c - Bump extensions/Timeline from `f9fda91` to `ef49888` [00:13:09] [02mediawiki] 07dependabot[bot] opened pull request 03#3747: Bump extensions/Timeline from `f9fda91` to `ef49888` - 13https://git.io/JKXjl [00:13:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Timeline-ef49888 - 13https://git.io/vbL5b [00:13:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#3747: Bump extensions/Timeline from `f9fda91` to `ef49888` - 13https://git.io/JKXjl [00:13:13] [02mediawiki] 07dependabot[bot] labeled pull request 03#3747: Bump extensions/Timeline from `f9fda91` to `ef49888` - 13https://git.io/JKXjl [00:13:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageTriage-fd05e35 [+0/-0/±1] 13https://git.io/JKXjB [00:13:16] [02miraheze/mediawiki] 07dependabot[bot] 03f3f514d - Bump extensions/PageTriage from `b782dbe` to `fd05e35` [00:13:18] [02mediawiki] 07dependabot[bot] opened pull request 03#3748: Bump extensions/PageTriage from `b782dbe` to `fd05e35` - 13https://git.io/JKXj0 [00:13:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageTriage-fd05e35 - 13https://git.io/vbL5b [00:13:21] [02mediawiki] 07dependabot[bot] labeled pull request 03#3748: Bump extensions/PageTriage from `b782dbe` to `fd05e35` - 13https://git.io/JKXj0 [00:13:22] [02mediawiki] 07dependabot[bot] labeled pull request 03#3748: Bump extensions/PageTriage from `b782dbe` to `fd05e35` - 13https://git.io/JKXj0 [00:13:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/DynamicPageList-4a70dbd [+0/-0/±1] 13https://git.io/JKXjE [00:13:25] [02miraheze/mediawiki] 07dependabot[bot] 03a011208 - Bump extensions/DynamicPageList from `6ab9c9c` to `4a70dbd` [00:13:27] [02mediawiki] 07dependabot[bot] opened pull request 03#3749: Bump extensions/DynamicPageList from `6ab9c9c` to `4a70dbd` - 13https://git.io/JKXju [00:13:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/DynamicPageList-4a70dbd - 13https://git.io/vbL5b [00:13:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.18, 6.94, 6.65 [00:13:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#3749: Bump extensions/DynamicPageList from `6ab9c9c` to `4a70dbd` - 13https://git.io/JKXju [00:13:31] [02mediawiki] 07dependabot[bot] labeled pull request 03#3749: Bump extensions/DynamicPageList from `6ab9c9c` to `4a70dbd` - 13https://git.io/JKXju [00:13:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Medik-b491fc8 [+0/-0/±1] 13https://git.io/JKXjg [00:13:34] [02miraheze/mediawiki] 07dependabot[bot] 035f48b39 - Bump skins/Medik from `c52dc4d` to `b491fc8` [00:13:36] [02mediawiki] 07dependabot[bot] opened pull request 03#3750: Bump skins/Medik from `c52dc4d` to `b491fc8` - 13https://git.io/JKXj2 [00:13:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Medik-b491fc8 - 13https://git.io/vbL5b [00:13:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#3750: Bump skins/Medik from `c52dc4d` to `b491fc8` - 13https://git.io/JKXj2 [00:13:40] [02mediawiki] 07dependabot[bot] labeled pull request 03#3750: Bump skins/Medik from `c52dc4d` to `b491fc8` - 13https://git.io/JKXj2 [00:13:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/SandboxLink-c14731c [+0/-0/±1] 13https://git.io/JKXjV [00:13:43] [02miraheze/mediawiki] 07dependabot[bot] 031efefa1 - Bump extensions/SandboxLink from `578e583` to `c14731c` [00:13:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/SandboxLink-c14731c - 13https://git.io/vbL5b [00:13:46] [02mediawiki] 07dependabot[bot] opened pull request 03#3751: Bump extensions/SandboxLink from `578e583` to `c14731c` - 13https://git.io/JKXjw [00:13:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#3751: Bump extensions/SandboxLink from `578e583` to `c14731c` - 13https://git.io/JKXjw [00:13:49] [02mediawiki] 07dependabot[bot] labeled pull request 03#3751: Bump extensions/SandboxLink from `578e583` to `c14731c` - 13https://git.io/JKXjw [00:13:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/WikiDiscover-749ead5 [+0/-0/±1] 13https://git.io/JKXjK [00:13:52] [02miraheze/mediawiki] 07dependabot[bot] 0328745d4 - Bump extensions/WikiDiscover from `f2fc860` to `749ead5` [00:13:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/WikiDiscover-749ead5 - 13https://git.io/vbL5b [00:13:55] [02mediawiki] 07dependabot[bot] opened pull request 03#3752: Bump extensions/WikiDiscover from `f2fc860` to `749ead5` - 13https://git.io/JKXj6 [00:13:57] [02mediawiki] 07dependabot[bot] labeled pull request 03#3752: Bump extensions/WikiDiscover from `f2fc860` to `749ead5` - 13https://git.io/JKXj6 [00:13:58] [02mediawiki] 07dependabot[bot] labeled pull request 03#3752: Bump extensions/WikiDiscover from `f2fc860` to `749ead5` - 13https://git.io/JKXj6 [00:14:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#3358: Bump extensions/WikiDiscover from `f2fc860` to `e03c07d` - 13https://git.io/JKXjy [00:14:01] [02mediawiki] 07dependabot[bot] closed pull request 03#3358: Bump extensions/WikiDiscover from `f2fc860` to `e03c07d` - 13https://git.io/JoSkx [00:14:03] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/WikiDiscover-e03c07d - 13https://git.io/vbL5b [00:14:04] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/WikiDiscover-e03c07d [00:14:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PollNY-ff9e754 [+0/-0/±1] 13https://git.io/JK1eJ [00:14:07] [02miraheze/mediawiki] 07dependabot[bot] 03aab0a35 - Bump extensions/PollNY from `fb88c8f` to `ff9e754` [00:14:09] [02mediawiki] 07dependabot[bot] opened pull request 03#3753: Bump extensions/PollNY from `fb88c8f` to `ff9e754` - 13https://git.io/JK1eU [00:14:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PollNY-ff9e754 - 13https://git.io/vbL5b [00:14:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#3753: Bump extensions/PollNY from `fb88c8f` to `ff9e754` - 13https://git.io/JK1eU [00:14:13] [02mediawiki] 07dependabot[bot] labeled pull request 03#3753: Bump extensions/PollNY from `fb88c8f` to `ff9e754` - 13https://git.io/JK1eU [00:14:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageForms-6c11e00 [+0/-0/±1] 13https://git.io/JK1eG [00:14:16] [02miraheze/mediawiki] 07dependabot[bot] 03413f590 - Bump extensions/PageForms from `fabcd30` to `6c11e00` [00:14:18] [02mediawiki] 07dependabot[bot] opened pull request 03#3754: Bump extensions/PageForms from `fabcd30` to `6c11e00` - 13https://git.io/JK1eZ [00:14:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageForms-6c11e00 - 13https://git.io/vbL5b [00:14:21] [02mediawiki] 07dependabot[bot] labeled pull request 03#3754: Bump extensions/PageForms from `fabcd30` to `6c11e00` - 13https://git.io/JK1eZ [00:14:22] [02mediawiki] 07dependabot[bot] labeled pull request 03#3754: Bump extensions/PageForms from `fabcd30` to `6c11e00` - 13https://git.io/JK1eZ [00:14:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Citizen-e9e7d1c [+0/-0/±1] 13https://git.io/JK1e4 [00:14:25] [02miraheze/mediawiki] 07dependabot[bot] 0312cc0fb - Bump skins/Citizen from `26cfa2d` to `e9e7d1c` [00:14:27] [02mediawiki] 07dependabot[bot] opened pull request 03#3755: Bump skins/Citizen from `26cfa2d` to `e9e7d1c` - 13https://git.io/JK1eB [00:14:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Citizen-e9e7d1c - 13https://git.io/vbL5b [00:14:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#3755: Bump skins/Citizen from `26cfa2d` to `e9e7d1c` - 13https://git.io/JK1eB [00:14:31] [02mediawiki] 07dependabot[bot] labeled pull request 03#3755: Bump skins/Citizen from `26cfa2d` to `e9e7d1c` - 13https://git.io/JK1eB [00:14:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/SocialProfile-aebbc70 [+0/-0/±1] 13https://git.io/JK1e6 [00:14:34] [02miraheze/mediawiki] 07dependabot[bot] 03a48256e - Bump extensions/SocialProfile from `41cd719` to `aebbc70` [00:14:36] [02mediawiki] 07dependabot[bot] opened pull request 03#3756: Bump extensions/SocialProfile from `41cd719` to `aebbc70` - 13https://git.io/JK1ei [00:14:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-aebbc70 - 13https://git.io/vbL5b [00:14:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#3756: Bump extensions/SocialProfile from `41cd719` to `aebbc70` - 13https://git.io/JK1ei [00:14:40] [02mediawiki] 07dependabot[bot] labeled pull request 03#3756: Bump extensions/SocialProfile from `41cd719` to `aebbc70` - 13https://git.io/JK1ei [00:14:42] [02mediawiki] 07dependabot[bot] commented on pull request 03#3627: Bump extensions/SocialProfile from `41cd719` to `3907194` - 13https://git.io/JK1ey [00:14:43] [02mediawiki] 07dependabot[bot] closed pull request 03#3627: Bump extensions/SocialProfile from `41cd719` to `3907194` - 13https://git.io/Jo9B0 [00:14:45] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-3907194 - 13https://git.io/vbL5b [00:14:46] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-3907194 [00:14:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/QuizGame-b267cc6 [+0/-0/±1] 13https://git.io/JK1ed [00:14:49] [02miraheze/mediawiki] 07dependabot[bot] 03bc8f6ac - Bump extensions/QuizGame from `ecb9da2` to `b267cc6` [00:14:51] [02mediawiki] 07dependabot[bot] opened pull request 03#3757: Bump extensions/QuizGame from `ecb9da2` to `b267cc6` - 13https://git.io/JK1eF [00:14:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/QuizGame-b267cc6 - 13https://git.io/vbL5b [00:14:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#3757: Bump extensions/QuizGame from `ecb9da2` to `b267cc6` - 13https://git.io/JK1eF [00:14:55] [02mediawiki] 07dependabot[bot] labeled pull request 03#3757: Bump extensions/QuizGame from `ecb9da2` to `b267cc6` - 13https://git.io/JK1eF [00:14:57] [02mediawiki] 07dependabot[bot] opened pull request 03#3758: Bump extensions/Cargo from `19b7b42` to `177f242` - 13https://git.io/JK1eb [00:14:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Cargo-177f242 [+0/-0/±1] 13https://git.io/JK1eN [00:15:00] [02miraheze/mediawiki] 07dependabot[bot] 031aac5de - Bump extensions/Cargo from `19b7b42` to `177f242` [00:15:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Cargo-177f242 - 13https://git.io/vbL5b [00:15:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#3758: Bump extensions/Cargo from `19b7b42` to `177f242` - 13https://git.io/JK1eb [00:15:04] [02mediawiki] 07dependabot[bot] labeled pull request 03#3758: Bump extensions/Cargo from `19b7b42` to `177f242` - 13https://git.io/JK1eb [00:15:06] [02mediawiki] 07dependabot[bot] commented on pull request 03#3429: Bump extensions/Cargo from `19b7b42` to `d378fdd` - 13https://git.io/JK1ex [00:15:07] [02mediawiki] 07dependabot[bot] closed pull request 03#3429: Bump extensions/Cargo from `19b7b42` to `d378fdd` - 13https://git.io/JoSqS [00:15:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Evelution-4bbe94b [+0/-0/±1] 13https://git.io/JK1eh [00:15:10] [02miraheze/mediawiki] 07dependabot[bot] 033795977 - Bump skins/Evelution from `55362e4` to `4bbe94b` [00:15:12] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Cargo-d378fdd - 13https://git.io/vbL5b [00:15:13] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Cargo-d378fdd [00:15:15] [02mediawiki] 07dependabot[bot] opened pull request 03#3759: Bump skins/Evelution from `55362e4` to `4bbe94b` - 13https://git.io/JK1ve [00:15:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Evelution-4bbe94b - 13https://git.io/vbL5b [00:15:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#3759: Bump skins/Evelution from `55362e4` to `4bbe94b` - 13https://git.io/JK1ve [00:15:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3759: Bump skins/Evelution from `55362e4` to `4bbe94b` - 13https://git.io/JK1ve [00:15:21] [02mediawiki] 07dependabot[bot] commented on pull request 03#3385: Bump skins/Evelution from `55362e4` to `a343d71` - 13https://git.io/JK1vJ [00:15:22] [02mediawiki] 07dependabot[bot] closed pull request 03#3385: Bump skins/Evelution from `55362e4` to `a343d71` - 13https://git.io/JoSLc [00:15:24] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Evelution-a343d71 [00:15:25] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Evelution-a343d71 - 13https://git.io/vbL5b [00:15:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/CreateWiki-8a4db79 [+0/-0/±1] 13https://git.io/JK1vU [00:15:28] [02miraheze/mediawiki] 07dependabot[bot] 0342f1408 - Bump extensions/CreateWiki from `ffffd57` to `8a4db79` [00:15:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-8a4db79 - 13https://git.io/vbL5b [00:15:31] [02mediawiki] 07dependabot[bot] opened pull request 03#3760: Bump extensions/CreateWiki from `ffffd57` to `8a4db79` - 13https://git.io/JK1vT [00:15:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#3760: Bump extensions/CreateWiki from `ffffd57` to `8a4db79` - 13https://git.io/JK1vT [00:15:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#3760: Bump extensions/CreateWiki from `ffffd57` to `8a4db79` - 13https://git.io/JK1vT [00:15:36] [02mediawiki] 07dependabot[bot] opened pull request 03#3761: Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` - 13https://git.io/JK1vk [00:15:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ShortDescription-d764a1f [+0/-0/±1] 13https://git.io/JK1vI [00:15:39] [02miraheze/mediawiki] 07dependabot[bot] 0373448d1 - Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` [00:15:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ShortDescription-d764a1f - 13https://git.io/vbL5b [00:15:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#3761: Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` - 13https://git.io/JK1vk [00:15:43] [02mediawiki] 07dependabot[bot] labeled pull request 03#3761: Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` - 13https://git.io/JK1vk [00:15:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Maps-5d39abc [+0/-0/±1] 13https://git.io/JK1vn [00:15:46] [02miraheze/mediawiki] 07dependabot[bot] 03acc3c8b - Bump extensions/Maps from `b43865b` to `5d39abc` [00:15:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Maps-5d39abc - 13https://git.io/vbL5b [00:15:49] [02mediawiki] 07dependabot[bot] opened pull request 03#3762: Bump extensions/Maps from `b43865b` to `5d39abc` - 13https://git.io/JK1vc [00:15:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#3762: Bump extensions/Maps from `b43865b` to `5d39abc` - 13https://git.io/JK1vc [00:15:52] [02mediawiki] 07dependabot[bot] labeled pull request 03#3762: Bump extensions/Maps from `b43865b` to `5d39abc` - 13https://git.io/JK1vc [00:17:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.19, 6.12, 6.38 [00:17:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [00:23:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-2/±1] 13https://git.io/JK1Th [00:23:05] [02miraheze/mediawiki] 07Universal-Omega 033f1dfbf - Remove AdvancedSearch and MediaModeration [00:23:06] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [00:25:43] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-2/±1] 13https://git.io/JK1IY [00:25:45] [02miraheze/mediawiki] 07Universal-Omega 031b35656 - Remove AdvancedSearch and MediaModeration [00:25:46] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [00:27:50] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-1/±1] 13https://git.io/JK1LI [00:27:52] [02miraheze/mediawiki] 07Universal-Omega 03372fe0a - Remove WebP [00:27:53] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbL5b [00:29:44] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-1/±1] 13https://git.io/JK1tT [00:29:45] [02miraheze/mediawiki] 07Universal-Omega 03aa48275 - Remove WebP [00:29:47] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbL5b [00:32:26] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:32:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.70, 4.18, 3.56 [00:36:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.68, 3.46 [00:36:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [00:37:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.21, 9.36, 7.46 [00:37:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.66, 10.86, 8.69 [00:39:36] PROBLEM - cp14 Stunnel Http for mw12 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:41:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 167.114.2.161/cpweb [00:41:28] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.05, 7.50, 7.10 [00:43:08] RECOVERY - cp14 Stunnel Http for mw12 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.074 second response time [00:43:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.23, 3.11, 3.27 [00:43:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GroupsSidebar-808fabc [+0/-0/±1] 13https://git.io/JK1s2 [00:43:52] [02miraheze/mediawiki] 07dependabot[bot] 037a65845 - Bump extensions/GroupsSidebar from `4c5b5bb` to `808fabc` [00:43:53] [02mediawiki] 07dependabot[bot] opened pull request 03#3763: Bump extensions/GroupsSidebar from `4c5b5bb` to `808fabc` - 13https://git.io/JK1sa [00:43:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GroupsSidebar-808fabc - 13https://git.io/vbL5b [00:43:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#3763: Bump extensions/GroupsSidebar from `4c5b5bb` to `808fabc` - 13https://git.io/JK1sa [00:43:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/QuizGame-b267cc6 [+0/-0/±1] 13https://git.io/JK1so [00:43:59] [02miraheze/mediawiki] 07dependabot[bot] 0369a74bb - Bump extensions/QuizGame from `b6971e5` to `b267cc6` [00:44:01] [02mediawiki] 07dependabot[bot] opened pull request 03#3764: Bump extensions/QuizGame from `b6971e5` to `b267cc6` - 13https://git.io/JK1sK [00:44:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/QuizGame-b267cc6 - 13https://git.io/vbL5b [00:44:04] [02mediawiki] 07dependabot[bot] labeled pull request 03#3764: Bump extensions/QuizGame from `b6971e5` to `b267cc6` - 13https://git.io/JK1sK [00:44:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Citizen-e9e7d1c [+0/-0/±1] 13https://git.io/JK1s6 [00:44:07] [02miraheze/mediawiki] 07dependabot[bot] 03f2b0a54 - Bump skins/Citizen from `f03f55b` to `e9e7d1c` [00:44:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Citizen-e9e7d1c - 13https://git.io/vbL5b [00:44:10] [02mediawiki] 07dependabot[bot] opened pull request 03#3765: Bump skins/Citizen from `f03f55b` to `e9e7d1c` - 13https://git.io/JK1si [00:44:11] [02mediawiki] 07dependabot[bot] labeled pull request 03#3765: Bump skins/Citizen from `f03f55b` to `e9e7d1c` - 13https://git.io/JK1si [00:44:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Calendar-8f3e2e5 [+0/-0/±1] 13https://git.io/JK1s1 [00:44:14] [02miraheze/mediawiki] 07dependabot[bot] 03a12d168 - Bump extensions/Calendar from `f6f5218` to `8f3e2e5` [00:44:16] [02mediawiki] 07dependabot[bot] opened pull request 03#3766: Bump extensions/Calendar from `f6f5218` to `8f3e2e5` - 13https://git.io/JK1sM [00:44:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Calendar-8f3e2e5 - 13https://git.io/vbL5b [00:44:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3766: Bump extensions/Calendar from `f6f5218` to `8f3e2e5` - 13https://git.io/JK1sM [00:44:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/NumberedHeadings-85cf6b4 [+0/-0/±1] 13https://git.io/JK1sS [00:44:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:44:22] [02miraheze/mediawiki] 07dependabot[bot] 0387719a0 - Bump extensions/NumberedHeadings from `ada69e7` to `85cf6b4` [00:44:23] [02mediawiki] 07dependabot[bot] opened pull request 03#3767: Bump extensions/NumberedHeadings from `ada69e7` to `85cf6b4` - 13https://git.io/JK1s9 [00:44:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/NumberedHeadings-85cf6b4 - 13https://git.io/vbL5b [00:44:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#3767: Bump extensions/NumberedHeadings from `ada69e7` to `85cf6b4` - 13https://git.io/JK1s9 [00:44:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Lingo-be4611e [+0/-0/±1] 13https://git.io/JK1s5 [00:44:29] [02miraheze/mediawiki] 07dependabot[bot] 037b3f45c - Bump extensions/Lingo from `f4d7133` to `be4611e` [00:44:31] [02mediawiki] 07dependabot[bot] opened pull request 03#3768: Bump extensions/Lingo from `f4d7133` to `be4611e` - 13https://git.io/JK1sd [00:44:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Lingo-be4611e - 13https://git.io/vbL5b [00:44:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#3768: Bump extensions/Lingo from `f4d7133` to `be4611e` - 13https://git.io/JK1sd [00:44:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PollNY-ff9e754 [+0/-0/±1] 13https://git.io/JK1sb [00:44:37] [02miraheze/mediawiki] 07dependabot[bot] 031601162 - Bump extensions/PollNY from `4b30a4f` to `ff9e754` [00:44:38] [02mediawiki] 07dependabot[bot] opened pull request 03#3769: Bump extensions/PollNY from `4b30a4f` to `ff9e754` - 13https://git.io/JK1sN [00:44:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PollNY-ff9e754 - 13https://git.io/vbL5b [00:44:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.62, 6.54, 6.79 [00:44:41] [02mediawiki] 07dependabot[bot] labeled pull request 03#3769: Bump extensions/PollNY from `4b30a4f` to `ff9e754` - 13https://git.io/JK1sN [00:44:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-4bbe94b [+0/-0/±1] 13https://git.io/JK1sh [00:44:44] [02miraheze/mediawiki] 07dependabot[bot] 0318eb352 - Bump skins/Evelution from `55362e4` to `4bbe94b` [00:44:46] [02mediawiki] 07dependabot[bot] opened pull request 03#3770: Bump skins/Evelution from `55362e4` to `4bbe94b` - 13https://git.io/JK1sj [00:44:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-4bbe94b - 13https://git.io/vbL5b [00:44:49] [02mediawiki] 07dependabot[bot] labeled pull request 03#3770: Bump skins/Evelution from `55362e4` to `4bbe94b` - 13https://git.io/JK1sj [00:44:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageSchemas-410b5a6 [+0/-0/±1] 13https://git.io/JK1Gv [00:44:52] [02miraheze/mediawiki] 07dependabot[bot] 03f152ed3 - Bump extensions/PageSchemas from `a698ba2` to `410b5a6` [00:44:53] [02mediawiki] 07dependabot[bot] opened pull request 03#3771: Bump extensions/PageSchemas from `a698ba2` to `410b5a6` - 13https://git.io/JK1Gf [00:44:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-410b5a6 - 13https://git.io/vbL5b [00:44:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#3771: Bump extensions/PageSchemas from `a698ba2` to `410b5a6` - 13https://git.io/JK1Gf [00:44:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SocialProfile-aebbc70 [+0/-0/±1] 13https://git.io/JK1GT [00:44:59] [02miraheze/mediawiki] 07dependabot[bot] 030cc6e4c - Bump extensions/SocialProfile from `41cd719` to `aebbc70` [00:45:01] [02mediawiki] 07dependabot[bot] opened pull request 03#3772: Bump extensions/SocialProfile from `41cd719` to `aebbc70` - 13https://git.io/JK1Gk [00:45:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-aebbc70 - 13https://git.io/vbL5b [00:45:04] [02mediawiki] 07dependabot[bot] labeled pull request 03#3772: Bump extensions/SocialProfile from `41cd719` to `aebbc70` - 13https://git.io/JK1Gk [00:45:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Video-10078a2 [+0/-0/±1] 13https://git.io/JK1Gq [00:45:07] [02miraheze/mediawiki] 07dependabot[bot] 03f473bcb - Bump extensions/Video from `c944e45` to `10078a2` [00:45:08] [02mediawiki] 07dependabot[bot] opened pull request 03#3773: Bump extensions/Video from `c944e45` to `10078a2` - 13https://git.io/JK1Gm [00:45:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Video-10078a2 - 13https://git.io/vbL5b [00:45:11] [02mediawiki] 07dependabot[bot] labeled pull request 03#3773: Bump extensions/Video from `c944e45` to `10078a2` - 13https://git.io/JK1Gm [00:45:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DataDump-971284c [+0/-0/±1] 13https://git.io/JK1GO [00:45:14] [02miraheze/mediawiki] 07dependabot[bot] 032638d22 - Bump extensions/DataDump from `5ab51f3` to `971284c` [00:45:16] [02mediawiki] 07dependabot[bot] opened pull request 03#3774: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JK1G3 [00:45:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DataDump-971284c - 13https://git.io/vbL5b [00:45:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3774: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JK1G3 [00:45:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-e5c14b0 [+0/-0/±1] 13https://git.io/JK1GG [00:45:22] [02miraheze/mediawiki] 07dependabot[bot] 03c0bbc16 - Bump extensions/MirahezeMagic from `37f09ac` to `e5c14b0` [00:45:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-e5c14b0 - 13https://git.io/vbL5b [00:45:25] [02mediawiki] 07dependabot[bot] opened pull request 03#3775: Bump extensions/MirahezeMagic from `37f09ac` to `e5c14b0` - 13https://git.io/JK1Gn [00:45:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#3775: Bump extensions/MirahezeMagic from `37f09ac` to `e5c14b0` - 13https://git.io/JK1Gn [00:45:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CreateWiki-8a4db79 [+0/-0/±1] 13https://git.io/JK1G8 [00:45:29] [02miraheze/mediawiki] 07dependabot[bot] 034d3f0ef - Bump extensions/CreateWiki from `b453f0a` to `8a4db79` [00:45:31] [02mediawiki] 07dependabot[bot] opened pull request 03#3776: Bump extensions/CreateWiki from `b453f0a` to `8a4db79` - 13https://git.io/JK1G4 [00:45:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-8a4db79 - 13https://git.io/vbL5b [00:45:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#3776: Bump extensions/CreateWiki from `b453f0a` to `8a4db79` - 13https://git.io/JK1G4 [00:45:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/cldr-de27f07 [+0/-0/±1] 13https://git.io/JK1GB [00:45:37] [02miraheze/mediawiki] 07dependabot[bot] 033adbd70 - Bump extensions/cldr from `415a585` to `de27f07` [00:45:38] [02mediawiki] 07dependabot[bot] opened pull request 03#3777: Bump extensions/cldr from `415a585` to `de27f07` - 13https://git.io/JK1GR [00:45:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/cldr-de27f07 - 13https://git.io/vbL5b [00:45:41] [02mediawiki] 07dependabot[bot] labeled pull request 03#3777: Bump extensions/cldr from `415a585` to `de27f07` - 13https://git.io/JK1GR [00:45:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Arrays-d266b32 [+0/-0/±1] 13https://git.io/JK1GE [00:45:44] [02miraheze/mediawiki] 07dependabot[bot] 0388d6e22 - Bump extensions/Arrays from `0c390e0` to `d266b32` [00:45:46] [02mediawiki] 07dependabot[bot] opened pull request 03#3778: Bump extensions/Arrays from `0c390e0` to `d266b32` - 13https://git.io/JK1Gu [00:45:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Arrays-d266b32 - 13https://git.io/vbL5b [00:45:49] [02mediawiki] 07dependabot[bot] labeled pull request 03#3778: Bump extensions/Arrays from `0c390e0` to `d266b32` - 13https://git.io/JK1Gu [00:45:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UserPageEditProtection-c838fbd [+0/-0/±1] 13https://git.io/JK1G2 [00:45:52] [02miraheze/mediawiki] 07dependabot[bot] 03b543922 - Bump extensions/UserPageEditProtection from `4422b67` to `c838fbd` [00:45:53] [02mediawiki] 07dependabot[bot] opened pull request 03#3779: Bump extensions/UserPageEditProtection from `4422b67` to `c838fbd` - 13https://git.io/JK1Ga [00:45:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UserPageEditProtection-c838fbd - 13https://git.io/vbL5b [00:45:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#3779: Bump extensions/UserPageEditProtection from `4422b67` to `c838fbd` - 13https://git.io/JK1Ga [00:45:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-6c11e00 [+0/-0/±1] 13https://git.io/JK1G6 [00:45:59] [02miraheze/mediawiki] 07dependabot[bot] 033f554b3 - Bump extensions/PageForms from `a7f59e9` to `6c11e00` [00:46:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-6c11e00 - 13https://git.io/vbL5b [00:46:02] [02mediawiki] 07dependabot[bot] opened pull request 03#3780: Bump extensions/PageForms from `a7f59e9` to `6c11e00` - 13https://git.io/JK1Gi [00:46:04] [02mediawiki] 07dependabot[bot] labeled pull request 03#3780: Bump extensions/PageForms from `a7f59e9` to `6c11e00` - 13https://git.io/JK1Gi [00:46:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MintyDocs-b9d49f6 [+0/-0/±1] 13https://git.io/JK1GX [00:46:07] [02miraheze/mediawiki] 07dependabot[bot] 032f2ee35 - Bump extensions/MintyDocs from `c787fc0` to `b9d49f6` [00:46:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MintyDocs-b9d49f6 - 13https://git.io/vbL5b [00:46:10] [02mediawiki] 07dependabot[bot] opened pull request 03#3781: Bump extensions/MintyDocs from `c787fc0` to `b9d49f6` - 13https://git.io/JK1G1 [00:46:11] [02mediawiki] 07dependabot[bot] labeled pull request 03#3781: Bump extensions/MintyDocs from `c787fc0` to `b9d49f6` - 13https://git.io/JK1G1 [00:46:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-177f242 [+0/-0/±1] 13https://git.io/JK1Gy [00:46:14] [02miraheze/mediawiki] 07dependabot[bot] 032df8187 - Bump extensions/Cargo from `14c497b` to `177f242` [00:46:16] [02mediawiki] 07dependabot[bot] opened pull request 03#3782: Bump extensions/Cargo from `14c497b` to `177f242` - 13https://git.io/JK1GS [00:46:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-177f242 - 13https://git.io/vbL5b [00:46:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3782: Bump extensions/Cargo from `14c497b` to `177f242` - 13https://git.io/JK1GS [00:46:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MobileTabsPlugin-f8e0ccf [+0/-0/±1] 13https://git.io/JK1GQ [00:46:22] [02miraheze/mediawiki] 07dependabot[bot] 03401e25b - Bump extensions/MobileTabsPlugin from `b1aded6` to `f8e0ccf` [00:46:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MobileTabsPlugin-f8e0ccf - 13https://git.io/vbL5b [00:46:25] [02mediawiki] 07dependabot[bot] opened pull request 03#3783: Bump extensions/MobileTabsPlugin from `b1aded6` to `f8e0ccf` - 13https://git.io/JK1G7 [00:46:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#3783: Bump extensions/MobileTabsPlugin from `b1aded6` to `f8e0ccf` - 13https://git.io/JK1G7 [00:46:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Maps-5d39abc [+0/-0/±1] 13https://git.io/JK1Gd [00:46:29] [02miraheze/mediawiki] 07dependabot[bot] 035acb1cf - Bump extensions/Maps from `4db8508` to `5d39abc` [00:46:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [00:46:31] [02mediawiki] 07dependabot[bot] opened pull request 03#3784: Bump extensions/Maps from `4db8508` to `5d39abc` - 13https://git.io/JK1GF [00:46:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Maps-5d39abc - 13https://git.io/vbL5b [00:46:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#3784: Bump extensions/Maps from `4db8508` to `5d39abc` - 13https://git.io/JK1GF [00:46:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-a0a6505 [+0/-0/±1] 13https://git.io/JK1GN [00:46:37] [02miraheze/mediawiki] 07dependabot[bot] 032b11cd5 - Bump extensions/Translate from `ebc6560` to `a0a6505` [00:46:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-a0a6505 - 13https://git.io/vbL5b [00:46:40] [02mediawiki] 07dependabot[bot] opened pull request 03#3785: Bump extensions/Translate from `ebc6560` to `a0a6505` - 13https://git.io/JK1Gx [00:46:41] [02mediawiki] 07dependabot[bot] labeled pull request 03#3785: Bump extensions/Translate from `ebc6560` to `a0a6505` - 13https://git.io/JK1Gx [00:46:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/EmbedVideo-f98a408 [+0/-0/±1] 13https://git.io/JK1Gj [00:46:44] [02miraheze/mediawiki] 07dependabot[bot] 0379140d9 - Bump extensions/EmbedVideo from `76cefce` to `f98a408` [00:46:46] [02mediawiki] 07dependabot[bot] opened pull request 03#3786: Bump extensions/EmbedVideo from `76cefce` to `f98a408` - 13https://git.io/JK1Ze [00:46:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/EmbedVideo-f98a408 - 13https://git.io/vbL5b [00:46:49] [02mediawiki] 07dependabot[bot] labeled pull request 03#3786: Bump extensions/EmbedVideo from `76cefce` to `f98a408` - 13https://git.io/JK1Ze [00:46:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/HAWelcome-0b5c826 [+0/-0/±1] 13https://git.io/JK1ZJ [00:46:52] [02miraheze/mediawiki] 07dependabot[bot] 0393dab71 - Bump extensions/HAWelcome from `fab7f75` to `0b5c826` [00:46:53] [02mediawiki] 07dependabot[bot] opened pull request 03#3787: Bump extensions/HAWelcome from `fab7f75` to `0b5c826` - 13https://git.io/JK1ZU [00:46:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/HAWelcome-0b5c826 - 13https://git.io/vbL5b [00:46:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#3787: Bump extensions/HAWelcome from `fab7f75` to `0b5c826` - 13https://git.io/JK1ZU [00:46:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ManageWiki-45bebc8 [+0/-0/±1] 13https://git.io/JK1Zt [00:46:59] [02miraheze/mediawiki] 07dependabot[bot] 03106bab2 - Bump extensions/ManageWiki from `56026ad` to `45bebc8` [00:47:01] [02mediawiki] 07dependabot[bot] opened pull request 03#3788: Bump extensions/ManageWiki from `56026ad` to `45bebc8` - 13https://git.io/JK1Zm [00:47:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-45bebc8 - 13https://git.io/vbL5b [00:47:04] [02mediawiki] 07dependabot[bot] labeled pull request 03#3788: Bump extensions/ManageWiki from `56026ad` to `45bebc8` - 13https://git.io/JK1Zm [00:47:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ShortDescription-d764a1f [+0/-0/±1] 13https://git.io/JK1Zs [00:47:07] [02miraheze/mediawiki] 07dependabot[bot] 038adab2c - Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` [00:47:08] [02mediawiki] 07dependabot[bot] opened pull request 03#3789: Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` - 13https://git.io/JK1ZG [00:47:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ShortDescription-d764a1f - 13https://git.io/vbL5b [00:47:11] [02mediawiki] 07dependabot[bot] labeled pull request 03#3789: Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` - 13https://git.io/JK1ZG [00:47:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Moderation-2b0bd8f [+0/-0/±1] 13https://git.io/JK1Zc [00:47:14] [02miraheze/mediawiki] 07dependabot[bot] 0301baee4 - Bump extensions/Moderation from `cbf109b` to `2b0bd8f` [00:47:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Moderation-2b0bd8f - 13https://git.io/vbL5b [00:47:17] [02mediawiki] 07dependabot[bot] opened pull request 03#3790: Bump extensions/Moderation from `cbf109b` to `2b0bd8f` - 13https://git.io/JK1ZC [00:47:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#3790: Bump extensions/Moderation from `cbf109b` to `2b0bd8f` - 13https://git.io/JK1ZC [00:47:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-23396f2 [+0/-0/±1] 13https://git.io/JK1Z8 [00:47:22] [02miraheze/mediawiki] 07dependabot[bot] 03e94d9ba - Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` [00:47:23] [02mediawiki] 07dependabot[bot] opened pull request 03#3791: Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` - 13https://git.io/JK1Z4 [00:47:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-23396f2 - 13https://git.io/vbL5b [00:47:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#3791: Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` - 13https://git.io/JK1Z4 [00:47:34] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 3.49, 6.36, 7.61 [00:55:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.57, 3.41 [00:56:41] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.32, 4.88, 6.36 [00:58:26] [02mediawiki] 07Universal-Omega opened pull request 03#3792: Remove AdvancedSearch and MediaModeration - 13https://git.io/JK1lF [00:58:53] [02mediawiki] 07Universal-Omega opened pull request 03#3793: Remove WebP - 13https://git.io/JK18m [00:59:16] [02mediawiki] 07Universal-Omega opened pull request 03#3794: Remove AdvancedSearch and MediaModeration - 13https://git.io/JK18R [00:59:28] [02mediawiki] 07Universal-Omega opened pull request 03#3795: Remove WebP - 13https://git.io/JK18V [01:00:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JK14v [01:00:19] [02miraheze/mediawiki] 07Universal-Omega 036ca8f82 - Update non-rel-extensions [01:00:20] [02mediawiki] 07Universal-Omega synchronize pull request 03#3793: Remove WebP - 13https://git.io/JK18m [01:00:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JK148 [01:00:53] [02miraheze/mediawiki] 07Universal-Omega 0302289cc - Update non-rel-extensions [01:00:55] [02mediawiki] 07Universal-Omega synchronize pull request 03#3795: Remove WebP - 13https://git.io/JK18V [01:01:08] [02mediawiki] 07Universal-Omega closed pull request 03#3793: Remove WebP - 13https://git.io/JK18m [01:01:10] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [01:01:11] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-1/±2] 13https://git.io/JK14V [01:01:13] [02miraheze/mediawiki] 07Universal-Omega 03dc1390e - Remove WebP (#3793) [01:01:14] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbL5b [01:01:26] [02mediawiki] 07Universal-Omega closed pull request 03#3792: Remove AdvancedSearch and MediaModeration - 13https://git.io/JK1lF [01:01:27] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:01:29] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-2/±1] 13https://git.io/JK14S [01:01:30] [02miraheze/mediawiki] 07Universal-Omega 033c63184 - Remove AdvancedSearch and MediaModeration (#3792) [01:01:32] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [01:02:18] [02mediawiki] 07Universal-Omega closed pull request 03#3794: Remove AdvancedSearch and MediaModeration - 13https://git.io/JK18R [01:02:20] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:02:21] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-2/±1] 13https://git.io/JK1BO [01:02:23] [02miraheze/mediawiki] 07Universal-Omega 0376254e3 - Remove AdvancedSearch and MediaModeration (#3794) [01:02:24] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [01:02:31] [02mediawiki] 07Universal-Omega closed pull request 03#3795: Remove WebP - 13https://git.io/JK18V [01:02:32] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-1/±2] 13https://git.io/JK1Bn [01:02:34] [02miraheze/mediawiki] 07Universal-Omega 03bccc844 - Remove WebP (#3795) [01:02:35] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbL5b [01:02:37] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [01:03:50] [02puppet] 07Universal-Omega opened pull request 03#2053: remove TransformWebPImageJob from jobQueueCollector - 13https://git.io/JK1RY [01:04:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 3.17, 3.38 [01:05:58] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to skip [01:06:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.79, 8.94, 7.25 [01:09:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.09, 8.80, 7.10 [01:10:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.86, 7.04, 6.79 [01:11:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.60, 3.24, 3.41 [01:12:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.97, 7.83, 7.03 [01:13:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.28, 7.06, 6.88 [01:13:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.86, 6.39, 6.61 [01:14:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.72, 3.74, 3.52 [01:16:59] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all [01:17:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:17:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.75, 20.12, 23.62 [01:18:53] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to skip - SUCCESS in 774s [01:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.08, 6.91, 6.06 [01:21:19] [02puppet] 07Universal-Omega edited pull request 03#2052: prometheus-es-exporter: add mediawiki_params_action aggs - 13https://git.io/JKij2 [01:21:21] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.19, 4.87, 5.95 [01:21:58] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.32, 8.02, 7.33 [01:22:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.71, 6.25, 5.94 [01:22:32] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.69, 6.90, 7.31 [01:23:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.93, 22.06, 23.10 [01:24:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.63, 3.63 [01:25:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.62, 7.72, 7.34 [01:26:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.08, 20.18, 22.24 [01:27:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.62, 7.98, 7.12 [01:27:53] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.82, 22.77, 20.55 [01:28:00] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.66, 8.74, 7.76 [01:29:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.34, 6.99, 6.47 [01:30:38] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.66, 7.58, 7.13 [01:30:53] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.01, 22.40, 20.81 [01:31:53] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.82, 5.66, 6.60 [01:32:50] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.44, 6.26, 6.28 [01:33:37] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.05, 5.37, 5.66 [01:34:08] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.46, 23.78, 21.61 [01:36:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.40, 5.05, 5.49 [01:37:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [01:40:28] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.43, 23.42, 22.38 [01:40:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.36, 7.98, 7.46 [01:42:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [01:42:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.76, 2.83, 3.23 [01:43:06] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.13, 5.74, 5.61 [01:43:12] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.93, 6.05, 6.58 [01:45:29] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.26, 25.58, 22.70 [01:46:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.37, 6.84, 7.99 [01:46:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.69, 5.77, 6.61 [01:49:43] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.87, 21.00, 21.23 [01:50:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.01, 3.92, 3.60 [01:51:15] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 2056s [01:51:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:51:58] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.43, 22.78, 22.42 [01:52:53] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.08, 23.66, 22.53 [01:53:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.55, 3.53 [01:58:12] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.99, 25.55, 23.55 [01:58:40] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.17, 4.48, 6.18 [01:59:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.05, 3.29 [02:01:58] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.27, 17.75, 20.12 [02:10:09] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.63, 22.47, 23.41 [02:21:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.30, 17.31, 20.02 [02:26:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMJz [02:26:14] [02miraheze/mw-config] 07Universal-Omega 03fd31444 - Add wgTabberNeueEnableMD5Hash [02:27:13] miraheze/mw-config - Universal-Omega the build passed. [02:29:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMUP [02:29:58] [02miraheze/mw-config] 07Universal-Omega 030a4c9c0 - Add wgTabberNeueEnableMD5Hash [02:31:02] miraheze/mw-config - Universal-Omega the build passed. [02:33:08] !log [@test3] starting deploy of {'config': True} to skip [02:33:09] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:33:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMTb [02:33:11] [02miraheze/mw-config] 07Universal-Omega 0371f4044 - Add wgScratchBlocks4Langs [02:33:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.26, 5.47, 5.89 [02:34:13] miraheze/mw-config - Universal-Omega the build passed. [02:35:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMkd [02:35:44] [02miraheze/mw-config] 07Universal-Omega 0343ee4cd - Remove wgScratchBlocks4Langs [02:36:10] !log [@mw11] starting deploy of {'config': True} to all [02:36:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:28] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 17s [02:36:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:45] miraheze/mw-config - Universal-Omega the build passed. [02:42:39] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.76, 7.07, 6.31 [02:50:06] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JKMYH [02:50:08] [02miraheze/DataDump] 07R4356th 03839c78f - Merge pull request from GHSA-9vf9-ff5x-vcpc [02:51:58] miraheze/DataDump - Universal-Omega the build has errored. [02:55:44] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMZJ [02:55:45] [02miraheze/DataDump] 07Universal-Omega 03249d4d8 - Fix [02:55:59] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMZZ [02:56:00] [02miraheze/DataDump] 07Universal-Omega 035096372 - Format [02:57:14] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMnt [02:57:15] [02miraheze/DataDump] 07Universal-Omega 038a8a6ce - Fix [02:57:35] miraheze/DataDump - Universal-Omega the build has errored. [02:57:53] miraheze/DataDump - Universal-Omega the build has errored. [02:58:48] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMnh [02:58:50] [02miraheze/DataDump] 07Universal-Omega 039c6e5fd - Format [02:58:58] miraheze/DataDump - Universal-Omega the build has errored. [02:59:52] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.16, 5.15, 5.71 [03:00:38] miraheze/DataDump - Universal-Omega the build has errored. [03:02:14] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMWt [03:02:15] [02miraheze/DataDump] 07Universal-Omega 03a97b5d2 - Fix [03:02:45] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.98, 6.03, 5.95 [03:03:24] [02miraheze/DataDump] 07github-actions[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKMWX [03:03:26] [02miraheze/DataDump] 07github-actions 03cca35eb - CI: lint code to MediaWiki standards [03:04:07] miraheze/DataDump - Universal-Omega the build passed. [03:05:41] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.88, 5.39, 5.71 [03:06:06] [02MirahezeMagic] 07Universal-Omega closed pull request 03#295: en.json: fix capitalisation and use sre-mediawiki email - 13https://git.io/JKzLe [03:06:07] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKM8L [03:06:09] [02miraheze/MirahezeMagic] 07Universal-Omega 031928ebf - en.json: fix capitalisation and use sre-mediawiki email (#295) [03:06:10] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [03:06:12] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [03:07:02] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:22:46] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.33, 4.71, 4.98 [04:25:38] A lot of revisions of TestWiki's Main Page return an internal error: https://publictestwiki.com/w/index.php?title=Main_Page&oldid=15 [04:25:40] [url] Internal error - TestWiki | publictestwiki.com [04:25:54] [02puppet] 07Universal-Omega opened pull request 03#2054: MediaWiki: remove libwebp-dev - 13https://git.io/JKDvx [04:29:53] Agent: same issue as https://phabricator.miraheze.org/T8184 I believe [04:29:54] [url] ⚓ T8184 Editing and contributions broken on FAMEPedia | phabricator.miraheze.org [04:31:17] this seems widespread then unfortunately [04:31:24] compression gone wrong but why [04:31:50] Agent: according to logs it is on at least a few other wikis also, such as privacywiki. [04:33:57] no progress upstream yet, right? [04:34:43] Agent: Nope. [05:00:12] [02mediawiki] 07Universal-Omega closed pull request 03#3208: Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` - 13https://git.io/JgQjH [05:00:14] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/TabberNeue-3358e2e - 13https://git.io/vbL5b [05:00:15] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/TabberNeue-3358e2e [05:00:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JKDnG [05:00:18] [02miraheze/mediawiki] 07dependabot[bot] 03774dd47 - Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` (#3208) [05:00:59] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all [05:01:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:43] [02mediawiki] 07Universal-Omega closed pull request 03#3791: Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` - 13https://git.io/JK1Z4 [05:04:45] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-23396f2 [05:04:46] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-23396f2 - 13https://git.io/vbL5b [05:04:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDCM [05:04:49] [02miraheze/mediawiki] 07dependabot[bot] 03257d119 - Bump extensions/UniversalLanguageSelector from `b5ec74d` to `23396f2` (#3791) [05:04:53] [02mediawiki] 07Universal-Omega closed pull request 03#3790: Bump extensions/Moderation from `cbf109b` to `2b0bd8f` - 13https://git.io/JK1ZC [05:04:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDC9 [05:04:56] [02miraheze/mediawiki] 07dependabot[bot] 03e533569 - Bump extensions/Moderation from `cbf109b` to `2b0bd8f` (#3790) [05:04:57] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Moderation-2b0bd8f - 13https://git.io/vbL5b [05:04:59] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Moderation-2b0bd8f [05:05:08] [02mediawiki] 07Universal-Omega closed pull request 03#3789: Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` - 13https://git.io/JK1ZG [05:05:10] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDCx [05:05:11] [02miraheze/mediawiki] 07dependabot[bot] 031558b5c - Bump extensions/ShortDescription from `2f3e5f0` to `d764a1f` (#3789) [05:05:13] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ShortDescription-d764a1f - 13https://git.io/vbL5b [05:05:14] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ShortDescription-d764a1f [05:05:19] [02mediawiki] 07Universal-Omega closed pull request 03#3788: Bump extensions/ManageWiki from `56026ad` to `45bebc8` - 13https://git.io/JK1Zm [05:05:20] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDWJ [05:05:22] [02miraheze/mediawiki] 07dependabot[bot] 03d3d6715 - Bump extensions/ManageWiki from `56026ad` to `45bebc8` (#3788) [05:05:23] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-45bebc8 - 13https://git.io/vbL5b [05:05:25] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-45bebc8 [05:05:31] [02mediawiki] 07Universal-Omega closed pull request 03#3787: Bump extensions/HAWelcome from `fab7f75` to `0b5c826` - 13https://git.io/JK1ZU [05:05:32] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDWq [05:05:34] [02miraheze/mediawiki] 07dependabot[bot] 032909de1 - Bump extensions/HAWelcome from `fab7f75` to `0b5c826` (#3787) [05:05:35] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/HAWelcome-0b5c826 - 13https://git.io/vbL5b [05:05:37] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/HAWelcome-0b5c826 [05:05:44] [02mediawiki] 07Universal-Omega closed pull request 03#3786: Bump extensions/EmbedVideo from `76cefce` to `f98a408` - 13https://git.io/JK1Ze [05:05:45] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDWs [05:05:47] [02miraheze/mediawiki] 07dependabot[bot] 030c01673 - Bump extensions/EmbedVideo from `76cefce` to `f98a408` (#3786) [05:05:48] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/EmbedVideo-f98a408 - 13https://git.io/vbL5b [05:05:50] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/EmbedVideo-f98a408 [05:05:54] [02mediawiki] 07Universal-Omega closed pull request 03#3785: Bump extensions/Translate from `ebc6560` to `a0a6505` - 13https://git.io/JK1Gx [05:05:56] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-a0a6505 [05:05:57] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDW8 [05:05:59] [02miraheze/mediawiki] 07dependabot[bot] 030998d3e - Bump extensions/Translate from `ebc6560` to `a0a6505` (#3785) [05:06:00] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-a0a6505 - 13https://git.io/vbL5b [05:06:09] [02mediawiki] 07Universal-Omega closed pull request 03#3784: Bump extensions/Maps from `4db8508` to `5d39abc` - 13https://git.io/JK1GF [05:06:11] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Maps-5d39abc - 13https://git.io/vbL5b [05:06:12] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Maps-5d39abc [05:06:14] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDWg [05:06:15] [02miraheze/mediawiki] 07dependabot[bot] 03f1ddea4 - Bump extensions/Maps from `4db8508` to `5d39abc` (#3784) [05:06:29] [02mediawiki] 07Universal-Omega closed pull request 03#3783: Bump extensions/MobileTabsPlugin from `b1aded6` to `f8e0ccf` - 13https://git.io/JK1G7 [05:06:31] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MobileTabsPlugin-f8e0ccf - 13https://git.io/vbL5b [05:06:32] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDW1 [05:06:34] [02miraheze/mediawiki] 07dependabot[bot] 0324b6b4a - Bump extensions/MobileTabsPlugin from `b1aded6` to `f8e0ccf` (#3783) [05:06:35] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MobileTabsPlugin-f8e0ccf [05:06:49] [02mediawiki] 07Universal-Omega closed pull request 03#3782: Bump extensions/Cargo from `14c497b` to `177f242` - 13https://git.io/JK1GS [05:06:50] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-177f242 - 13https://git.io/vbL5b [05:06:52] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-177f242 [05:06:53] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDWA [05:06:55] [02miraheze/mediawiki] 07dependabot[bot] 03741cc5a - Bump extensions/Cargo from `14c497b` to `177f242` (#3782) [05:06:58] [02mediawiki] 07Universal-Omega closed pull request 03#3781: Bump extensions/MintyDocs from `c787fc0` to `b9d49f6` - 13https://git.io/JK1G1 [05:06:59] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MintyDocs-b9d49f6 [05:07:01] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MintyDocs-b9d49f6 - 13https://git.io/vbL5b [05:07:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDlv [05:07:04] [02miraheze/mediawiki] 07dependabot[bot] 03d26de65 - Bump extensions/MintyDocs from `c787fc0` to `b9d49f6` (#3781) [05:07:09] [02mediawiki] 07Universal-Omega closed pull request 03#3780: Bump extensions/PageForms from `a7f59e9` to `6c11e00` - 13https://git.io/JK1Gi [05:07:11] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-6c11e00 - 13https://git.io/vbL5b [05:07:12] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-6c11e00 [05:07:14] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDlI [05:07:15] [02miraheze/mediawiki] 07dependabot[bot] 03849d7cb - Bump extensions/PageForms from `a7f59e9` to `6c11e00` (#3780) [05:07:22] [02mediawiki] 07Universal-Omega closed pull request 03#3779: Bump extensions/UserPageEditProtection from `4422b67` to `c838fbd` - 13https://git.io/JK1Ga [05:07:24] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UserPageEditProtection-c838fbd - 13https://git.io/vbL5b [05:07:25] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UserPageEditProtection-c838fbd [05:07:27] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDlC [05:07:28] [02miraheze/mediawiki] 07dependabot[bot] 03a39b345 - Bump extensions/UserPageEditProtection from `4422b67` to `c838fbd` (#3779) [05:07:31] [02mediawiki] 07Universal-Omega closed pull request 03#3772: Bump extensions/SocialProfile from `41cd719` to `aebbc70` - 13https://git.io/JK1Gk [05:07:32] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-aebbc70 - 13https://git.io/vbL5b [05:07:34] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-aebbc70 [05:07:35] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDl4 [05:07:37] [02miraheze/mediawiki] 07dependabot[bot] 03746945d - Bump extensions/SocialProfile from `41cd719` to `aebbc70` (#3772) [05:07:50] [02mediawiki] 07Universal-Omega commented on pull request 03#3774: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JKDlK [05:07:52] [02mediawiki] 07dependabot[bot] edited pull request 03#3774: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JK1G3 [05:08:00] [02mediawiki] 07Universal-Omega closed pull request 03#3778: Bump extensions/Arrays from `0c390e0` to `d266b32` - 13https://git.io/JK1Gu [05:08:01] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Arrays-d266b32 - 13https://git.io/vbL5b [05:08:02] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Arrays-d266b32 [05:08:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDly [05:08:06] [02miraheze/mediawiki] 07dependabot[bot] 03d1cca4b - Bump extensions/Arrays from `0c390e0` to `d266b32` (#3778) [05:08:12] [02mediawiki] 07Universal-Omega closed pull request 03#3777: Bump extensions/cldr from `415a585` to `de27f07` - 13https://git.io/JK1GR [05:08:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDlN [05:08:15] [02miraheze/mediawiki] 07dependabot[bot] 03f5d8195 - Bump extensions/cldr from `415a585` to `de27f07` (#3777) [05:08:16] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/cldr-de27f07 - 13https://git.io/vbL5b [05:08:18] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/cldr-de27f07 [05:08:31] [02mediawiki] 07Universal-Omega closed pull request 03#3776: Bump extensions/CreateWiki from `b453f0a` to `8a4db79` - 13https://git.io/JK1G4 [05:08:33] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-8a4db79 - 13https://git.io/vbL5b [05:08:34] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD8t [05:08:36] [02miraheze/mediawiki] 07dependabot[bot] 0309ae401 - Bump extensions/CreateWiki from `b453f0a` to `8a4db79` (#3776) [05:08:37] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-8a4db79 [05:08:51] [02mediawiki] 07Universal-Omega commented on pull request 03#3775: Bump extensions/MirahezeMagic from `37f09ac` to `e5c14b0` - 13https://git.io/JKD8l [05:08:53] [02mediawiki] 07dependabot[bot] edited pull request 03#3775: Bump extensions/MirahezeMagic from `37f09ac` to `e5c14b0` - 13https://git.io/JK1Gn [05:08:58] [02mediawiki] 07dependabot[bot] edited pull request 03#3774: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JK1G3 [05:09:02] [02mediawiki] 07Universal-Omega closed pull request 03#3773: Bump extensions/Video from `c944e45` to `10078a2` - 13https://git.io/JK1Gm [05:09:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD8u [05:09:05] [02miraheze/mediawiki] 07dependabot[bot] 039535805 - Bump extensions/Video from `c944e45` to `10078a2` (#3773) [05:09:06] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Video-10078a2 - 13https://git.io/vbL5b [05:09:08] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Video-10078a2 [05:09:09] [02mediawiki] 07dependabot[bot] opened pull request 03#3796: Bump extensions/DataDump from `5ab51f3` to `cca35eb` - 13https://git.io/JKD82 [05:09:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DataDump-cca35eb [+0/-0/±1] 13https://git.io/JKD8a [05:09:12] [02miraheze/mediawiki] 07dependabot[bot] 03e1c6989 - Bump extensions/DataDump from `5ab51f3` to `cca35eb` [05:09:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DataDump-cca35eb - 13https://git.io/vbL5b [05:09:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#3796: Bump extensions/DataDump from `5ab51f3` to `cca35eb` - 13https://git.io/JKD82 [05:09:17] [02mediawiki] 07dependabot[bot] commented on pull request 03#3774: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JKD8o [05:09:19] [02mediawiki] 07dependabot[bot] closed pull request 03#3774: Bump extensions/DataDump from `5ab51f3` to `971284c` - 13https://git.io/JK1G3 [05:09:20] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/DataDump-971284c [05:09:22] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/DataDump-971284c - 13https://git.io/vbL5b [05:09:23] [02mediawiki] 07Universal-Omega closed pull request 03#3763: Bump extensions/GroupsSidebar from `4c5b5bb` to `808fabc` - 13https://git.io/JK1sa [05:09:25] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD8P [05:09:26] [02miraheze/mediawiki] 07dependabot[bot] 03831d647 - Bump extensions/GroupsSidebar from `4c5b5bb` to `808fabc` (#3763) [05:09:28] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GroupsSidebar-808fabc [05:09:29] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GroupsSidebar-808fabc - 13https://git.io/vbL5b [05:09:31] [02mediawiki] 07Universal-Omega closed pull request 03#3764: Bump extensions/QuizGame from `b6971e5` to `b267cc6` - 13https://git.io/JK1sK [05:09:32] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/QuizGame-b267cc6 [05:09:34] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/QuizGame-b267cc6 - 13https://git.io/vbL5b [05:09:35] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD89 [05:09:37] [02miraheze/mediawiki] 07dependabot[bot] 031d4a28c - Bump extensions/QuizGame from `b6971e5` to `b267cc6` (#3764) [05:09:46] [02mediawiki] 07Universal-Omega closed pull request 03#3765: Bump skins/Citizen from `f03f55b` to `e9e7d1c` - 13https://git.io/JK1si [05:09:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD4v [05:09:49] [02miraheze/mediawiki] 07dependabot[bot] 0386a6278 - Bump skins/Citizen from `f03f55b` to `e9e7d1c` (#3765) [05:09:50] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Citizen-e9e7d1c - 13https://git.io/vbL5b [05:09:52] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Citizen-e9e7d1c [05:09:53] [02mediawiki] 07dependabot[bot] edited pull request 03#3775: Bump extensions/MirahezeMagic from `37f09ac` to `e5c14b0` - 13https://git.io/JK1Gn [05:09:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-1928ebf [+0/-0/±1] 13https://git.io/JKD4T [05:09:56] [02miraheze/mediawiki] 07dependabot[bot] 039976ef8 - Bump extensions/MirahezeMagic from `37f09ac` to `1928ebf` [05:09:58] [02mediawiki] 07dependabot[bot] opened pull request 03#3797: Bump extensions/MirahezeMagic from `37f09ac` to `1928ebf` - 13https://git.io/JKD4k [05:09:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-1928ebf - 13https://git.io/vbL5b [05:10:01] [02mediawiki] 07dependabot[bot] labeled pull request 03#3797: Bump extensions/MirahezeMagic from `37f09ac` to `1928ebf` - 13https://git.io/JKD4k [05:10:02] [02mediawiki] 07dependabot[bot] commented on pull request 03#3775: Bump extensions/MirahezeMagic from `37f09ac` to `e5c14b0` - 13https://git.io/JKD4L [05:10:04] [02mediawiki] 07dependabot[bot] closed pull request 03#3775: Bump extensions/MirahezeMagic from `37f09ac` to `e5c14b0` - 13https://git.io/JK1Gn [05:10:05] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-e5c14b0 [05:10:07] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-e5c14b0 - 13https://git.io/vbL5b [05:10:08] [02mediawiki] 07Universal-Omega closed pull request 03#3766: Bump extensions/Calendar from `f6f5218` to `8f3e2e5` - 13https://git.io/JK1sM [05:10:10] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Calendar-8f3e2e5 - 13https://git.io/vbL5b [05:10:11] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Calendar-8f3e2e5 [05:10:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD4m [05:10:14] [02miraheze/mediawiki] 07dependabot[bot] 030136c8f - Bump extensions/Calendar from `f6f5218` to `8f3e2e5` (#3766) [05:10:16] [02mediawiki] 07Universal-Omega closed pull request 03#3767: Bump extensions/NumberedHeadings from `ada69e7` to `85cf6b4` - 13https://git.io/JK1s9 [05:10:17] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/NumberedHeadings-85cf6b4 - 13https://git.io/vbL5b [05:10:19] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/NumberedHeadings-85cf6b4 [05:10:21] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD4n [05:10:22] [02miraheze/mediawiki] 07dependabot[bot] 038efb433 - Bump extensions/NumberedHeadings from `ada69e7` to `85cf6b4` (#3767) [05:10:25] [02mediawiki] 07Universal-Omega closed pull request 03#3768: Bump extensions/Lingo from `f4d7133` to `be4611e` - 13https://git.io/JK1sd [05:10:27] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD40 [05:10:28] [02miraheze/mediawiki] 07dependabot[bot] 036fcfe53 - Bump extensions/Lingo from `f4d7133` to `be4611e` (#3768) [05:10:30] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Lingo-be4611e - 13https://git.io/vbL5b [05:10:31] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Lingo-be4611e [05:10:45] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 586s [05:10:45] [02mediawiki] 07Universal-Omega closed pull request 03#3769: Bump extensions/PollNY from `4b30a4f` to `ff9e754` - 13https://git.io/JK1sN [05:10:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD4r [05:10:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:48] [02miraheze/mediawiki] 07dependabot[bot] 03b0e6728 - Bump extensions/PollNY from `4b30a4f` to `ff9e754` (#3769) [05:10:50] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PollNY-ff9e754 - 13https://git.io/vbL5b [05:10:51] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PollNY-ff9e754 [05:10:57] [02mediawiki] 07Universal-Omega closed pull request 03#3770: Bump skins/Evelution from `55362e4` to `4bbe94b` - 13https://git.io/JK1sj [05:10:59] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD4M [05:11:00] [02miraheze/mediawiki] 07dependabot[bot] 0331da5e9 - Bump skins/Evelution from `55362e4` to `4bbe94b` (#3770) [05:11:02] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-4bbe94b [05:11:03] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-4bbe94b - 13https://git.io/vbL5b [05:11:09] [02mediawiki] 07Universal-Omega closed pull request 03#3771: Bump extensions/PageSchemas from `a698ba2` to `410b5a6` - 13https://git.io/JK1Gf [05:11:10] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-410b5a6 - 13https://git.io/vbL5b [05:11:12] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD4Q [05:11:13] [02miraheze/mediawiki] 07dependabot[bot] 039a78ec9 - Bump extensions/PageSchemas from `a698ba2` to `410b5a6` (#3771) [05:11:15] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-410b5a6 [05:11:18] [02mediawiki] 07Universal-Omega closed pull request 03#3797: Bump extensions/MirahezeMagic from `37f09ac` to `1928ebf` - 13https://git.io/JKD4k [05:11:20] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKD4N [05:11:21] [02miraheze/mediawiki] 07dependabot[bot] 03cc5a528 - Bump extensions/MirahezeMagic from `37f09ac` to `1928ebf` (#3797) [05:11:23] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-1928ebf [05:11:24] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-1928ebf - 13https://git.io/vbL5b [05:11:37] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DataDump-cca35eb [05:11:38] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKDBL [05:11:40] [02miraheze/mediawiki] 07dependabot[bot] 032c1658a - Bump extensions/DataDump from `5ab51f3` to `cca35eb` (#3796) [05:11:41] [02mediawiki] 07Universal-Omega closed pull request 03#3796: Bump extensions/DataDump from `5ab51f3` to `cca35eb` - 13https://git.io/JKD82 [05:11:43] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DataDump-cca35eb - 13https://git.io/vbL5b [05:11:49] [02mediawiki] 07Universal-Omega closed pull request 03#3759: Bump skins/Evelution from `55362e4` to `4bbe94b` - 13https://git.io/JK1ve [05:11:51] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JKDBZ [05:11:52] [02miraheze/mediawiki] 07dependabot[bot] 03e49406b - Bump skins/Evelution from `55362e4` to `4bbe94b` (#3759) [05:11:54] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/skins/Evelution-4bbe94b - 13https://git.io/vbL5b [05:11:55] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/skins/Evelution-4bbe94b [05:12:06] [02mediawiki] 07Universal-Omega closed pull request 03#3760: Bump extensions/CreateWiki from `ffffd57` to `8a4db79` - 13https://git.io/JK1vT [05:12:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JKDBl [05:12:09] [02miraheze/mediawiki] 07dependabot[bot] 03e893f25 - Bump extensions/CreateWiki from `ffffd57` to `8a4db79` (#3760) [05:12:11] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-8a4db79 [05:12:12] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-8a4db79 - 13https://git.io/vbL5b [05:12:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03revert-3760-dependabot/submodules/REL1_36/extensions/CreateWiki-8a4db79 [+0/-0/±1] 13https://git.io/JKDBu [05:12:18] [02miraheze/mediawiki] 07Universal-Omega 030312079 - Revert "Bump extensions/CreateWiki from `ffffd57` to `8a4db79` (#3760)" [05:12:20] [02mediawiki] 07Universal-Omega created branch 03revert-3760-dependabot/submodules/REL1_36/extensions/CreateWiki-8a4db79 - 13https://git.io/vbL5b [05:12:21] [02mediawiki] 07Universal-Omega opened pull request 03#3798: Revert "Bump extensions/CreateWiki from `ffffd57` to `8a4db79`" - 13https://git.io/JKDB2 [05:13:00] [02mediawiki] 07Universal-Omega closed pull request 03#3798: Revert "Bump extensions/CreateWiki from `ffffd57` to `8a4db79`" - 13https://git.io/JKDB2 [05:13:03] [02mediawiki] 07Universal-Omega deleted branch 03revert-3760-dependabot/submodules/REL1_36/extensions/CreateWiki-8a4db79 - 13https://git.io/vbL5b [05:13:05] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03revert-3760-dependabot/submodules/REL1_36/extensions/CreateWiki-8a4db79 [05:13:13] [02mediawiki] 07Universal-Omega closed pull request 03#3762: Bump extensions/Maps from `b43865b` to `5d39abc` - 13https://git.io/JK1vc [05:13:14] [02mediawiki] 07dependabot[bot] commented on pull request 03#3762: Bump extensions/Maps from `b43865b` to `5d39abc` - 13https://git.io/JKDRI [05:13:16] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/Maps-5d39abc - 13https://git.io/vbL5b [05:13:17] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/Maps-5d39abc [05:13:42] [02mediawiki] 07Universal-Omega closed pull request 03#3736: Bump extensions/SpriteSheet from `d62d68a` to `d62639c` - 13https://git.io/JKXhM [05:13:43] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JKDR8 [05:13:45] [02miraheze/mediawiki] 07dependabot[bot] 03398eeba - Bump extensions/SpriteSheet from `d62d68a` to `d62639c` (#3736) [05:13:46] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/SpriteSheet-d62639c [05:13:48] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/SpriteSheet-d62639c - 13https://git.io/vbL5b [05:19:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKDuw [05:19:25] [02miraheze/mw-config] 07Universal-Omega 03e1f0042 - Remove mtimeFile [05:20:29] miraheze/mw-config - Universal-Omega the build passed. [05:33:11] !log [@test3] starting deploy of {'config': True} to skip [05:33:12] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [05:33:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:27] [02puppet] 07Reception123 closed pull request 03#2053: remove TransformWebPImageJob from jobQueueCollector - 13https://git.io/JK1RY [05:34:28] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKDoK [05:34:30] [02miraheze/puppet] 07Universal-Omega 03cc4c06f - remove TransformWebPImageJob from jobQueueCollector (#2053) [05:34:34] [02puppet] 07Reception123 closed pull request 03#2054: MediaWiki: remove libwebp-dev - 13https://git.io/JKDvx [05:34:35] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKDo1 [05:34:37] [02miraheze/puppet] 07Universal-Omega 0316bdaeb - MediaWiki: remove libwebp-dev (#2054) [05:35:44] !log [@mw11] starting deploy of {'config': True} to all [05:35:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:35:59] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 14s [05:36:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:15] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03revert-3208-dependabot/submodules/REL1_36/extensions/TabberNeue-3358e2e [+0/-0/±1] 13https://git.io/JKDi5 [05:40:17] [02miraheze/mediawiki] 07Universal-Omega 03629564f - Revert "Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` (#3208)" [05:40:18] [02mediawiki] 07Universal-Omega created branch 03revert-3208-dependabot/submodules/REL1_36/extensions/TabberNeue-3358e2e - 13https://git.io/vbL5b [05:40:20] [02mediawiki] 07Universal-Omega opened pull request 03#3799: Revert "Bump extensions/TabberNeue from `0dc1b34` to `3358e2e`" - 13https://git.io/JKDiF [05:40:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JKDih [05:40:29] [02miraheze/mediawiki] 07Universal-Omega 03a928a09 - Revert "Bump extensions/TabberNeue from `0dc1b34` to `3358e2e` (#3208)" (#3799) [05:40:31] [02mediawiki] 07Universal-Omega deleted branch 03revert-3208-dependabot/submodules/REL1_36/extensions/TabberNeue-3358e2e - 13https://git.io/vbL5b [05:40:32] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03revert-3208-dependabot/submodules/REL1_36/extensions/TabberNeue-3358e2e [05:40:34] [02mediawiki] 07Universal-Omega closed pull request 03#3799: Revert "Bump extensions/TabberNeue from `0dc1b34` to `3358e2e`" - 13https://git.io/JKDiF [05:40:50] !log [universalomega@mw11] starting deploy of {'world': True, 'gitinfo': True} to all [05:40:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:45:48] !log [universalomega@mw11] finished deploy of {'world': True, 'gitinfo': True} to all - SUCCESS in 297s [05:45:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:22] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+0/-1/±10] 13https://git.io/JKDSd [05:54:24] [02miraheze/ssl] 07Reception123 034a910a1 - renew SSL certs [05:54:25] [02miraheze/ssl] 07Reception123 038cc145f - remove wiki.redeemer.live cert (not pointing) [06:08:23] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:09:14] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:09:40] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:09:54] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:09:59] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.92, 7.07, 4.70 [06:10:42] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:10:52] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:11:37] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:12:26] PROBLEM - cp15 Puppet on cp15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:13:17] PROBLEM - cp14 Puppet on cp14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:16:21] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:18:18] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.18, 6.10, 5.32 [06:27:40] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:33:29] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:33:38] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:34:45] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:34:52] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:59] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:35:49] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:36:36] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:37:02] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:38:52] RECOVERY - cp14 Puppet on cp14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:40:32] RECOVERY - cp15 Puppet on cp15 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:40:51] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:55:13] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:58:34] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:37:30] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Sun 16 Jan 2022 04:40:54 GMT +0000. [07:38:06] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Sun 16 Jan 2022 04:39:56 GMT +0000. [07:39:02] RECOVERY - pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Sun 16 Jan 2022 04:43:10 GMT +0000. [07:39:08] RECOVERY - idleon.info - LetsEncrypt on sslhost is OK: OK - Certificate 'idleon.info' will expire on Sun 16 Jan 2022 04:45:16 GMT +0000. [07:39:30] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Sun 16 Jan 2022 04:50:13 GMT +0000. [07:40:02] RECOVERY - wiki.etn.link - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.etn.link' will expire on Sun 16 Jan 2022 04:51:05 GMT +0000. [07:40:40] RECOVERY - lcn.zfc.id.lv - LetsEncrypt on sslhost is OK: OK - Certificate 'lcn.zfc.id.lv' will expire on Sun 16 Jan 2022 04:52:25 GMT +0000. [07:43:33] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Sun 16 Jan 2022 04:41:58 GMT +0000. [07:44:26] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Sun 16 Jan 2022 04:38:17 GMT +0000. [08:18:29] PROBLEM - db13 APT on db13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:21:29] RECOVERY - db13 APT on db13 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [08:21:58] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 4.81, 8.73, 6.20 [08:23:05] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [08:23:46] Reception123: can you ^ [08:23:54] I assume it's the dbs like last night [08:24:38] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 2.14, 5.74, 5.44 [08:26:11] https://phabricator.miraheze.org/T8190 [08:26:12] [url] ⚓ T8190 Database connection errors spiking | phabricator.miraheze.org [08:27:47] PROBLEM - www.highdeologywiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.highdeologywiki.com could not be found [08:28:05] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [08:37:29] Spookreeeno: Do you know if it is possible to make alert not have the !_sre on resolved? [08:38:52] And remove the labels from the IRC notifications? [08:54:05] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [08:59:05] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [09:58:39] CosmicAlpha: we could handle adding the ping on the web hook based on the label [09:59:25] https://github.com/miraheze/puppet/blob/master/modules/monitoring/files/grafana-webhook.py#L23 [09:59:26] [url] puppet/grafana-webhook.py at master · miraheze/puppet · GitHub | github.com [09:59:43] You'd just have to see if it has the page label [09:59:59] Not sure for the labels [10:00:17] They come from grafana [10:01:24] Spookreeeno: re https://phabricator.miraheze.org/T8190, looking at everything I have available, I strongly believe it is a client side problem - so would be MediaWiki generating an excess amount of connections [10:01:24] [url] ⚓ T8190 Database connection errors spiking | phabricator.miraheze.org [10:01:46] JohnLewis: can we have an example of them connections [10:02:01] I can't give you an example as it's not on-going [10:02:05] So we can investigate making mediawiki not send too many [10:02:18] JohnLewis: binlogs from when we paged? [10:02:36] I can try and look... hold on [10:02:38] There was one 5 minutes before you joined [10:04:14] there wasn't according to Grafana [10:05:43] 09:54:06 alerting : [FIRING:1] (!_sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [10:06:02] JohnLewis: the alarm went off and every time I've checked it's been because of the database [10:06:08] Exceptions can be anything [10:06:16] Grafana doesn't show high connections since 1am this morning [10:06:30] CosmicAlpha: ^ [10:06:40] I'm out all day between work stuff and dentists [10:16:26] Spookreeeno: nothing in binlogs, it seems they were non-transactional requests (SELECTs likely) [10:18:21] https://grafana.miraheze.org/d/MQWgroiiz/mariadb-overview?orgId=1&var-interval=%24__auto_interval_interval&var-host=db11.miraheze.org%3A9104&from=1634509305221&to=1634511740260 [10:18:31] shows a significant increase in change DB commands [10:19:40] JohnLewis: is it a poor optimisation then? [10:20:03] Without seeing the query, I can't say [10:20:25] True [10:25:02] !log started a pt-stalk on db11 for 50 simultaneous query commands [10:25:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:28:51] Spookreeeno: ^^ so if there are 50 query commands at the same time, it'll dump the whole processlist - so let's see if that's a trigger [11:06:39] JohnLewis: ack [11:08:06] [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JK9Ef [11:08:08] [02miraheze/RemovePII] 07translatewiki 039aa7cca - Localisation updates from https://translatewiki.net. [11:08:09] [url] Main page - translatewiki.net | translatewiki.net [11:08:09] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JK9EJ [11:08:11] [02miraheze/ManageWiki] 07translatewiki 030d8524a - Localisation updates from https://translatewiki.net. [11:08:12] [url] Main page - translatewiki.net | translatewiki.net [11:17:13] miraheze/ManageWiki - translatewiki the build passed. [11:17:47] miraheze/RemovePII - translatewiki the build passed. [11:37:08] [02miraheze/mediawiki] 07RhinosF1 pushed 032 commits to 03REL1_37 [+0/-0/±4] 13https://git.io/JK91Y [11:37:10] [02miraheze/mediawiki] 07plstand 03efaae65 - Always encode spaces in cookie values as "%20" [11:37:11] [02miraheze/mediawiki] 07RhinosF1 030470eab - Merge branch 'wikimedia:REL1_37' into REL1_37 [11:42:37] !log [rhinos@test3] starting deploy of {'world': True} to skip [11:42:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:48] Agent: around? [11:46:34] !log [rhinos@test3] finished deploy of {'world': True} to skip - SUCCESS in 235s [11:46:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:47:45] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_36 [+0/-0/±2] 13https://git.io/JK99t [11:47:46] [02miraheze/mediawiki] 07plstand 036616eb7 - Always encode spaces in cookie values as "%20" [11:47:48] Reception123: ^ [11:48:24] ack [11:49:15] !log [rhinos@mw11] starting deploy of {'files': 'w/includes/libs/http/SetCookieCompatt.php'} to skip [11:49:16] !log [rhinos@mw11] finished deploy of {'files': 'w/includes/libs/http/SetCookieCompatt.php'} to skip - FAIL: [5888] in 0s [11:49:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:25] !log [rhinos@mw11] starting deploy of {'files': 'w/includes/libs/http/SetCookieCompat.php'} to skip [11:49:26] !log [rhinos@mw11] finished deploy of {'files': 'w/includes/libs/http/SetCookieCompat.php'} to skip - SUCCESS in 0s [11:49:30] !log [rhinos@mw11] starting deploy of {'files': 'w/includes/libs/http/SetCookieCompat.php'} to all [11:49:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:42] !log [rhinos@mw11] finished deploy of {'files': 'w/includes/libs/http/SetCookieCompat.php'} to all - SUCCESS in 12s [11:49:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:10] !log [rhinos@mw11] starting deploy of {'folders': 'w/tests/phpunit'} to all [11:50:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:50:16] !log [rhinos@mw11] finished deploy of {'folders': 'w/tests/phpunit'} to all - SUCCESS in 6s [11:50:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:23] !log rhinos@test3:/srv/mediawiki-staging/w/extensions/NumberedHeadings$ sudo -u www-data git apply ~/3.patch cc paladox [11:58:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:47] !log [rhinos@test3] starting deploy of {'folders': 'w/extensions/NumberedHeadings'} to skip [11:58:48] !log [rhinos@test3] finished deploy of {'folders': 'w/extensions/NumberedHeadings'} to skip - SUCCESS in 0s [11:58:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:18:42] Spookreeeno: I am now, what's up! [12:18:47] s/!/?/ [12:18:47] Agent meant to say: Spookreeeno: I am now, what's up? [12:21:28] Agent: is test3 login working for you [12:28:53] Spookreeeno: no, I still get logged off and get CentralAuth errors when logging in [12:32:45] Agent: so do I on my staff account [12:32:53] Not sure why my patch didn't work [13:08:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.01, 7.14, 5.30 [13:11:20] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.05, 5.24, 4.87 [13:24:31] PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meregos.com' expires in 15 day(s) (Wed 03 Nov 2021 13:14:35 GMT +0000). [13:37:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.41, 5.59, 5.01 [13:39:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.53, 5.29, 5.00 [13:42:38] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.78, 4.68, 4.81 [13:51:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.38, 5.19, 5.02 [13:54:22] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.87, 6.17, 5.45 [13:57:14] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.00, 5.50, 5.31 [14:02:46] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.69, 5.68, 5.36 [14:05:33] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.01, 5.31, 5.26 [14:08:18] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.80, 6.74, 5.81 [14:16:21] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.50, 5.55, 5.64 [14:19:07] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.13, 5.75, 5.70 [14:21:57] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.39, 5.85, 5.77 [14:28:20] PROBLEM - files.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'files.pornwiki.org' expires in 15 day(s) (Wed 03 Nov 2021 14:20:41 GMT +0000). [14:33:23] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.23, 6.25, 5.83 [14:36:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.10, 5.48, 5.58 [14:39:19] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.85, 6.23, 5.88 [14:40:18] PROBLEM - en.pornwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.pornwiki.org' expires in 15 day(s) (Wed 03 Nov 2021 14:30:16 GMT +0000). [14:42:08] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.21, 5.97, 5.85 [14:47:47] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.06, 6.49, 5.98 [15:05:18] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.43, 5.19, 5.92 [15:16:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.71, 5.39, 5.56 [15:19:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.18, 5.11, 5.42 [15:25:00] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.01, 3.77, 4.74 [15:49:19] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.48, 5.53, 5.23 [15:52:08] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.81, 4.96, 5.08 [16:12:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.78, 5.29, 5.12 [16:15:38] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.68, 4.71, 4.93 [16:24:59] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.75, 5.97, 5.36 [16:30:29] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.94, 5.69, 5.40 [16:39:06] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.77, 6.24, 5.56 [16:41:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.58, 5.69, 5.46 [16:44:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.37, 6.58, 5.83 [16:50:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.29, 5.96, 5.80 [16:56:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JKd09 [16:56:36] [02miraheze/mw-config] 07Universal-Omega 030c2b21f - Update ManageWikiExtensions.php [16:56:37] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [16:57:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JKdEy [16:57:13] [02miraheze/mw-config] 07Universal-Omega 038ecdf1a - Update extension-list [16:57:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JKduN [16:57:59] [02miraheze/mw-config] 07Universal-Omega 0363fd605 - Update LocalSettings.php [16:58:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JKdzb [16:58:46] [02miraheze/mw-config] 07Universal-Omega 036028a0f - Update Defines.php [16:59:07] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.97, 5.50, 5.52 [17:00:28] [02mw-config] 07Universal-Omega opened pull request 03#4160: Remove Collection + ElectronPdfService - 13https://git.io/JKda8 [17:01:30] miraheze/mw-config - Universal-Omega the build passed. [17:01:56] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.34, 5.70, 5.61 [17:04:47] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.51, 6.06, 5.72 [17:06:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JKd13 [17:06:38] [02miraheze/mw-config] 07Universal-Omega 036d1d904 - Update GlobalLogging.php [17:06:40] [02mw-config] 07Universal-Omega synchronize pull request 03#4160: Remove Collection + ElectronPdfService - 13https://git.io/JKda8 [17:07:33] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.09, 5.17, 5.44 [17:07:48] miraheze/mw-config - Universal-Omega the build passed. [17:15:58] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.22, 4.68, 5.09 [17:22:06] CosmicAlpha: feel free to merge [17:27:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.27, 5.30, 5.25 [17:29:47] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.26, 4.73, 5.03 [17:36:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.69, 5.68, 5.43 [17:50:04] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.60, 5.98, 5.61 [17:52:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.64, 5.40, 5.45 [17:53:09] CosmicAlpha: are you able to take a look at https://phabricator.miraheze.org/T8184 as I haven’t found anything obviously wrong. Tbh it kind of meets criteria for a UBN at least mitigation [17:53:10] [url] ⚓ T8184 Editing and contributions broken on FAMEPedia | phabricator.miraheze.org [17:53:38] I wonder if an XML dump, and then reimport might fix the problem [18:05:29] majavah: hi, do you have any ideas about https://phabricator.miraheze.org/T8184 ? [18:05:30] [url] ⚓ T8184 Editing and contributions broken on FAMEPedia | phabricator.miraheze.org [18:09:49] hey, I'm not at all familiar with revision storage [18:11:23] I see [18:12:05] The problem has already been identified, it's just a case of firstly debugging the issues to fix, and then probably trying to prevent it upstream [18:12:34] It was reported back in 1.19 but never fixed because the reporter never followed up with requests from developers [18:21:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.70, 4.76, 5.06 [18:32:56] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.54, 7.65, 6.18 [18:41:30] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.20, 5.84, 5.98 [18:44:04] JohnLewis: Commented on task. I have also ran that script on metawiki, compared to a revision it said was error, and it does produce error on metawiki also. So it seems to happen on most wikis, some revisions anyways. [18:44:45] what script? [18:45:01] Meta doesn't have any of these errors as far as I can see [18:45:05] checkStorage.php [18:45:12] https://meta.miraheze.org/w/index.php?diff=15358 [18:45:13] [url] Internal error - Miraheze Meta | meta.miraheze.org [18:45:28] Fun [18:46:28] I didn't even know about checkStorage, what is the output of it? [18:47:29] JohnLewis: https://phabricator.miraheze.org/T8184#165103 [18:47:31] [url] ⚓ T8184 Editing and contributions broken on FAMEPedia | phabricator.miraheze.org [18:48:40] CosmicAlpha: that's a lot of revisions [18:49:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.52, 5.98, 5.76 [18:52:41] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.21, 5.85, 5.77 [18:53:45] I don't understand how they are problematic, they look okay to me [19:01:15] concat seems problematic form of compression [19:01:20] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.76, 5.32, 5.48 [19:01:23] gzip would have been safer and better [19:03:30] I guess fixing is hard now? [19:03:48] gzip effects latest though doesn't it [19:03:57] yeah it does [19:04:15] and for how, probably impossible for private wikis [19:04:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.08, 6.85, 5.44 [19:04:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.44, 5.40, 5.49 [19:04:32] public ones should be recoverable as Reception123 does dump all public wikis in XML form [19:04:48] What about your DB backups? [19:04:58] Although data drifts [19:06:12] they're daily and we did this a few days ago [19:06:50] :( [19:07:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.32, 7.51, 5.99 [19:07:17] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.14, 5.57, 5.51 [19:07:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.87, 22.24, 19.14 [19:08:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.03, 7.50, 6.09 [19:10:09] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.20, 6.54, 5.89 [19:10:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.54, 7.66, 6.55 [19:13:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.89, 6.46, 6.08 [19:17:10] Luckily I did wikibackups just a few days ago [19:19:25] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.31, 8.32, 7.25 [19:19:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.78, 22.57, 20.67 [19:21:59] Reception123: shame it's 14G and is going to take a while to get back onto our servers :P [19:22:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.00, 21.14, 20.52 [19:23:10] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.68, 7.28, 6.41 [19:23:11] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.16, 7.98, 6.59 [19:23:17] anyway, checkStorage can fix it with an XML dump, so with that, I'll leave it with the MW team while I go back and deal with the DB issues [19:24:00] yeah and I deleted it from my homedir because it was taking up a lot [19:25:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.15, 5.33, 5.73 [19:25:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.30, 7.47, 7.35 [19:25:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.28, 19.58, 20.00 [19:28:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.46, 7.41, 6.64 [19:29:14] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.62, 6.94, 6.55 [19:31:42] [02puppet] 07Universal-Omega opened pull request 03#2055: Revert "Run compressOld.php weekly (#2035)" - 13https://git.io/JKAqy [19:31:43] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 7.69, 6.88 [19:32:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.42, 7.50, 7.07 [19:32:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.14, 6.12, 6.27 [19:32:17] Reception123: ^ Probably should do that. [19:34:18] No real point for reverting [19:34:39] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.33, 7.74, 7.32 [19:34:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.21, 6.72, 6.63 [19:35:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.59, 7.82, 7.24 [19:35:10] We're kind of stuck with this now, work needs to go with an upstream fix otherwise we've not achieved database compression but we're over half compressed [19:37:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.12, 8.31, 7.05 [19:37:19] JohnLewis: Won't it keep happening if not, if we restore from XML backup, then won't it happen again next time it runs? But yeah just noticed even if we revert that patch is wrong as cron would need absent first, I think. But will close if you don't think we need to revert. [19:38:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.08, 22.84, 21.32 [19:38:06] I mean, I guess it can be removed until a fix goes upstream, but it's kind of an irreversible change compression anyway [19:38:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [19:39:16] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKAlq [19:39:18] [02miraheze/puppet] 07Universal-Omega 03b65ebb1 - Revert "Run compressOld.php weekly (#2035)" (#2055) [19:39:19] [02puppet] 07JohnFLewis closed pull request 03#2055: Revert "Run compressOld.php weekly (#2035)" - 13https://git.io/JKAqy [19:40:29] JohnLewis: Does that not need to first be set to absent? I just thought of that after I opened the PR. [19:40:44] already removed it [19:41:09] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.45, 21.36, 21.04 [19:41:15] ah alright, thanks [19:41:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 7.68, 7.49 [19:42:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.14, 6.86, 6.83 [19:43:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 8.32, 7.11 [19:43:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.14, 7.25, 7.04 [19:43:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [19:44:08] Someone just needs to resolve the problems now [19:44:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.79, 22.82, 21.64 [19:44:16] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.53, 8.33, 7.78 [19:44:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.25, 6.44, 6.65 [19:46:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.44, 6.96, 6.79 [19:46:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.82, 6.35, 6.73 [19:47:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.00, 19.58, 20.62 [19:50:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.26, 22.30, 21.50 [19:51:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.99, 6.35, 5.78 [19:52:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.09, 6.82, 6.80 [19:52:35] JohnLewis: is it worth an alert for when we run out of mysql connections [19:53:09] Nope, because you can’t do anything but monitor it [19:53:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.97, 20.20, 20.87 [19:53:17] Restarting would be damaging potentially [19:53:23] True [19:53:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 6.87, 6.87 [19:54:24] We monitor it, so it can inform debugging but alerting is not of noticeable purpose [19:54:49] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.61, 5.94, 5.71 [19:54:49] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-2/±1] 13https://git.io/JKAM8 [19:54:51] [02miraheze/mediawiki] 07Universal-Omega 0330906d8 - Remove Collection and ElectronPdfService [19:54:52] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [19:55:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.06, 6.01, 6.67 [19:55:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.18, 6.13, 6.54 [19:56:03] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.87, 7.18, 7.94 [19:56:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.56, 5.41, 6.31 [19:56:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-2/±1] 13https://git.io/JKASO [19:56:49] [02miraheze/mediawiki] 07Universal-Omega 03c6dfa5e - Remove Collection and ElectronPdfService [19:56:51] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [19:57:40] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.43, 6.05, 5.78 [19:57:52] [02mediawiki] 07Universal-Omega opened pull request 03#3800: Remove Collection and ElectronPdfService - 13https://git.io/JKA9Q [19:57:55] [02mediawiki] 07Universal-Omega opened pull request 03#3801: Remove Collection and ElectronPdfService - 13https://git.io/JKA9N [19:59:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.86, 7.53, 7.80 [19:59:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.45, 19.30, 20.17 [20:00:36] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.20, 5.97, 5.81 [20:02:18] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JKAbT [20:02:19] [02miraheze/MirahezeMagic] 07Universal-Omega 0378b472c - Remove Collection from addWikiToServices [20:02:21] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [20:02:26] [02MirahezeMagic] 07Universal-Omega opened pull request 03#296: Remove Collection from addWikiToServices - 13https://git.io/JKAbl [20:03:31] miraheze/MirahezeMagic - Universal-Omega the build passed. [20:05:19] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.70, 8.77, 8.17 [20:05:20] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.98, 7.62, 7.44 [20:05:56] [02MirahezeMagic] 07Universal-Omega closed pull request 03#296: Remove Collection from addWikiToServices - 13https://git.io/JKAbl [20:05:58] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKAh1 [20:05:59] [02miraheze/MirahezeMagic] 07Universal-Omega 03a4145af - Remove Collection from addWikiToServices (#296) [20:06:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [20:06:01] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [20:06:03] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [20:06:56] miraheze/MirahezeMagic - Universal-Omega the build passed. [20:07:10] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.04, 20.67, 19.54 [20:08:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.01, 11.24, 8.71 [20:08:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 8.81, 7.60 [20:09:09] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JKxJ8 [20:09:11] [02miraheze/mediawiki] 07Universal-Omega 03d8ebaf2 - Update MirahezeMagic [20:10:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.77, 22.34, 21.46 [20:10:16] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.50, 19.00, 19.08 [20:10:25] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.22, 7.31, 6.84 [20:10:59] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JKxTy [20:11:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [20:11:01] [02miraheze/mediawiki] 07Universal-Omega 034946ff7 - Update MirahezeMagic [20:11:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.26, 7.11, 7.13 [20:12:41] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.03, 6.73, 6.00 [20:13:23] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.34, 6.51, 6.62 [20:14:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.93, 7.07, 7.76 [20:14:38] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.82, 7.70, 7.33 [20:17:12] [02mw-config] 07Universal-Omega closed pull request 03#4160: Remove Collection + ElectronPdfService - 13https://git.io/JKda8 [20:17:13] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [20:17:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JKx3E [20:17:16] [02miraheze/mw-config] 07Universal-Omega 03155b54e - Remove Collection + ElectronPdfService (#4160) [20:17:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.79, 8.00, 7.98 [20:17:18] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [20:17:26] [02mediawiki] 07Universal-Omega closed pull request 03#3801: Remove Collection and ElectronPdfService - 13https://git.io/JKA9N [20:17:27] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [20:17:29] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [20:17:30] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-2/±1] 13https://git.io/JKx37 [20:17:32] [02miraheze/mediawiki] 07Universal-Omega 03ea1a1e2 - Remove Collection and ElectronPdfService (#3801) [20:17:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.69, 6.99, 7.09 [20:17:40] [02mediawiki] 07Universal-Omega closed pull request 03#3800: Remove Collection and ElectronPdfService - 13https://git.io/JKA9Q [20:17:41] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-2/±1] 13https://git.io/JKxsZ [20:17:43] [02miraheze/mediawiki] 07Universal-Omega 0349fc403 - Remove Collection and ElectronPdfService (#3800) [20:17:44] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [20:17:46] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [20:18:15] miraheze/mw-config - Universal-Omega the build passed. [20:19:58] !log [@test3] starting deploy of {'config': True} to skip [20:19:59] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:20:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.09, 7.81, 7.76 [20:20:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.79, 7.17, 6.88 [20:20:51] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to skip [20:20:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.86, 5.75, 5.87 [20:22:11] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.45, 19.45, 20.27 [20:23:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.31, 7.50, 7.86 [20:25:47] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.76, 8.19, 7.22 [20:26:24] !log [@mw11] starting deploy of {'config': True} to all [20:26:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:43] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 19s [20:26:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:27] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.31, 21.94, 20.67 [20:29:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.49, 6.09, 6.68 [20:30:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.03, 5.71, 5.69 [20:31:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.12, 7.80, 7.38 [20:33:39] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.33, 5.38, 5.59 [20:34:33] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to skip - SUCCESS in 822s [20:34:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.34, 7.74, 7.40 [20:35:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.94, 9.85, 8.14 [20:35:50] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.02, 22.19, 21.06 [20:36:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [20:36:02] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.99, 25.69, 21.88 [20:36:02] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.25, 7.40, 7.12 [20:36:19] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.33, 8.92, 7.88 [20:36:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.41, 6.59, 6.02 [20:37:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.48, 8.68, 7.90 [20:38:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.81, 6.90, 7.15 [20:39:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 18.15, 20.54, 20.63 [20:39:16] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.97, 7.11, 7.09 [20:39:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.03, 23.23, 21.74 [20:40:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.34, 7.49, 7.58 [20:41:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [20:41:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.16, 7.54, 7.78 [20:42:23] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.88, 7.97, 7.42 [20:42:42] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.48, 7.47, 7.71 [20:43:08] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.82, 5.87, 5.99 [20:44:16] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all [20:44:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.17, 6.59, 7.87 [20:45:01] [02mediawiki] 07Universal-Omega closed pull request 03#3408: Bump extensions/Collection from `40eb9be` to `6379f22` - 13https://git.io/JoStV [20:45:03] [02mediawiki] 07dependabot[bot] commented on pull request 03#3408: Bump extensions/Collection from `40eb9be` to `6379f22` - 13https://git.io/JKxb2 [20:45:05] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.75, 18.71, 19.94 [20:45:10] [02mediawiki] 07Universal-Omega closed pull request 03#2845: Bump extensions/Collection from `40eb9be` to `8f53fcd` - 13https://git.io/JumHZ [20:45:12] [02mediawiki] 07dependabot[bot] commented on pull request 03#2845: Bump extensions/Collection from `40eb9be` to `8f53fcd` - 13https://git.io/JKxb9 [20:45:15] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Collection-6379f22 - 13https://git.io/vbL5b [20:45:17] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Collection-6379f22 [20:45:17] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.91, 7.87, 7.51 [20:45:18] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Collection-8f53fcd [20:45:20] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Collection-8f53fcd - 13https://git.io/vbL5b [20:45:27] [02mediawiki] 07Universal-Omega closed pull request 03#3274: Bump extensions/ElectronPdfService from `afc9c83` to `9636bed` - 13https://git.io/JoSeb [20:45:29] [02mediawiki] 07dependabot[bot] commented on pull request 03#3274: Bump extensions/ElectronPdfService from `afc9c83` to `9636bed` - 13https://git.io/JKxN3 [20:45:37] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/ElectronPdfService-9636bed - 13https://git.io/vbL5b [20:45:39] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/ElectronPdfService-9636bed [20:45:45] [02mediawiki] 07Universal-Omega closed pull request 03#3397: Bump extensions/AdvancedSearch from `5b64b4d` to `829a984` - 13https://git.io/JoSLp [20:45:47] [02mediawiki] 07dependabot[bot] commented on pull request 03#3397: Bump extensions/AdvancedSearch from `5b64b4d` to `829a984` - 13https://git.io/JKxNr [20:45:53] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/AdvancedSearch-829a984 - 13https://git.io/vbL5b [20:45:55] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/AdvancedSearch-829a984 [20:47:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.54, 6.04, 6.66 [20:48:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.99, 18.89, 20.24 [20:48:42] [02mediawiki] 07Universal-Omega closed pull request 03#3724: Bump vendor from `8740687` to `17951d4` - 13https://git.io/JKXpx [20:48:44] [02mediawiki] 07dependabot[bot] commented on pull request 03#3724: Bump vendor from `8740687` to `17951d4` - 13https://git.io/JKxh7 [20:48:53] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/vendor-17951d4 - 13https://git.io/vbL5b [20:48:55] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/vendor-17951d4 [20:49:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.22, 6.12, 6.77 [20:50:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.61, 5.55, 6.68 [20:51:11] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.38, 5.90, 6.73 [20:51:36] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.02, 5.38, 6.54 [20:56:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.38, 7.37, 7.42 [21:00:12] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.48, 4.78, 5.08 [21:01:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.32, 7.46, 7.43 [21:07:55] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.04, 5.58, 6.61 [21:09:11] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.23, 25.22, 21.92 [21:10:06] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.42, 5.98, 5.67 [21:12:17] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 1680s [21:12:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.00, 22.87, 21.54 [21:13:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.89, 6.93, 6.09 [21:15:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.01, 23.79, 22.08 [21:16:31] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.91, 23.27, 21.77 [21:16:56] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 7.51, 6.32, 4.92 [21:18:13] !log sudo -u www-data rm /srv/mediawiki/cache/extension-list.json on mw* / mwtask1 / test3 [21:18:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.46, 23.13, 22.13 [21:19:38] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 4.33, 5.44, 4.81 [21:21:27] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/migrateAccount.php --wiki=loginwiki --auto --username=Seamus Donohue (END - exit=256) [21:21:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:50] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --wiki=loginwiki --auto --username=Seamus Donohue (END - exit=0) [21:21:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:22:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.72, 17.94, 19.82 [21:22:16] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/migrateAccount.php --wiki=loginwiki --auto --username=Seamus Donohue (END - exit=0) [21:22:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:22:27] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.65, 4.50, 4.58 [21:23:45] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=frackinuniversewiki --username=Seamus Donohue (END - exit=256) [21:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:24] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=frackinuniversewiki Seamus Donohue (END - exit=0) [21:24:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:32] !log manually confirm email on above accountt [21:30:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.15, 22.03, 21.33 [21:30:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.99, 21.15, 21.05 [21:34:55] [02puppet] 07Universal-Omega opened pull request 03#2056: prometheus-es-exporter: notify to restart on config changes - 13https://git.io/JKhfp [21:36:32] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 124.04 ms [21:37:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.58, 25.23, 22.79 [21:39:41] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 113.34 ms [21:40:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.37, 23.51, 22.52 [21:43:34] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.11, 5.78, 5.17 [21:46:16] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.89, 4.45, 4.76 [21:48:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.59, 18.36, 20.28 [22:03:38] [02puppet] 07paladox closed pull request 03#2056: prometheus-es-exporter: notify to restart on config changes - 13https://git.io/JKhfp [22:03:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKhPh [22:03:41] [02miraheze/puppet] 07Universal-Omega 03da3f7c0 - prometheus-es-exporter: notify to restart on config changes (#2056) [22:03:58] [02puppet] 07paladox closed pull request 03#2052: prometheus-es-exporter: add mediawiki_params_action aggs - 13https://git.io/JKij2 [22:03:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JKhXl [22:04:01] [02miraheze/puppet] 07Universal-Omega 03c31b510 - prometheus-es-exporter: add mediawiki_params_action aggs (#2052) [22:04:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.14, 21.85, 20.62 [22:07:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.33, 20.80, 20.54 [22:10:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.01, 19.37, 20.03 [22:15:37] thanks paladox [22:22:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.46, 5.57, 5.95 [22:42:21] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.51, 4.19, 4.88 [22:43:38] https://phabricator.miraheze.org/T6979 is finally resolved now! [22:43:39] [url] ⚓ T6979 Collect Statistics for API Requests (Including Module Type) | phabricator.miraheze.org [22:48:45] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.22, 5.99, 5.37 [22:51:28] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.41, 5.10, 5.13 [22:54:13] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.87, 4.59, 4.93 [23:04:06] [02puppet] 07Universal-Omega opened pull request 03#2057: prometheus-es-exporter: query last one minute - 13https://git.io/J6eJE [23:04:35] [02puppet] 07Universal-Omega synchronize pull request 03#2057: prometheus-es-exporter: query last one minute - 13https://git.io/J6eJE [23:06:00] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [23:08:03] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 9.28, 6.66 [23:08:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.45, 7.10, 5.55 [23:08:26] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 14.30, 11.45, 7.52 [23:08:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.23, 20.91, 18.17 [23:09:10] RECOVERY - mw11 APT on mw11 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:09:10] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.46, 21.40, 18.13 [23:10:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.60, 6.96, 5.76 [23:10:48] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.64, 6.64, 6.07 [23:11:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.64, 5.54, 5.20 [23:11:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki?orgId=1 [23:11:42] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.02, 5.27, 4.91 [23:11:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.55, 18.39, 17.71 [23:12:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.46, 19.04, 17.83 [23:12:59] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.24, 5.49, 5.40 [23:14:03] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.65, 6.37, 6.48 [23:19:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.32, 6.18, 5.50 [23:22:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.00, 5.32, 5.28 [23:28:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.78, 4.52, 4.95 [23:38:30] PROBLEM - mw11 APT on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:39:09] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 16.08, 10.93, 7.12 [23:41:26] RECOVERY - mw11 APT on mw11 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:44:36] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.71, 6.03, 6.11