[00:11:55] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2936: Bump extensions/cldr from `27dbc42` to `d2748ba` - 13https://git.io/Ju1Ol
[00:11:57] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/cldr-d2748ba [+0/-0/±1] 13https://git.io/Ju1O8
[00:11:58] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03e6fc0fd - Bump extensions/cldr from `27dbc42` to `d2748ba`
[00:12:00] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2936: Bump extensions/cldr from `27dbc42` to `d2748ba` - 13https://git.io/Ju1Ol
[00:12:01] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2936: Bump extensions/cldr from `27dbc42` to `d2748ba` - 13https://git.io/Ju1Ol
[00:12:03] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MintyDocs-8b90804 [+0/-0/±1] 13https://git.io/Ju1O4
[00:12:04] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03f815ff5 - Bump extensions/MintyDocs from `77fdf03` to `8b90804`
[00:12:06] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/cldr-d2748ba - 13https://git.io/vbL5b
[00:12:07] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2937: Bump extensions/MintyDocs from `77fdf03` to `8b90804` - 13https://git.io/Ju1OB
[00:12:09] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2937: Bump extensions/MintyDocs from `77fdf03` to `8b90804` - 13https://git.io/Ju1OB
[00:12:10] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2937: Bump extensions/MintyDocs from `77fdf03` to `8b90804` - 13https://git.io/Ju1OB
[00:12:12] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MintyDocs-8b90804 - 13https://git.io/vbL5b
[00:12:13] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Truglass-600e1ca [+0/-0/±1] 13https://git.io/Ju1O0
[00:12:15] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03854d79d - Bump skins/Truglass from `b177740` to `600e1ca`
[00:12:16] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2938: Bump skins/Truglass from `b177740` to `600e1ca` - 13https://git.io/Ju1OE
[00:12:18] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Truglass-600e1ca - 13https://git.io/vbL5b
[00:12:19] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2938: Bump skins/Truglass from `b177740` to `600e1ca` - 13https://git.io/Ju1OE
[00:12:21] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2938: Bump skins/Truglass from `b177740` to `600e1ca` - 13https://git.io/Ju1OE
[00:12:22] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-f794314 [+0/-0/±1] 13https://git.io/Ju1Ou
[00:12:24] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 034bcdb21 - Bump extensions/ApprovedRevs from `4fed9f0` to `f794314`
[00:12:25] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2939: Bump extensions/ApprovedRevs from `4fed9f0` to `f794314` - 13https://git.io/Ju1Oz
[00:12:27] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-f794314 - 13https://git.io/vbL5b
[00:12:28] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2939: Bump extensions/ApprovedRevs from `4fed9f0` to `f794314` - 13https://git.io/Ju1Oz
[00:12:30] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2939: Bump extensions/ApprovedRevs from `4fed9f0` to `f794314` - 13https://git.io/Ju1Oz
[00:12:31] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/GlobalNewFiles-552acbc [+0/-0/±1] 13https://git.io/Ju1Og
[00:12:33] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03dd037db - Bump extensions/GlobalNewFiles from `8edd36d` to `552acbc`
[00:12:34] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/GlobalNewFiles-552acbc - 13https://git.io/vbL5b
[00:12:36] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2940: Bump extensions/GlobalNewFiles from `8edd36d` to `552acbc` - 13https://git.io/Ju1O2
[00:12:37] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2940: Bump extensions/GlobalNewFiles from `8edd36d` to `552acbc` - 13https://git.io/Ju1O2
[00:12:39] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2940: Bump extensions/GlobalNewFiles from `8edd36d` to `552acbc` - 13https://git.io/Ju1O2
[00:12:40] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/RottenLinks-d062d98 [+0/-0/±1] 13https://git.io/Ju1OV
[00:12:42] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03d000468 - Bump extensions/RottenLinks from `09b83d7` to `d062d98`
[00:12:43] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2941: Bump extensions/RottenLinks from `09b83d7` to `d062d98` - 13https://git.io/Ju1Ow
[00:12:45] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/RottenLinks-d062d98 - 13https://git.io/vbL5b
[00:12:46] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2941: Bump extensions/RottenLinks from `09b83d7` to `d062d98` - 13https://git.io/Ju1Ow
[00:12:48] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2941: Bump extensions/RottenLinks from `09b83d7` to `d062d98` - 13https://git.io/Ju1Ow
[00:12:49] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/chameleon-dad20f9 [+0/-0/±1] 13https://git.io/Ju1Or
[00:12:51] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03c43083d - Bump skins/chameleon from `2054484` to `dad20f9`
[00:12:52] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2942: Bump skins/chameleon from `2054484` to `dad20f9` - 13https://git.io/Ju1Oo
[00:12:54] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/chameleon-dad20f9 - 13https://git.io/vbL5b
[00:12:54] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.48, 21.00, 17.61
[00:12:55] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2942: Bump skins/chameleon from `2054484` to `dad20f9` - 13https://git.io/Ju1Oo
[00:12:57] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2942: Bump skins/chameleon from `2054484` to `dad20f9` - 13https://git.io/Ju1Oo
[00:12:58] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/RemovePII-1fcb40f [+0/-0/±1] 13https://git.io/Ju1O6
[00:13:00] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 038d5e2c1 - Bump extensions/RemovePII from `1e50332` to `1fcb40f`
[00:13:01] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2943: Bump extensions/RemovePII from `1e50332` to `1fcb40f` - 13https://git.io/Ju1Oi
[00:13:03] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/RemovePII-1fcb40f - 13https://git.io/vbL5b
[00:13:04] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2943: Bump extensions/RemovePII from `1e50332` to `1fcb40f` - 13https://git.io/Ju1Oi
[00:13:06] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2943: Bump extensions/RemovePII from `1e50332` to `1fcb40f` - 13https://git.io/Ju1Oi
[00:13:07] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MatomoAnalytics-d8e33e3 [+0/-0/±1] 13https://git.io/Ju1OP
[00:13:09] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03e12ed7c - Bump extensions/MatomoAnalytics from `afd8e91` to `d8e33e3`
[00:13:10] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2944: Bump extensions/MatomoAnalytics from `afd8e91` to `d8e33e3` - 13https://git.io/Ju1OX
[00:13:12] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MatomoAnalytics-d8e33e3 - 13https://git.io/vbL5b
[00:13:13] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2944: Bump extensions/MatomoAnalytics from `afd8e91` to `d8e33e3` - 13https://git.io/Ju1OX
[00:13:15] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2944: Bump extensions/MatomoAnalytics from `afd8e91` to `d8e33e3` - 13https://git.io/Ju1OX
[00:13:16] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2917: Bump extensions/RemovePII from `1e50332` to `b8ff8ad` - 13https://git.io/Ju1O1
[00:13:18] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2917: Bump extensions/RemovePII from `1e50332` to `b8ff8ad` - 13https://git.io/Jum55
[00:13:19] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/RemovePII-b8ff8ad - 13https://git.io/vbL5b
[00:13:21] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/RemovePII-b8ff8ad
[00:13:22] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2884: Bump extensions/MatomoAnalytics from `afd8e91` to `40aa1aa` - 13https://git.io/Ju1OD
[00:13:24] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Comments-c2f20a5 [+0/-0/±1] 13https://git.io/Ju1Oy
[00:13:25] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 030f82942 - Bump extensions/Comments from `d59530c` to `c2f20a5`
[00:13:27] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2884: Bump extensions/MatomoAnalytics from `afd8e91` to `40aa1aa` - 13https://git.io/JumQ7
[00:13:28] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2945: Bump extensions/Comments from `d59530c` to `c2f20a5` - 13https://git.io/Ju1OS
[00:13:30] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Comments-c2f20a5 - 13https://git.io/vbL5b
[00:13:31] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/MatomoAnalytics-40aa1aa - 13https://git.io/vbL5b
[00:13:33] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/MatomoAnalytics-40aa1aa
[00:13:34] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2945: Bump extensions/Comments from `d59530c` to `c2f20a5` - 13https://git.io/Ju1OS
[00:13:36] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2945: Bump extensions/Comments from `d59530c` to `c2f20a5` - 13https://git.io/Ju1OS
[00:13:37] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/WikibaseLocalMedia-daba741 [+0/-0/±1] 13https://git.io/Ju1OH
[00:13:39] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 030ed207e - Bump extensions/WikibaseLocalMedia from `e268f35` to `daba741`
[00:13:40] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2946: Bump extensions/WikibaseLocalMedia from `e268f35` to `daba741` - 13https://git.io/Ju1OQ
[00:13:42] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/WikibaseLocalMedia-daba741 - 13https://git.io/vbL5b
[00:13:43] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2946: Bump extensions/WikibaseLocalMedia from `e268f35` to `daba741` - 13https://git.io/Ju1OQ
[00:13:45] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2946: Bump extensions/WikibaseLocalMedia from `e268f35` to `daba741` - 13https://git.io/Ju1OQ
[00:13:46] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/QuizGame-a948937 [+0/-0/±1] 13https://git.io/Ju1O7
[00:13:48] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03c9c0d82 - Bump extensions/QuizGame from `ecb9da2` to `a948937`
[00:13:49] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2947: Bump extensions/QuizGame from `ecb9da2` to `a948937` - 13https://git.io/Ju1O5
[00:13:51] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/QuizGame-a948937 - 13https://git.io/vbL5b
[00:13:52] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2947: Bump extensions/QuizGame from `ecb9da2` to `a948937` - 13https://git.io/Ju1O5
[00:13:54] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2947: Bump extensions/QuizGame from `ecb9da2` to `a948937` - 13https://git.io/Ju1O5
[00:13:55] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Nimbus-b5c9b16 [+0/-0/±1] 13https://git.io/Ju1Od
[00:13:57] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03b407789 - Bump skins/Nimbus from `c0e7626` to `b5c9b16`
[00:13:58] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2948: Bump skins/Nimbus from `c0e7626` to `b5c9b16` - 13https://git.io/Ju1OF
[00:14:00] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Nimbus-b5c9b16 - 13https://git.io/vbL5b
[00:14:01] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2948: Bump skins/Nimbus from `c0e7626` to `b5c9b16` - 13https://git.io/Ju1OF
[00:14:03] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2948: Bump skins/Nimbus from `c0e7626` to `b5c9b16` - 13https://git.io/Ju1OF
[00:14:04] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-a445ddf [+0/-0/±1] 13https://git.io/Ju1ON
[00:14:06] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 034e1eca7 - Bump extensions/UniversalLanguageSelector from `27d15a9` to `a445ddf`
[00:14:07] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2949: Bump extensions/UniversalLanguageSelector from `27d15a9` to `a445ddf` - 13https://git.io/Ju1OA
[00:14:09] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-a445ddf - 13https://git.io/vbL5b
[00:14:10] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2949: Bump extensions/UniversalLanguageSelector from `27d15a9` to `a445ddf` - 13https://git.io/Ju1OA
[00:14:12] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2949: Bump extensions/UniversalLanguageSelector from `27d15a9` to `a445ddf` - 13https://git.io/Ju1OA
[00:14:13] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2861: Bump extensions/UniversalLanguageSelector from `27d15a9` to `35f48f6` - 13https://git.io/Ju1Ox
[00:14:15] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2861: Bump extensions/UniversalLanguageSelector from `27d15a9` to `35f48f6` - 13https://git.io/JumH7
[00:14:16] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-35f48f6 - 13https://git.io/vbL5b
[00:14:18] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-35f48f6
[00:14:19] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Cargo-713415d [+0/-0/±1] 13https://git.io/Ju1Op
[00:14:21] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 0359c5386 - Bump extensions/Cargo from `9d5cfb7` to `713415d`
[00:14:22] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2950: Bump extensions/Cargo from `9d5cfb7` to `713415d` - 13https://git.io/Ju1Oh
[00:14:24] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Cargo-713415d - 13https://git.io/vbL5b
[00:14:25] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2950: Bump extensions/Cargo from `9d5cfb7` to `713415d` - 13https://git.io/Ju1Oh
[00:14:27] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2950: Bump extensions/Cargo from `9d5cfb7` to `713415d` - 13https://git.io/Ju1Oh
[00:14:28] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ManageWiki-99bf640 [+0/-0/±1] 13https://git.io/Ju1Oj
[00:14:30] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 0372b91b3 - Bump extensions/ManageWiki from `4ed0092` to `99bf640`
[00:14:31] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2951: Bump extensions/ManageWiki from `4ed0092` to `99bf640` - 13https://git.io/Ju13e
[00:14:33] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2886: Bump extensions/Cargo from `9d5cfb7` to `4a012a6` - 13https://git.io/Ju13v
[00:14:34] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ManageWiki-99bf640 - 13https://git.io/vbL5b
[00:14:36] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2951: Bump extensions/ManageWiki from `4ed0092` to `99bf640` - 13https://git.io/Ju13e
[00:14:37] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2951: Bump extensions/ManageWiki from `4ed0092` to `99bf640` - 13https://git.io/Ju13e
[00:14:39] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2886: Bump extensions/Cargo from `9d5cfb7` to `4a012a6` - 13https://git.io/JumQx
[00:14:40] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Cargo-4a012a6 - 13https://git.io/vbL5b
[00:14:42] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Cargo-4a012a6
[00:14:43] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Translate-cc61770 [+0/-0/±1] 13https://git.io/Ju13f
[00:14:45] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 036db031d - Bump extensions/Translate from `a77087e` to `cc61770`
[00:14:46] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2952: Bump extensions/Translate from `a77087e` to `cc61770` - 13https://git.io/Ju13J
[00:14:48] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Translate-cc61770 - 13https://git.io/vbL5b
[00:14:49] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2952: Bump extensions/Translate from `a77087e` to `cc61770` - 13https://git.io/Ju13J
[00:14:51] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2952: Bump extensions/Translate from `a77087e` to `cc61770` - 13https://git.io/Ju13J
[00:14:52] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2919: Bump extensions/Translate from `a77087e` to `eceaaf4` - 13https://git.io/Ju13U
[00:14:54] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2919: Bump extensions/Translate from `a77087e` to `eceaaf4` - 13https://git.io/JuYJa
[00:14:55] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Translate-eceaaf4 - 13https://git.io/vbL5b
[00:14:57] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Translate-eceaaf4
[00:14:58] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/WikiDiscover-2b1afef [+0/-0/±1] 13https://git.io/Ju13T
[00:15:00] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03e6e6707 - Bump extensions/WikiDiscover from `bc961ce` to `2b1afef`
[00:15:01] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/WikiDiscover-2b1afef - 13https://git.io/vbL5b
[00:15:03] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2953: Bump extensions/WikiDiscover from `bc961ce` to `2b1afef` - 13https://git.io/Ju13k
[00:15:04] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2953: Bump extensions/WikiDiscover from `bc961ce` to `2b1afef` - 13https://git.io/Ju13k
[00:15:06] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2953: Bump extensions/WikiDiscover from `bc961ce` to `2b1afef` - 13https://git.io/Ju13k
[00:15:07] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/BlogPage-0533410 [+0/-0/±1] 13https://git.io/Ju13L
[00:15:09] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03b8a3c50 - Bump extensions/BlogPage from `b0f26f4` to `0533410`
[00:15:10] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2954: Bump extensions/BlogPage from `b0f26f4` to `0533410` - 13https://git.io/Ju13t
[00:15:12] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/BlogPage-0533410 - 13https://git.io/vbL5b
[00:15:13] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2954: Bump extensions/BlogPage from `b0f26f4` to `0533410` - 13https://git.io/Ju13t
[00:15:15] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2954: Bump extensions/BlogPage from `b0f26f4` to `0533410` - 13https://git.io/Ju13t
[00:15:16] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/foreground-ab4aa18 [+0/-0/±1] 13https://git.io/Ju133
[00:15:18] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 0331e36d5 - Bump skins/foreground from `d8fe013` to `ab4aa18`
[00:15:19] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/foreground-ab4aa18 - 13https://git.io/vbL5b
[00:15:21] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2955: Bump skins/foreground from `d8fe013` to `ab4aa18` - 13https://git.io/Ju13s
[00:15:22] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2955: Bump skins/foreground from `d8fe013` to `ab4aa18` - 13https://git.io/Ju13s
[00:15:24] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2955: Bump skins/foreground from `d8fe013` to `ab4aa18` - 13https://git.io/Ju13s
[00:15:25] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/LocalisationUpdate-a968199 [+0/-0/±1] 13https://git.io/Ju13G
[00:15:27] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03c6d3048 - Bump extensions/LocalisationUpdate from `7645909` to `a968199`
[00:15:28] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/LocalisationUpdate-a968199 - 13https://git.io/vbL5b
[00:15:30] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2956: Bump extensions/LocalisationUpdate from `7645909` to `a968199` - 13https://git.io/Ju13Z
[00:15:31] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2956: Bump extensions/LocalisationUpdate from `7645909` to `a968199` - 13https://git.io/Ju13Z
[00:15:33] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2956: Bump extensions/LocalisationUpdate from `7645909` to `a968199` - 13https://git.io/Ju13Z
[00:15:34] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageForms-90e6e29 [+0/-0/±1] 13https://git.io/Ju13n
[00:15:36] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 0352ea9c9 - Bump extensions/PageForms from `f3dabf1` to `90e6e29`
[00:15:37] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2957: Bump extensions/PageForms from `f3dabf1` to `90e6e29` - 13https://git.io/Ju13c
[00:15:39] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageForms-90e6e29 - 13https://git.io/vbL5b
[00:15:40] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2957: Bump extensions/PageForms from `f3dabf1` to `90e6e29` - 13https://git.io/Ju13c
[00:15:42] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2957: Bump extensions/PageForms from `f3dabf1` to `90e6e29` - 13https://git.io/Ju13c
[00:15:43] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Maps-476f850 [+0/-0/±1] 13https://git.io/Ju13C
[00:15:45] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03f897914 - Bump extensions/Maps from `b43865b` to `476f850`
[00:15:46] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Maps-476f850 - 13https://git.io/vbL5b
[00:15:48] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2931: Bump extensions/PageForms from `f3dabf1` to `123a7c4` - 13https://git.io/Ju13W
[00:15:49] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2958: Bump extensions/Maps from `b43865b` to `476f850` - 13https://git.io/Ju13l
[00:15:51] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2931: Bump extensions/PageForms from `f3dabf1` to `123a7c4` - 13https://git.io/JuYUY
[00:15:52] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2958: Bump extensions/Maps from `b43865b` to `476f850` - 13https://git.io/Ju13l
[00:15:54] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2958: Bump extensions/Maps from `b43865b` to `476f850` - 13https://git.io/Ju13l
[00:15:55] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/PageForms-123a7c4 - 13https://git.io/vbL5b
[00:15:57] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/PageForms-123a7c4
[00:15:58] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2853: Bump extensions/Maps from `b43865b` to `5d74ef5` - 13https://git.io/Ju134
[00:16:00] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2853: Bump extensions/Maps from `b43865b` to `5d74ef5` - 13https://git.io/JumHa
[00:16:01] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-270b3a6 [+0/-0/±1] 13https://git.io/Ju13B
[00:16:03] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 0365e7a49 - Bump extensions/MirahezeMagic from `a44b22c` to `270b3a6`
[00:16:04] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Maps-5d74ef5
[00:16:06] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2959: Bump extensions/MirahezeMagic from `a44b22c` to `270b3a6` - 13https://git.io/Ju13R
[00:16:07] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Maps-5d74ef5 - 13https://git.io/vbL5b
[00:16:09] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-270b3a6 - 13https://git.io/vbL5b
[00:16:10] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2959: Bump extensions/MirahezeMagic from `a44b22c` to `270b3a6` - 13https://git.io/Ju13R
[00:16:12] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2959: Bump extensions/MirahezeMagic from `a44b22c` to `270b3a6` - 13https://git.io/Ju13R
[00:16:13] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/DiscordNotifications-37615a3 [+0/-0/±1] 13https://git.io/Ju13u
[00:16:15] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 031132662 - Bump extensions/DiscordNotifications from `f5d70db` to `37615a3`
[00:16:16] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/DiscordNotifications-37615a3 - 13https://git.io/vbL5b
[00:16:18] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2960: Bump extensions/DiscordNotifications from `f5d70db` to `37615a3` - 13https://git.io/Ju13z
[00:16:19] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2960: Bump extensions/DiscordNotifications from `f5d70db` to `37615a3` - 13https://git.io/Ju13z
[00:16:21] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2960: Bump extensions/DiscordNotifications from `f5d70db` to `37615a3` - 13https://git.io/Ju13z
[00:16:22] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Cosmos-9f29f04 [+0/-0/±1] 13https://git.io/Ju13g
[00:16:24] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03c1aee26 - Bump skins/Cosmos from `ae4f902` to `9f29f04`
[00:16:25] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2961: Bump skins/Cosmos from `ae4f902` to `9f29f04` - 13https://git.io/Ju132
[00:16:27] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Cosmos-9f29f04 - 13https://git.io/vbL5b
[00:16:28] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2961: Bump skins/Cosmos from `ae4f902` to `9f29f04` - 13https://git.io/Ju132
[00:16:30] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2961: Bump skins/Cosmos from `ae4f902` to `9f29f04` - 13https://git.io/Ju132
[00:16:31] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PollNY-0361e36 [+0/-0/±1] 13https://git.io/Ju13a
[00:16:33] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 0335dbb0f - Bump extensions/PollNY from `d3f170d` to `0361e36`
[00:16:34] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PollNY-0361e36 - 13https://git.io/vbL5b
[00:16:36] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2962: Bump extensions/PollNY from `d3f170d` to `0361e36` - 13https://git.io/Ju13V
[00:16:37] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2962: Bump extensions/PollNY from `d3f170d` to `0361e36` - 13https://git.io/Ju13V
[00:16:39] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2962: Bump extensions/PollNY from `d3f170d` to `0361e36` - 13https://git.io/Ju13V
[00:16:40] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/snapwikiskin-3e052f1 [+0/-0/±1] 13https://git.io/Ju13w
[00:16:42] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03d19ff44 - Bump skins/snapwikiskin from `e2fc153` to `3e052f1`
[00:16:43] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2963: Bump skins/snapwikiskin from `e2fc153` to `3e052f1` - 13https://git.io/Ju13r
[00:16:45] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/snapwikiskin-3e052f1 - 13https://git.io/vbL5b
[00:16:46] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2963: Bump skins/snapwikiskin from `e2fc153` to `3e052f1` - 13https://git.io/Ju13r
[00:16:48] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2963: Bump skins/snapwikiskin from `e2fc153` to `3e052f1` - 13https://git.io/Ju13r
[00:16:49] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/SocialProfile-ad72aef [+0/-0/±1] 13https://git.io/Ju13o
[00:16:51] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03268a55f - Bump extensions/SocialProfile from `cdfc049` to `ad72aef`
[00:16:52] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2964: Bump extensions/SocialProfile from `cdfc049` to `ad72aef` - 13https://git.io/Ju13K
[00:16:54] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-ad72aef - 13https://git.io/vbL5b
[00:16:55] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2964: Bump extensions/SocialProfile from `cdfc049` to `ad72aef` - 13https://git.io/Ju13K
[00:16:57] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2964: Bump extensions/SocialProfile from `cdfc049` to `ad72aef` - 13https://git.io/Ju13K
[00:16:58] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/CategoryTree-0899a28 [+0/-0/±1] 13https://git.io/Ju136
[00:17:00] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03c0ede62 - Bump extensions/CategoryTree from `03489fc` to `0899a28`
[00:17:01] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/CategoryTree-0899a28 - 13https://git.io/vbL5b
[00:17:03] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2965: Bump extensions/CategoryTree from `03489fc` to `0899a28` - 13https://git.io/Ju13i
[00:17:04] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2965: Bump extensions/CategoryTree from `03489fc` to `0899a28` - 13https://git.io/Ju13i
[00:17:06] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2965: Bump extensions/CategoryTree from `03489fc` to `0899a28` - 13https://git.io/Ju13i
[00:17:07] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/DataDump-3305c04 [+0/-0/±1] 13https://git.io/Ju13P
[00:17:09] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03c7d553d - Bump extensions/DataDump from `2a22c9c` to `3305c04`
[00:17:10] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2966: Bump extensions/DataDump from `2a22c9c` to `3305c04` - 13https://git.io/Ju13X
[00:17:10] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.64, 19.21, 17.83
[00:17:12] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/DataDump-3305c04 - 13https://git.io/vbL5b
[00:17:13] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2966: Bump extensions/DataDump from `2a22c9c` to `3305c04` - 13https://git.io/Ju13X
[00:17:15] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2966: Bump extensions/DataDump from `2a22c9c` to `3305c04` - 13https://git.io/Ju13X
[00:17:16] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2925: Bump extensions/DataDump from `2a22c9c` to `2fa05c6` - 13https://git.io/Ju131
[00:17:18] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/VoteNY-7eb7d90 [+0/-0/±1] 13https://git.io/Ju13M
[00:17:19] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 034e76435 - Bump extensions/VoteNY from `084ac2e` to `7eb7d90`
[00:17:21] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2925: Bump extensions/DataDump from `2a22c9c` to `2fa05c6` - 13https://git.io/JuYJd
[00:17:22] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2967: Bump extensions/VoteNY from `084ac2e` to `7eb7d90` - 13https://git.io/Ju13D
[00:17:24] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/VoteNY-7eb7d90 - 13https://git.io/vbL5b
[00:17:25] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/DataDump-2fa05c6 - 13https://git.io/vbL5b
[00:17:27] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/DataDump-2fa05c6
[00:17:28] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2967: Bump extensions/VoteNY from `084ac2e` to `7eb7d90` - 13https://git.io/Ju13D
[00:17:30] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2967: Bump extensions/VoteNY from `084ac2e` to `7eb7d90` - 13https://git.io/Ju13D
[00:17:31] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2898: Bump extensions/VoteNY from `084ac2e` to `6d35d98` - 13https://git.io/Ju13y
[00:17:33] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/CreateWiki-cc9f1e0 [+0/-0/±1] 13https://git.io/Ju13S
[00:17:34] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] 03c9339a0 - Bump extensions/CreateWiki from `36a1f88` to `cc9f1e0`
[00:17:36] <Not>	 [02mediawiki] 07dependabot[bot] opened pull request 03#2968: Bump extensions/CreateWiki from `36a1f88` to `cc9f1e0` - 13https://git.io/Ju139
[00:17:37] <Not>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-cc9f1e0 - 13https://git.io/vbL5b
[00:17:39] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2898: Bump extensions/VoteNY from `084ac2e` to `6d35d98` - 13https://git.io/Jum7X
[00:17:40] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2968: Bump extensions/CreateWiki from `36a1f88` to `cc9f1e0` - 13https://git.io/Ju139
[00:17:42] <Not>	 [02mediawiki] 07dependabot[bot] labeled pull request 03#2968: Bump extensions/CreateWiki from `36a1f88` to `cc9f1e0` - 13https://git.io/Ju139
[00:17:43] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/VoteNY-6d35d98 - 13https://git.io/vbL5b
[00:17:45] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/VoteNY-6d35d98
[00:17:46] <Not>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#2880: Bump extensions/CreateWiki from `36a1f88` to `75cea2b` - 13https://git.io/Ju13Q
[00:17:48] <Not>	 [02mediawiki] 07dependabot[bot] closed pull request 03#2880: Bump extensions/CreateWiki from `36a1f88` to `75cea2b` - 13https://git.io/JumQw
[00:17:49] <Not>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-75cea2b - 13https://git.io/vbL5b
[00:17:51] <Not>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-75cea2b
[00:23:01] <icinga-miraheze>	 PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg]
[00:37:06] <icinga-miraheze>	 RECOVERY - studentwiki.ddns.net - reverse DNS on sslhost is OK: SSL OK - studentwiki.ddns.net reverse DNS resolves to cp14.miraheze.org - CNAME OK
[00:47:05] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.59, 3.12
[00:48:57] <icinga-miraheze>	 RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[00:53:46] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.28, 3.16
[00:58:33] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.43, 3.27
[01:02:50] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.23, 3.24
[01:07:28] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.72, 3.45
[01:14:01] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.17, 3.28, 3.34
[01:18:42] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 3.91, 3.57
[01:20:55] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.80, 3.58
[01:23:08] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.97, 3.68
[01:25:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.87, 3.67
[01:27:25] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 4.37, 3.90
[01:28:02] <Not>	 [02mw-config] 07Universal-Omega edited pull request 03#4089: T8005: Use new 'ErrorPages' repository - 13https://git.io/JuBoj
[01:29:21] <Not>	 [02mediawiki] 07Universal-Omega closed pull request 03#2935: Remove 404.php from .gitignore - 13https://git.io/JuoMI
[01:29:22] <Not>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/Ju18s
[01:29:24] <Not>	 [02miraheze/mediawiki] 07Universal-Omega 039c33704 - Remove 404.php from .gitignore (#2935)
[01:29:25] <Not>	 [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b
[01:29:27] <Not>	 [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1
[01:30:51] <Not>	 [02mw-config] 07Universal-Omega closed pull request 03#4089: T8005: Use new 'ErrorPages' repository - 13https://git.io/JuBoj
[01:30:52] <Not>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3
[01:30:54] <Not>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3
[01:30:55] <Not>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-2/±1] 13https://git.io/Ju180
[01:30:57] <Not>	 [02miraheze/mw-config] 07Universal-Omega 031616742 - T8005: Use new 'ErrorPages' repository (#4089)
[01:31:56] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.96, 3.95, 3.87
[01:31:58] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[01:33:49] <MirahezeLSBot>	 !log [@test3] starting deploy of {'config': True} to skip
[01:33:50] <MirahezeLSBot>	 !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
[01:33:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:33:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:36:50] <Not>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Ju14J
[01:36:52] <Not>	 [02miraheze/mw-config] 07Universal-Omega 03d26f4ed - Fix tabbing, caused by CI when it tried to replace spaces
[01:36:54] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#4097: LS: disable jquery migration on testwikis - 13https://git.io/JuP3z
[01:37:57] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[01:42:56] <icinga-miraheze>	 PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[01:43:00] <icinga-miraheze>	 PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[01:43:04] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[01:44:20] <icinga-miraheze>	 PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[01:44:32] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[01:45:15] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.43, 3.52
[01:45:33] <icinga-miraheze>	 PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[01:47:26] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.41, 3.50
[01:49:45] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.49, 3.95, 3.68
[01:56:28] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.93, 3.82
[01:56:43] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.61, 1.77, 1.49
[01:58:55] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.27, 1.58, 1.45
[02:05:20] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[02:05:51] <icinga-miraheze>	 RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[02:05:53] <icinga-miraheze>	 RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:06:35] <icinga-miraheze>	 RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[02:06:41] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.02, 5.88, 5.31
[02:06:50] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[02:07:06] <icinga-miraheze>	 RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[02:08:54] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.11, 5.51, 5.25
[02:24:38] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 3.20, 3.39
[02:51:45] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.45, 3.20
[02:53:54] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.22, 3.08, 3.10
[03:04:42] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.61, 3.29
[03:11:10] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.29, 3.27
[03:30:14] <icinga-miraheze>	 PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.28, 5.14, 3.16
[03:32:45] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 16.43, 10.14, 6.39
[03:32:56] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 8.29, 5.90
[03:33:22] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 35.19, 25.98, 17.09
[03:33:39] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.59, 7.10, 5.04
[03:35:12] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.16, 6.59, 5.59
[03:35:24] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.94, 6.82, 5.82
[03:35:54] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.82, 5.77, 4.82
[03:37:14] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.06, 6.72, 5.93
[03:37:25] <icinga-miraheze>	 RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.31, 5.02, 4.13
[03:37:31] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.44, 6.36, 5.78
[03:37:46] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.08, 20.64, 17.21
[03:39:51] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.97, 17.75, 16.59
[05:07:48] <Not>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JuMUS
[05:07:49] <Not>	 [02miraheze/dns] 07Reception123 03c5fc175 - add jp.popira.info zone
[05:14:18] <Not>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuMT1
[05:14:19] <Not>	 [02miraheze/dns] 07Reception123 03f11598e - fix
[05:16:29] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.77, 7.88, 5.49
[05:16:59] <Not>	 [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/JuMkU
[05:17:00] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.10, 8.19, 5.54
[05:17:01] <Not>	 [02miraheze/ssl] 07Reception123 03eee34a1 - add wiki.potabi.com cert
[05:17:02] <Not>	 [02miraheze/ssl] 07Reception123 03c9364a2 - add wiki.apadotech.ga cert
[05:17:21] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.10, 7.97, 5.73
[05:18:12] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 7.65, 5.61
[05:18:48] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.54, 7.06, 5.54
[05:19:10] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.42, 7.43, 5.62
[05:19:27] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.54, 7.05, 5.68
[05:20:22] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.74, 6.79, 5.58
[05:21:35] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 2.61, 5.52, 5.17
[05:21:37] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.68, 5.12, 5.04
[05:21:40] <CosmicAlpha|AWAY>	 Umm... why did MirahezeBot kick Southparkfan?
[05:21:51] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 2.45, 5.32, 5.23
[05:22:41] <Reception123>	 hmm, we should ask RhinosF1 when he joins
[05:24:01] <CosmicAlpha|AWAY>	 Yeah and I think that it's weird it only kicked from here and not #miraheze as well.
[05:27:59] <Agent>	 MirahezeBot attempted takeover of this channel???
[05:57:25] <Reception123>	 !log reimaging mwtask1 to debian 11 (bullseye)
[05:57:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:00:55] <icinga-miraheze>	 PROBLEM - ping4 on mwtask1 is CRITICAL: PING CRITICAL - Packet loss = 100%
[06:01:43] <icinga-miraheze>	 PROBLEM - Host mwtask1 is DOWN: PING CRITICAL - Packet loss = 100%
[06:06:53] <Reception123>	 CosmicAlpha|AWAY: still around?
[06:07:36] <CosmicAlpha|AWAY>	 Reception123: semi, will be going completely for the night in ~15 minutes.
[06:07:55] <Reception123>	 CosmicAlpha|AWAY: any idea why the VM would be stuck on 'booting from hard disk'?
[06:08:29] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.67, 6.72, 4.63
[06:08:52] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.80, 7.08, 5.09
[06:09:32] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.98, 7.73, 5.52
[06:11:34] <CosmicAlpha|AWAY>	 Hmm. Not sure.
[06:11:53] <Reception123>	 I'm trying to reinstall since I'm not sure how else it could be fixed
[06:12:09] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.24, 8.34, 6.09
[06:14:06] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.18, 7.08, 5.73
[06:14:41] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.77, 7.04, 5.95
[06:14:58] <Reception123>	 seems to work now
[06:15:17] <CosmicAlpha|AWAY>	 Great!
[06:16:14] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.34, 5.87, 5.20
[06:16:40] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.85, 6.06, 5.54
[06:17:19] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.88, 5.52, 5.53
[06:18:55] <CosmicAlpha|AWAY>	 OK I am going to go now.
[06:19:42] <icinga-miraheze>	 RECOVERY - Host mwtask1 is UP: PING OK - Packet loss = 0%, RTA = 1.37 ms
[06:20:06] <icinga-miraheze>	 PROBLEM - ping6 on mwtask1 is CRITICAL: CRITICAL - Destination Unreachable (2001:41d0:800:1bbd::15)
[06:20:07] <icinga-miraheze>	 RECOVERY - ping6 on mwtask1 is OK: PING OK - Packet loss = 0%, RTA = 1.97 ms
[06:20:13] <icinga-miraheze>	 RECOVERY - ping4 on mwtask1 is OK: PING OK - Packet loss = 0%, RTA = 2.42 ms
[06:30:38] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.87, 6.77, 5.82
[06:32:06] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.64, 7.23, 5.84
[06:32:42] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.78, 18.56, 15.92
[06:32:48] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.13, 7.93, 6.40
[06:34:16] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.41, 3.17
[06:35:30] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.90, 8.97, 6.66
[06:37:31] <Not>	 [02puppet] 07Reception123 closed pull request 03#1899: mwtask1: update php - 13https://git.io/JE2YX
[06:37:33] <Not>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuM8m
[06:37:34] <Not>	 [02miraheze/puppet] 07RhinosF1 033fc69a8 - mwtask1: update php (#1899)
[06:37:52] <paladox>	 Nice
[06:40:26] <Reception123>	 paladox: hi. Thankfully everything seems to be working fine for the moment :) 
[06:40:33] <Reception123>	 though it does take quite a bit of time to reimage in total
[06:40:47] <Reception123>	 (currently running puppet with --server)
[06:40:55] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.40, 3.27
[06:46:16] <paladox>	 Hi
[06:47:54] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.10, 20.06, 19.11
[06:49:33] <Reception123>	 paladox: hmm, just as I was saying everything was going fine for some reason I was just randomly disconnected from the server
[06:49:37] <Reception123>	 and it seems like it randomly shut down
[06:50:11] <Reception123>	 and in the notes it says "Error: Connection error - server offline"
[06:50:31] <Reception123>	 oh I see, my SSH connection locally timed out for some odd reason
[06:50:53] <Reception123>	 I hope that doesn't mean I have to redo everything though if I'm not able to access root again
[06:51:12] <Reception123>	 ok, thankfully it seems to be back now
[06:52:01] <icinga-miraheze>	 PROBLEM - mwtask1 JobRunner Service on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[06:52:02] <icinga-miraheze>	 PROBLEM - mwtask1 APT on mwtask1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (5 critical updates).
[06:52:16] <icinga-miraheze>	 PROBLEM - mwtask1 MirahezeRenewSsl on mwtask1 is CRITICAL: connect to address 198.244.181.23 and port 5000: Connection refused
[06:52:19] <icinga-miraheze>	 PROBLEM - mwtask1 MediaWiki Rendering on mwtask1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1287 bytes in 0.008 second response time
[06:52:45] <icinga-miraheze>	 PROBLEM - mwtask1 Puppet on mwtask1 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file
[06:53:03] <icinga-miraheze>	 PROBLEM - mwtask1 Check Gluster Clients on mwtask1 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs'
[06:54:06] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.05, 7.19, 7.98
[06:54:16] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.91, 6.60, 7.77
[06:54:50] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.05, 7.12, 7.98
[06:58:49] <icinga-miraheze>	 PROBLEM - mwtask1 MediaWiki Rendering on mwtask1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.064 second response time
[07:00:51] <icinga-miraheze>	 RECOVERY - mwtask1 JobRunner Service on mwtask1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[07:01:05] <icinga-miraheze>	 PROBLEM - mwtask1 MediaWiki Rendering on mwtask1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 761 bytes in 0.815 second response time
[07:01:25] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.64, 7.36, 7.64
[07:01:53] <icinga-miraheze>	 RECOVERY - mwtask1 Check Gluster Clients on mwtask1 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs'
[07:03:02] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 5.68, 6.71
[07:03:31] <Not>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuMEL
[07:03:33] <Not>	 [02miraheze/puppet] 07Reception123 03b373698 - fix typo
[07:03:38] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.46, 6.34, 7.21
[07:07:12] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.37, 5.45, 6.66
[07:10:00] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.15, 5.37, 6.55
[07:11:16] <Reception123>	 paladox: for some reason renewSSL won't start
[08:23:55] <Reception123>	 !log depool mw8
[08:24:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:24:05] <Reception123>	 !log starting reimage for mw8 (bullseye)
[08:24:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:28:01] <icinga-miraheze>	 PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8
[08:28:24] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8
[08:29:31] <icinga-miraheze>	 PROBLEM - mw8 APT on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:29:39] <icinga-miraheze>	 PROBLEM - mw8 JobRunner Service on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:29:43] <icinga-miraheze>	 PROBLEM - mw8 HTTPS on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:29:51] <icinga-miraheze>	 PROBLEM - ping4 on mw8 is CRITICAL: PING CRITICAL - Packet loss = 100%
[08:29:54] <icinga-miraheze>	 PROBLEM - ping6 on mw8 is CRITICAL: PING CRITICAL - Packet loss = 100%
[08:30:28] <icinga-miraheze>	 PROBLEM - mw8 PowerDNS Recursor on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:30:28] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8
[08:30:33] <icinga-miraheze>	 PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8
[08:30:49] <icinga-miraheze>	 PROBLEM - mw8 SSH on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:30:51] <icinga-miraheze>	 PROBLEM - mw8 Disk Space on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:31:05] <icinga-miraheze>	 PROBLEM - mw8 NTP time on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:31:11] <icinga-miraheze>	 PROBLEM - mw8 Check Gluster Clients on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:31:15] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:31:16] <icinga-miraheze>	 PROBLEM - mw8 php-fpm on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:31:34] <icinga-miraheze>	 PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100%
[08:31:41] <icinga-miraheze>	 PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:31:45] <icinga-miraheze>	 PROBLEM - cp14 Stunnel Http for mw8 on cp14 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:32:11] <icinga-miraheze>	 PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:32:34] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:40:01] <icinga-miraheze>	 RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 5.95 ms
[08:40:01] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: connect to address 51.195.236.221 port 5666: Connection refusedconnect to host 51.195.236.221 port 5666: Connection refused
[08:40:21] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: connect to address 51.195.236.221 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[08:40:31] <icinga-miraheze>	 RECOVERY - ping4 on mw8 is OK: PING OK - Packet loss = 0%, RTA = 2.35 ms
[08:41:01] <icinga-miraheze>	 RECOVERY - mw8 SSH on mw8 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0)
[08:45:02] <icinga-miraheze>	 RECOVERY - ping6 on mw8 is OK: PING OK - Packet loss = 0%, RTA = 2.07 ms
[08:51:42] <Reception123>	 paladox: RhinosF1 still around?
[08:52:30] <Reception123>	 nevermind, fixed.
[08:59:39] <Not>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuMbX
[08:59:40] <Not>	 [02miraheze/puppet] 07RhinosF1 03a4ea5ff - mw8: update php (#1897)
[08:59:42] <Not>	 [02puppet] 07Reception123 closed pull request 03#1897: mw8: update php - 13https://git.io/JE2Yo
[09:01:53] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw8 on cp3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.746 second response time
[09:02:29] <icinga-miraheze>	 PROBLEM - cp14 Stunnel Http for mw8 on cp14 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.003 second response time
[09:02:31] <icinga-miraheze>	 PROBLEM - cp13 Stunnel Http for mw8 on cp13 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.003 second response time
[09:02:55] <icinga-miraheze>	 PROBLEM - cp12 Stunnel Http for mw8 on cp12 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.232 second response time
[09:18:40] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 1.018 second response time
[09:18:55] <icinga-miraheze>	 RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15711 bytes in 0.005 second response time
[09:18:55] <icinga-miraheze>	 RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.005 second response time
[09:19:17] <icinga-miraheze>	 RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.324 second response time
[09:21:51] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 761 bytes in 0.073 second response time
[09:21:58] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file
[09:22:50] <Reception123>	 JohnLewis: around by any chance?
[09:23:03] <icinga-miraheze>	 PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates).
[09:24:55] <JohnLewis>	 Yeah?
[09:26:25] <Reception123>	 JohnLewis: there were two things r.e. reimaging. Right now on mw8 there's a weird error that I didn't see on mwtask1 or test3 at all
[09:26:27] <Reception123>	 Error: 'npm install --cache /tmp/npm-cache' returned 1 instead of one of [0]
[09:26:27] <Reception123>	 Error: /Stage[main]/Mediawiki::Servicessetup/Exec[3d2png_npm]/returns: change from 'notrun' to ['0'] failed: 'npm install --cache /tmp/npm-cache' returned 1 instead of one of [0] (corrective)
[09:26:42] <Reception123>	 and when trying that command manually it brings a whole new set of problems
[09:26:45] <Reception123>	 https://www.irccloud.com/pastebin/Gn64l3E8/
[09:26:56] <Reception123>	 I'm not sure how it worked fine on the two other servers but suddenly it's problematic here
[09:30:50] <paladox>	 It cannot find python?!
[09:31:10] <Reception123>	 paladox: yeah I'm not sure what's going on with that
[09:33:37] <Reception123>	 any ideas about what I can try?
[09:34:34] <majavah>	 bullseye does not include python2
[09:34:53] <JohnLewis>	 ^^
[09:35:08] <Reception123>	 but how come I didn't encounter any issue with puppet like that on test3?
[09:35:10] <JohnLewis>	 Python3 is the command not python
[09:35:14] <Reception123>	 ah
[09:35:32] <Reception123>	 paladox: the other issue was (perhaps that's why I didn't see the error on mwtask1) that mirahezerenewssl doesn't want to start
[09:35:43] <Reception123>	 https://www.irccloud.com/pastebin/Ve7CL02e/
[09:36:28] <Reception123>	 so if the command is python3, then where can that be fixed so that npm does that ?
[09:40:22] <Reception123>	 JohnLewis: ^
[09:42:30] <JohnLewis>	 There’s a —python option
[09:44:31] <Reception123>	 hmm when I did help I didn't see one but let me look
[09:46:06] <Reception123>	 https://www.irccloud.com/pastebin/0y1lBjtG/
[09:48:47] <JohnLewis>	 It says < 3 so never mind.
[09:49:31] <Reception123>	 yeah that's kind of confusing, so what am I supposed to do then?
[09:52:20] <JohnLewis>	 All I can think is this is a problem with 3d2png here as Mathoid worked on mw8
[09:54:45] <Reception123>	 JohnLewis: yeah, though I wonder if the reason why I didn't see this happen on mwtask1 is because it was later during the puppet run perhaps? (which is currently stuck on the other renewssl error that I also don't know what do about)
[09:54:50] <Reception123>	 though on test3 there was also no issue
[09:56:40] <JohnLewis>	 What’s the problem with sslrenew?
[09:57:28] <Reception123>	 I have no idea all I've seen is https://www.irccloud.com/pastebin/Ve7CL02e/
[09:57:40] <Reception123>	 https://www.irccloud.com/pastebin/3sozjt0s/
[10:01:36] <Reception123>	 hmm now I have a feeling it has to be related to the recent changes - https://github.com/miraheze/puppet/commits/master/modules/letsencrypt/files/mirahezerenewssl.py
[10:01:37] <MirahezeBot>	 [url] History for modules/letsencrypt/files/mirahezerenewssl.py - miraheze/puppet · GitHub | github.com
[10:02:45] <JohnLewis>	 FileNotFoundError: [Errno 2] No such file or directory: '/var/log/letsencrypt/miraheze-renewal.log'
[10:04:11] <icinga-miraheze>	 RECOVERY - mwtask1 MirahezeRenewSsl on mwtask1 is OK: TCP OK - 0.000 second response time on 198.244.181.23 port 5000
[10:04:28] <Reception123>	 hmm, why didn't that dir get created automatically?
[10:05:09] <JohnLewis>	 Probably because puppet doesn’t create it?
[10:05:38] <Reception123>	 ah, so we probably need to have something for that then
[10:06:03] <Reception123>	 well it seems even with this fixed mwtask1 is saying nothing about 3d2png
[10:06:58] <Reception123>	 probably minor but it's a bit strange that every time I run puppet on mwtask1 it says stuff about timezone
[10:10:11] <Reception123>	 I copied some files over from mwtask1 to mw8 and puppet seems to be running fine but it's doing that timezone thing as well
[10:10:29] <Reception123>	 and there's still a HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 761 bytes in 0.061 second response time
[10:15:24] <icinga-miraheze>	 RECOVERY - mwtask1 MediaWiki Rendering on mwtask1 is OK: HTTP OK: HTTP/1.1 200 OK - 20004 bytes in 2.129 second response time
[10:15:45] <JohnLewis>	 It’s always done the timezone thing
[10:16:20] <icinga-miraheze>	 RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19994 bytes in 0.100 second response time
[10:16:21] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki on mwtask1/mw8
[10:16:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:17:00] <Reception123>	 I don't remember it doing it all the time, but either way it's not a big deal
[10:17:11] <Reception123>	 !log repool mw8
[10:17:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:17:47] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[10:18:18] <icinga-miraheze>	 RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy
[10:19:04] <Reception123>	 JohnLewis: one last issue, icinga doesn't want to check the status of puppet because 
[10:19:04] <Reception123>	 UNKNOWN: Failed to check. Reason is: no_summary_file
[10:19:35] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy
[10:19:35] <icinga-miraheze>	 RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy
[10:21:26] <RhinosF1>	 Reception123: apt upgrade for puppet I think
[10:21:32] <RhinosF1>	 But no I'm not here
[10:21:42] <RhinosF1>	 Only getting fresh air from cold
[10:23:25] <Reception123>	 an upgrade?
[10:34:29] <Reception123>	 !log depool mw8 (500 HTTP error)
[10:34:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:35:54] <Reception123>	 !log rm -rf /srv/mediawiki/w on mw8 (and re-run puppet)
[10:35:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:36:55] <icinga-miraheze>	 PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8
[10:37:46] <icinga-miraheze>	 PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8
[10:37:57] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8
[10:38:36] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8
[10:38:54] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1287 bytes in 0.004 second response time
[10:49:03] <icinga-miraheze>	 RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19995 bytes in 0.198 second response time
[10:51:55] <Reception123>	 !log repool mw8
[10:51:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:52:34] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy
[10:53:22] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[10:53:42] <icinga-miraheze>	 RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy
[10:54:11] <icinga-miraheze>	 RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy
[11:07:36] <Not>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JuDwh
[11:07:38] <Not>	 [02miraheze/CreateWiki] 07translatewiki 03dc5a476 - Localisation updates from https://translatewiki.net.
[11:07:38] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:39] <Not>	 [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+8/-0/±0] 13https://git.io/JuDre
[11:07:40] <Not>	 [02miraheze/GlobalNewFiles] 07translatewiki 03d43474d - Localisation updates from https://translatewiki.net.
[11:07:41] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:42] <Not>	 [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+6/-0/±0] 13https://git.io/JuDrf
[11:07:44] <Not>	 [02miraheze/RemovePII] 07translatewiki 039053857 - Localisation updates from https://translatewiki.net.
[11:07:45] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:45] <Not>	 [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuDrU
[11:07:46] <Not>	 [02miraheze/landing] 07translatewiki 039bbe02e - Localisation updates from https://translatewiki.net.
[11:07:47] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:07:48] <Not>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+8/-0/±2] 13https://git.io/JuDrT
[11:07:50] <Not>	 [02miraheze/MirahezeMagic] 07translatewiki 03937f519 - Localisation updates from https://translatewiki.net.
[11:07:50] <MirahezeBot>	 [url] Main page - translatewiki.net | translatewiki.net
[11:08:36] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[11:08:40] <miraheze-github_>	 miraheze/MirahezeMagic - translatewiki the build passed.
[11:08:47] <miraheze-github_>	 miraheze/landing - translatewiki the build passed.
[11:09:27] <miraheze-github>	 miraheze/GlobalNewFiles - translatewiki the build passed.
[11:10:00] <miraheze-github>	 miraheze/RemovePII - translatewiki the build passed.
[11:33:37] <paladox>	 JohnLewis: I thought python is symlinkrd?
[11:33:44] <paladox>	 *symlinked
[11:34:04] <JohnLewis>	 Paladox: doesn’t seem so
[11:34:17] <paladox>	 I think we can do that then
[11:35:42] <paladox>	 Reception123: you copied all the certificates right?
[11:35:47] <paladox>	 For mwtask1
[11:36:22] <paladox>	 Otherwise when the certs need renewing you’ll need to do it all manually including private keys need changing (for obvious reasons the keys would be lost)
[11:37:25] <Reception123>	 Oh, I didn't realise. So then everything will need to be regenerated?
[11:37:37] <paladox>	 Yes
[11:38:19] <Reception123>	 That's fun..
[12:41:06] <icinga-miraheze>	 PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8
[12:41:40] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.749 second response time
[12:41:45] <icinga-miraheze>	 PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8
[12:41:45] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.005 second response time
[12:41:55] <icinga-miraheze>	 PROBLEM - cp14 Stunnel Http for mw8 on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.003 second response time
[12:41:56] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8
[12:41:59] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8
[12:42:06] <icinga-miraheze>	 PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time
[12:42:30] <icinga-miraheze>	 PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.003 second response time
[12:42:44] <icinga-miraheze>	 PROBLEM - mw8 php-fpm on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4'
[12:43:28] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.51, 7.84, 5.98
[12:46:17] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.22, 7.67, 5.83
[12:47:11] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.00, 7.41, 5.76
[12:51:19] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.67, 3.00
[12:53:36] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.21, 2.91
[12:58:02] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 7.62, 6.93
[12:59:23] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.93, 7.73, 7.27
[13:01:37] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.52, 7.80, 7.33
[13:02:54] <icinga-miraheze>	 RECOVERY - mw8 php-fpm on mw8 is OK: PROCS OK: 33 processes with command name 'php-fpm7.4'
[13:03:24] <icinga-miraheze>	 RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy
[13:03:33] <icinga-miraheze>	 RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19995 bytes in 0.218 second response time
[13:03:48] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 1.037 second response time
[13:03:53] <icinga-miraheze>	 RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.332 second response time
[13:03:59] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy
[13:04:11] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[13:04:12] <icinga-miraheze>	 RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 6.075 second response time
[13:04:12] <icinga-miraheze>	 RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.019 second response time
[13:04:29] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.46, 7.90, 7.21
[13:05:30] <icinga-miraheze>	 RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy
[13:05:59] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.39, 7.65, 7.47
[13:06:40] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.83, 6.71, 6.86
[13:08:46] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.97, 5.81, 6.51
[13:09:28] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.14, 3.00
[13:09:38] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.59, 6.85, 7.86
[13:10:11] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.09, 5.53, 6.63
[13:13:49] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 2.99, 2.98
[13:18:05] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.27, 5.02, 6.53
[14:40:00] <RhinosF1>	 paladox, Reception123: ping
[14:47:11] <RhinosF1>	 !log stop nginx on mw8 to force a depool
[14:47:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:50:41] <RhinosF1>	 !log rebuild l10n, fix composer & reset ManageWiki cache
[14:50:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:51:00] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: connect to address 51.195.236.221 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[14:51:04] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.06, 6.81, 5.52
[14:51:12] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host
[14:51:23] <icinga-miraheze>	 PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: HTTP CRITICAL - No data received from host
[14:51:41] <icinga-miraheze>	 PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8
[14:51:46] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.77, 7.88, 5.87
[14:51:57] <icinga-miraheze>	 PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: HTTP CRITICAL - No data received from host
[14:52:07] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8
[14:52:09] <icinga-miraheze>	 PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8
[14:52:14] <icinga-miraheze>	 PROBLEM - mw8 HTTPS on mw8 is CRITICAL: connect to address 51.195.236.221 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[14:52:18] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8
[14:52:19] <icinga-miraheze>	 PROBLEM - cp14 Stunnel Http for mw8 on cp14 is CRITICAL: HTTP CRITICAL - No data received from host
[14:52:25] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 3.68, 3.22
[14:53:18] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.82, 7.96, 6.15
[14:56:57] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.83, 8.72, 6.34
[15:00:32] <icinga-miraheze>	 PROBLEM - mw8 JobRunner Service on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[15:00:41] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.26, 7.94, 6.98
[15:01:29] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.65, 7.54, 6.48
[15:01:34] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.16, 3.55, 3.54
[15:04:41] <MirahezeLSBot>	 !log [rhinos@mw8] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=2)
[15:04:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:05:04] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.45, 7.65, 7.06
[15:05:52] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 7.52, 6.69
[15:06:02] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.91, 3.67
[15:08:11] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.63, 7.68, 6.88
[15:08:22] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.69, 3.64
[15:10:23] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.23, 7.93, 7.06
[15:12:33] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.98, 7.51, 7.02
[15:14:46] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.09, 7.59, 7.09
[15:19:02] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.78, 7.98, 7.39
[15:19:32] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.75, 3.60
[15:20:49] <MirahezeLSBot>	 !log [rhinos@mw8] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0)
[15:21:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:21:28] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.95, 9.17, 7.92
[15:22:27] <RhinosF1>	 !log restart jobrunner (but not traffic) on mw8 - should be safe
[15:22:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:23:23] <RhinosF1>	 !log and depool again
[15:23:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:24:19] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 4.00, 3.78
[15:24:48] <RhinosF1>	 majavah: any idea how to fix PHP Warning: curl_multi_setopt(): CURLPIPE_HTTP1 is no longer supported
[15:25:01] <RhinosF1>	 coming from /srv/mediawiki/w/maintenance/runJobs.php --wiki=r2wiki --type=htmlCacheUpdate --maxtime=60 --memory-limit=192M --result=json
[15:25:14] <RhinosF1>	 https://www.irccloud.com/pastebin/fslMSVHx/
[15:25:53] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.46, 20.81, 19.15
[15:26:08] <paladox>	 Hi
[15:26:24] <RhinosF1>	 paladox: can you properly depool mw8
[15:28:21] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.51, 20.17, 19.14
[15:29:36] <majavah>	 RhinosF1: https://phabricator.wikimedia.org/T264735 suggests a core bug
[15:29:37] <MirahezeBot>	 [url] ⚓ T264735 MultiHttpClient has hard-coded deprecated cURL option used by CdnCacheUpdate | phabricator.wikimedia.org
[15:30:54] <RhinosF1>	 majavah: does that mean mediawiki doesn't work with php7.4
[15:31:14] <RhinosF1>	 i guess we need to downgrade curl
[15:31:45] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.73, 3.70
[15:31:49] <paladox>	 No
[15:32:06] <paladox>	 RhinosF1: did it say warning or error?
[15:32:24] <majavah>	 warning
[15:32:25] <RhinosF1>	 paladox: ErrorException
[15:32:35] <paladox>	 It’s a warning
[15:32:47] <paladox>	 So it won’t affect anything but we will get log spam
[15:32:54] <RhinosF1>	 ok
[15:33:00] <RhinosF1>	 i can exclude that then
[15:33:07] <RhinosF1>	 !log restart jobrunner
[15:33:13] <RhinosF1>	 !log on mw8
[15:33:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:34:01] <paladox>	 We should try and resolve the issue upstream
[15:34:30] <RhinosF1>	 yes
[15:34:38] <RhinosF1>	 i'm concerned about triage now
[15:34:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:34:56] <RhinosF1>	 jacknjellifysbfdiwiki was generating a few errors on the reset cache
[15:35:17] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[15:35:50] <icinga-miraheze>	 RECOVERY - mw8 JobRunner Service on mw8 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[15:36:08] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb
[15:36:53] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.96, 3.85
[15:37:53] <MirahezeLSBot>	 !log [rhinos@mw8] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0)
[15:38:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:38:36] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[15:40:05] <RhinosF1>	 !log repool mw8
[15:40:07] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.79, 20.50, 19.89
[15:40:10] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[15:40:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:40:31] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 1.023 second response time
[15:41:04] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy
[15:41:08] <RhinosF1>	 paladox: scream if errors rise
[15:41:13] <RhinosF1>	 hope I fixed everything
[15:41:20] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[15:41:21] <icinga-miraheze>	 RECOVERY - mw8 HTTPS on mw8 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 541 bytes in 0.027 second response time
[15:41:27] <icinga-miraheze>	 RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19995 bytes in 0.284 second response time
[15:41:33] <icinga-miraheze>	 RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.009 second response time
[15:41:38] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 4.00, 3.90
[15:41:57] <icinga-miraheze>	 RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.050 second response time
[15:42:01] <icinga-miraheze>	 RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.353 second response time
[15:42:36] <icinga-miraheze>	 RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy
[15:42:54] <icinga-miraheze>	 RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy
[15:44:01] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.73, 3.60, 3.77
[15:44:52] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.91, 19.85, 19.83
[15:51:09] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.12, 3.92, 3.79
[15:55:44] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.71, 3.79
[15:57:49] <Reception123>	 paladox: So just to make sure, every single LE cert will have to be fully regenerated including private keys? I assume it's better to do them when the icinga warnings come up rather than say in a week or else there will be a lot of renewals at once
[15:58:10] <RhinosF1>	 Why not recover from private puppet
[15:58:18] <RhinosF1>	 Shouldn't all private keys be on puppet3
[15:58:21] <RhinosF1>	 Hi darkmatterman450
[15:58:32] <darkmatterman450>	 Oh hello, once again.
[16:01:06] <RhinosF1>	 darkmatterman450: I haven't broken anything have i
[16:01:17] <darkmatterman450>	 Not that I know of, no.
[16:01:48] <RhinosF1>	 Good
[16:03:50] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.768 second response time
[16:04:02] <RhinosF1>	 Huh
[16:04:07] <RhinosF1>	 That's not great
[16:04:17] <MH-Discord>	 <DarkMatterMan4500> What happened now?
[16:04:27] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.028 second response time
[16:04:30] <icinga-miraheze>	 PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.003 second response time
[16:04:34] <RhinosF1>	 php-fpm has crashed
[16:04:35] <RhinosF1>	 One sec
[16:04:50] <icinga-miraheze>	 PROBLEM - mw8 php-fpm on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4'
[16:04:51] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.52, 3.60
[16:04:51] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8
[16:04:52] <icinga-miraheze>	 PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.238 second response time
[16:04:54] <icinga-miraheze>	 PROBLEM - cp14 Stunnel Http for mw8 on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.003 second response time
[16:05:33] <icinga-miraheze>	 PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8
[16:05:55] <RhinosF1>	 php7.4-fpm.service: A process of this unit has been killed by the OOM killer.
[16:06:16] <icinga-miraheze>	 PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8
[16:06:36] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8
[16:06:46] <RhinosF1>	 paladox, Reception123: ^
[16:06:54] <RhinosF1>	 Mem:            3960         319        3089           1         551        3325
[16:07:01] <RhinosF1>	 total        used        free      shared  buff/cache   available
[16:07:21] <MH-Discord>	 <DarkMatterMan4500> Were you just copypasting that from where you saw the error?
[16:07:25] <RhinosF1>	 !log stop nginx; start php; start nginx
[16:07:25] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.61, 3.62
[16:07:28] <RhinosF1>	 from now
[16:07:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:08:45] <icinga-miraheze>	 RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy
[16:08:49] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15711 bytes in 1.014 second response time
[16:09:03] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy
[16:09:15] <icinga-miraheze>	 RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.008 second response time
[16:09:15] <icinga-miraheze>	 RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19995 bytes in 0.264 second response time
[16:09:38] <icinga-miraheze>	 RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15705 bytes in 0.311 second response time
[16:09:43] <icinga-miraheze>	 RECOVERY - mw8 php-fpm on mw8 is OK: PROCS OK: 33 processes with command name 'php-fpm7.4'
[16:09:44] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[16:09:48] <icinga-miraheze>	 RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.004 second response time
[16:10:19] <icinga-miraheze>	 RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy
[16:18:32] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.11, 3.39
[16:19:09] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.95, 6.14, 7.88
[16:22:57] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.96, 5.71, 7.57
[16:23:22] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.65, 3.55
[16:23:34] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.87, 6.26, 7.75
[16:27:15] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.11, 4.50, 6.60
[16:27:39] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.15, 3.36
[16:27:47] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.33, 4.79, 6.61
[16:29:45] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.01, 4.83, 6.55
[16:48:49] <paladox>	 Reception123: yes
[16:55:14] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.41, 3.28
[16:55:25] <RhinosF1>	 paladox: if you have access can you double check syslog for info on the OOM
[16:57:27] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.13, 3.18
[17:02:05] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.77, 3.46
[17:03:17] <Not>	 [02mediawiki] 07R4356th opened pull request 03#2969: Install EmbedSpotify (T7567) - 13https://git.io/JuS3K
[17:04:17] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.98, 3.18, 3.29
[17:18:09] <icinga-miraheze>	 PROBLEM - mw8 php-fpm on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4'
[17:18:12] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8
[17:18:12] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.054 second response time
[17:18:14] <icinga-miraheze>	 PROBLEM - cp14 Stunnel Http for mw8 on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.003 second response time
[17:18:19] <Reception123>	 paladox: what about what RhinosF1 said? couldn't we copy keys from puppet3?
[17:18:32] <RhinosF1>	 Reception123: depool mw8
[17:18:35] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8
[17:18:38] <RhinosF1>	 Something with php is broke
[17:18:45] <icinga-miraheze>	 PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8
[17:18:45] <icinga-miraheze>	 PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8
[17:18:51] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.21, 6.88, 5.64
[17:19:00] <RhinosF1>	 Reception123: I don't think we can trust it
[17:19:04] <Reception123>	 !log depool mw8
[17:19:07] <icinga-miraheze>	 PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.245 second response time
[17:19:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:19:17] <icinga-miraheze>	 PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.004 second response time
[17:19:20] <RhinosF1>	 Reception123: something isn't right
[17:19:23] <Reception123>	 What could be going so terribly wrong?
[17:19:25] <RhinosF1>	 It shouldn't OOM 3 times
[17:19:28] <RhinosF1>	 Memory issues
[17:19:46] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.786 second response time
[17:19:51] <RhinosF1>	 Could be renamed config or something
[17:21:15] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.66, 6.73, 5.76
[17:21:29] <Reception123>	 paladox: do you have any idea?
[17:22:50] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.01, 7.41, 6.32
[17:23:05] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.87, 6.80, 5.85
[17:24:57] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.95, 7.19, 6.37
[17:25:47] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.29, 6.86, 6.11
[17:27:19] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.95, 7.69, 6.39
[17:27:57] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.67, 6.56, 6.07
[17:29:24] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.98, 8.54, 7.07
[17:32:25] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.20, 8.01, 6.77
[17:34:33] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.44, 7.66, 6.80
[17:36:46] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.67, 8.48, 7.25
[17:38:14] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.51, 7.61, 7.16
[17:38:51] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.38, 7.43, 7.01
[17:40:22] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.02, 7.58, 7.19
[17:42:12] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.10, 7.59, 7.53
[17:42:26] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.80, 6.23, 6.76
[17:43:15] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.00, 5.67, 6.41
[17:46:13] <RhinosF1>	 majavah: any idea why php on Debian bullseye persistently gets killed by OOM reaper
[17:46:27] <majavah>	 no
[17:46:41] <majavah>	 do you have the full stack trace somewhere?
[17:48:41] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.39, 7.73, 7.39
[17:48:45] <RhinosF1>	 As much as I have is 17:05:55 <RhinosF1> php7.4-fpm.service: A process of this unit has been killed by the OOM killer. majavah
[17:49:15] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.75, 8.24, 7.30
[17:49:29] <majavah>	 that's practically useless for trying to figure out why it died
[17:50:57] <majavah>	 what do kernel logs look like?
[17:51:00] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.91, 3.32
[17:52:15] <majavah>	 do you have any non-default fpm config?
[17:52:29] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.38, 7.64, 6.99
[17:53:14] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.37, 3.19
[17:54:15] <RhinosF1>	 majavah: I don't have kernel log access
[17:55:02] <RhinosF1>	 And https://github.com/miraheze/puppet/tree/master/modules/php/templates for config
[17:55:02] <MirahezeBot>	 [url] puppet/modules/php/templates at master · miraheze/puppet · GitHub | github.com
[17:56:19] <majavah>	 the real config seems to be coming from https://github.com/miraheze/puppet/blob/master/modules/php/manifests/fpm/pool.pp#L55
[17:56:20] <MirahezeBot>	 [url] puppet/pool.pp at master · miraheze/puppet · GitHub | github.com
[17:58:04] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.64, 3.37
[17:58:07] <majavah>	 trying to decrease the worker count is one option, attaching a debugger and trying to locate a memory leak is another, I don't really know how to do the second one really
[17:59:21] <paladox>	 Reception123: no we couldn’t I don’t think. You’ll complicate it by 1000%. Easier and quicker to just regenerate.
[17:59:37] <RhinosF1>	 paladox: help!
[17:59:48] <RhinosF1>	 If puppet is disabled it may be enabled
[18:00:24] <paladox>	 Oh I can have a look in a little bit
[18:00:35] <paladox>	 I’m currently about to have dinner shortly (mobile)
[18:01:05] <paladox>	 majavah: exactly what I was going to try
[18:01:14] <paladox>	 (To the first option)
[18:01:33] <RhinosF1>	 paladox: ok
[18:02:28] <RhinosF1>	 paladox: but why would it only OOM on bullseye
[18:05:02] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.31, 3.32
[18:12:15] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.71, 3.46
[18:14:36] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.64, 3.47
[18:17:00] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.89, 3.59
[18:19:28] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.88, 3.64
[18:21:53] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.83, 3.64
[18:26:35] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.85, 3.74
[18:29:25] <paladox>	 RhinosF1: i have no idea
[18:31:13] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.88, 3.75
[18:32:14] <paladox>	 found out why python isn't symlinked cc Reception123 RhinosF1 
[18:32:42] <Reception123>	 why's that?
[18:33:27] <Reception123>	 paladox: well I don't think regeneration is that easy since we have like hundreds of certs, but if you think there's no chance of copying working then I guess we'll do that
[18:33:32] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.68, 3.70
[18:34:06] <paladox>	 Reception123: well because python installed python-minimal  which did that.. but python3 doesn't do the same. You have to manually install python3-minimal .
[18:34:23] <Reception123>	 oh
[18:35:20] <Not>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JuSVn
[18:35:21] <Not>	 [02miraheze/puppet] 07paladox 033bfbf7e - mediawiki: Install python3-minimal
[18:35:23] <Not>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[18:35:24] <Not>	 [02puppet] 07paladox opened pull request 03#1968: mediawiki: Install python3-minimal - 13https://git.io/JuSVC
[18:35:36] <paladox>	 we may want to move that to base or what ever but for now mediawiki is fine :)
[18:35:49] <Not>	 [02puppet] 07paladox closed pull request 03#1968: mediawiki: Install python3-minimal - 13https://git.io/JuSVC
[18:35:50] <Not>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JuSVB
[18:35:52] <Not>	 [02miraheze/puppet] 07paladox 033bfbf7e - mediawiki: Install python3-minimal
[18:35:53] <Not>	 [02miraheze/puppet] 07paladox 03116b7bb - Merge pull request #1968 from miraheze/paladox-patch-1
[18:35:55] <Not>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[18:35:55] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.06, 4.25, 3.91
[18:35:56] <Not>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[18:36:32] <Reception123>	 Well it seems then I'll have to pause any further plans to move to bullseye until we figure out what's going on
[18:37:58] <icinga-miraheze>	 RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15705 bytes in 0.321 second response time
[18:38:16] <icinga-miraheze>	 RECOVERY - mw8 php-fpm on mw8 is OK: PROCS OK: 33 processes with command name 'php-fpm7.4'
[18:38:30] <icinga-miraheze>	 RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.005 second response time
[18:38:55] <icinga-miraheze>	 RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19995 bytes in 0.830 second response time
[18:38:55] <RhinosF1>	 Reception123: yes
[18:38:59] <icinga-miraheze>	 RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 0.010 second response time
[18:39:07] <RhinosF1>	 paladox: ok
[18:39:13] <RhinosF1>	 Who younched mw8
[18:39:34] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:39:36] <Reception123>	 I've only connected to it, didn't touch anything
[18:39:36] <paladox>	 me
[18:39:48] <paladox>	 !log properly depooling mw8
[18:40:00] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15697 bytes in 1.015 second response time
[18:40:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:40:45] <RhinosF1>	 paladox: what's properly depool
[18:41:05] <paladox>	 depooled via cache proxies
[18:41:22] <paladox>	 https://www.irccloud.com/pastebin/faalOMOz/
[18:41:24] <paladox>	 huh
[18:41:33] <paladox>	 ohhh
[18:41:40] <paladox>	 i misread the listed files :ultraseeth:
[18:42:22] <RhinosF1>	 paladox: python SHOULD NOT be used
[18:42:35] <paladox>	 huh?
[18:42:35] <RhinosF1>	 There's a Python-is-python3 package if we want to be stupid
[18:42:48] <RhinosF1>	 paladox: it not being there is deliberate and by design
[18:42:56] <RhinosF1>	 Python == python2 =! Python3
[18:43:31] <paladox>	 ok, well i think that it should be at least symlinked otherwise we're going to be waiting a while to fix that npm package only for us to have it fixed now...
[18:43:57] <paladox>	 ohhh
[18:44:01] <paladox>	 i see what you mean
[18:45:49] <RhinosF1>	 paladox: if the npm package needs Python, it means python2 and is broke
[18:45:57] <RhinosF1>	 My only question is why test3 didn't cry
[18:46:15] <Reception123>	 or mwtask1
[18:46:19] <Reception123>	 nothing happened there either
[18:46:33] <Reception123>	 I thought it was because mirahezerenewssl made puppet stuck but still nothing after that
[18:46:48] <Reception123>	 so that's why I thought it was really weird - two mw servers say nothing but mw8 randomly does?
[18:47:36] <RhinosF1>	 yes weird
[18:47:50] <Reception123>	 I honestly can't think of anything that would cause that
[18:47:56] <RhinosF1>	 But I am sending a very clear -2 to that npm package on bullseye
[18:48:12] <Reception123>	 though what's weirder is the other issues its been having 
[18:48:16] <RhinosF1>	 It's unlikely if it's worth it's salt to work anyway
[18:48:21] <RhinosF1>	 Yes they are more important
[18:48:37] <paladox>	 https://github.com/nodejs/node-gyp/issues/1687
[18:48:38] <MirahezeBot>	 [url] Why no support Python3 · Issue #1687 · nodejs/node-gyp · GitHub | github.com
[18:48:42] <paladox>	 RhinosF1: ^
[18:51:51] <RhinosF1>	 paladox: that says support is out
[18:53:54] * Reception123 wonders what more we can do to get to the bottom of the mw8 mystery
[18:54:21] <paladox>	 RhinosF1: yeh
[18:54:22] <paladox>	 https://gerrit.wikimedia.org/r/c/3d2png/+/720699
[18:54:43] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.52, 3.83
[18:57:37] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[18:58:01] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[18:59:08] <icinga-miraheze>	 PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:59:25] <Reception123>	 really?
[18:59:34] <RhinosF1>	 paladox: k
[18:59:45] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 50%
[18:59:55] <RhinosF1>	 majavah: can I use your wonderful +2 powers
[19:00:30] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[19:01:14] <majavah>	 sorry, I have no access there
[19:01:30] <icinga-miraheze>	 RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20007 bytes in 0.629 second response time
[19:02:13] <majavah>	 RhinosF1: I don't have access there
[19:02:16] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 2.76, 3.32
[19:02:29] <RhinosF1>	 majavah: I thought you had +2
[19:02:35] <RhinosF1>	 Who would
[19:02:45] <majavah>	 https://gerrit.wikimedia.org/r/admin/groups/9d675dd6ee2d14a54f58d9db1bc86f11b6a62d49,members
[19:03:23] <majavah>	 I have on mw/* and some other misc, but it doesn't extend there
[19:04:00] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.75, 8.16, 5.86
[19:04:13] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.16, 8.49, 6.27
[19:06:01] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[19:06:34] <icinga-miraheze>	 PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.64, 7.17, 5.88
[19:06:50] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.00, 6.77, 5.99
[19:07:04] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 8%
[19:07:34] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[19:08:05] <RhinosF1>	 Reception123: what happened with servers
[19:08:10] <RhinosF1>	 Except mwtrash
[19:08:10] <paladox>	 RhinosF1: i've cherry picked the commit
[19:08:23] <RhinosF1>	 paladox: ok cool
[19:08:32] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[19:08:35] <RhinosF1>	 Now just need to make it not OOM
[19:08:49] <paladox>	 npm ERR! command git ls-remote ssh://git@github.com/wikimedia/kad.git
[19:08:50] <paladox>	 huh
[19:08:53] <paladox>	 why does it use ssh
[19:08:59] <icinga-miraheze>	 RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.07, 5.95, 5.61
[19:09:24] <RhinosF1>	 paladox: stupid
[19:15:12] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.86, 20.62, 19.76
[19:17:37] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.00, 20.22, 19.73
[19:19:56] <paladox>	 RhinosF1: https://github.com/wikimedia/limitation/pull/6
[19:19:57] <MirahezeBot>	 [url] package.json: Use GitHub-style reference to wikimedia/kad by kostajh · Pull Request #6 · wikimedia/limitation · GitHub | github.com
[19:22:12] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb
[19:23:12] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb
[19:25:05] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw13 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:26:31] <icinga-miraheze>	 PROBLEM - cp13 Stunnel Http for mw13 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:26:39] <icinga-miraheze>	 PROBLEM - cp12 Stunnel Http for mw13 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:27:43] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw13 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15708 bytes in 6.161 second response time
[19:28:58] <icinga-miraheze>	 RECOVERY - cp13 Stunnel Http for mw13 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15708 bytes in 5.849 second response time
[19:29:05] <icinga-miraheze>	 RECOVERY - cp12 Stunnel Http for mw13 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15708 bytes in 5.763 second response time
[19:29:31] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[19:30:40] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[19:39:17] <icinga-miraheze>	 PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.95, 20.65, 20.18
[19:43:48] <icinga-miraheze>	 RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.67, 19.79, 20.02
[19:51:01] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.68, 3.32
[19:55:43] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.97, 3.11, 3.21
[20:22:31] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.94, 3.44
[20:29:25] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.28, 3.77, 3.61
[20:36:07] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.26, 4.06, 3.72
[20:40:44] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.88, 3.73
[20:42:25] <icinga-miraheze>	 PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[20:50:05] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.77, 3.67
[20:54:51] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.86, 3.74
[21:01:52] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.70, 2.96, 3.35
[21:08:10] <icinga-miraheze>	 RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:20:35] <RhinosF1>	 paladox: do we have any idea when mw8 will be safe to pool
[21:21:00] <paladox>	 No, we need to fix python first.
[21:21:58] <RhinosF1>	 paladox: why didn't this appear on mwtask/test
[21:22:07] <paladox>	 i have no idea
[21:22:19] <paladox>	 i didn't reimage either of those.
[21:22:22] <RhinosF1>	 But I thought you cherry picked the fix
[21:22:32] <paladox>	 onto mw8
[21:22:41] <RhinosF1>	 Which led to an even stupider error about ssh
[21:22:42] <paladox>	 but it's trying to use ssh for one of the packages
[21:24:18] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.44, 3.27
[21:24:27] <paladox>	 we'll have to get it to use the github account i guess (to clone)
[21:26:37] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.04, 3.01, 3.14
[21:27:28] <RhinosF1>	 paladox: chatting to Petr
[21:27:33] <paladox>	 ok
[21:27:36] <paladox>	 he's around?
[21:27:37] <RhinosF1>	 What about the OOMs
[21:27:39] <RhinosF1>	 Yes
[21:27:42] <RhinosF1>	 In core
[21:27:45] <paladox>	 i haven't looked at the OOMs yet
[21:28:00] <RhinosF1>	 They're my big issue
[21:28:49] <paladox>	 !log repooling mw8 to have a look at memory usage
[21:28:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:30:14] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy
[21:30:28] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[21:30:47] <paladox>	 i wonder if the OOMs could be the npm modules related
[21:30:58] <icinga-miraheze>	 RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy
[21:31:02] <icinga-miraheze>	 RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy
[21:31:02] <RhinosF1>	 paladox: why repool
[21:31:12] <RhinosF1>	 Surely you can throw fake traffic at it with a script
[21:31:17] <RhinosF1>	 But I have 0 idea
[21:31:18] <paladox>	 I need to see this happen life and try and figure it out
[21:31:25] <RhinosF1>	 Also help in -core
[21:39:08] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.50, 5.36, 7.79
[21:40:27] <RhinosF1>	 JohnLewis: mw8 went very wrong
[21:40:40] <RhinosF1>	 It is currently depooled and we are working on the many issues
[21:40:52] <RhinosF1>	 I'll try and talk to Reception123 in the morning
[21:41:04] <RhinosF1>	 But we've got patches up upstream
[21:42:08] <JohnLewis>	 Still not resolved from this afternoon when it was first reported about 12 hours ago?
[21:43:32] <RhinosF1>	 JohnLewis: as soon as I fixed the obvious composer + l10n weren't working and did the reset caches to avoid the inevitable namespace issues, the server OOM'd and killed php so I restarted php and it did it again
[21:43:42] <RhinosF1>	 Then Reception123 depooled it properly
[21:43:59] <RhinosF1>	 Then paladox decided to look at the npm errors it spewed this morning
[21:44:03] <RhinosF1>	 And we stand here now
[21:44:20] <RhinosF1>	 With npm being worked on and it being watched to debug OOM's
[21:44:41] <RhinosF1>	 I think paladox just repooled actually to make it OOM again and debug
[21:44:44] <RhinosF1>	 I'm tired
[21:44:50] <paladox>	 It's not OOM'g
[21:45:19] <paladox>	 https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw8.miraheze.org&var-port=9100&viewPanel=78 though i'm keeping an eye on this
[21:45:20] <MirahezeBot>	 [url] Grafana | grafana.miraheze.org
[21:46:20] <RhinosF1>	 paladox: it took ~30 minutes the last 2 times
[21:46:26] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.41, 6.52, 7.83
[21:46:30] <paladox>	 ok
[21:46:34] <RhinosF1>	 Before we got the killed by OOM killer message
[21:46:54] <RhinosF1>	 I got that from sudo service php7.4-fpm status
[21:46:57] <icinga-miraheze>	 RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:47:07] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.80, 6.25, 7.70
[21:47:34] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.70, 5.16, 6.68
[21:56:12] <paladox>	 RhinosF1 / Reception123 it seems some lessons need to be learnt by this as compatibility with debian bullseye wasn't fully tested.
[21:56:45] <RhinosF1>	 paladox: I have no idea why it didnt show up on test3
[21:57:05] <RhinosF1>	 It's been sat on bullseye for weeks
[21:57:07] <paladox>	 who ever install test3 must have seen an error when puppet was running
[21:57:21] <paladox>	 as test3 doesn't have 'python' either and it's node_modules directory is half empty
[21:57:26] <paladox>	 missing packages
[21:57:30] <RhinosF1>	 Aug 26
[21:57:46] <RhinosF1>	 nothing will have 'python', it's now 'python3'
[21:57:51] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.51, 5.25, 6.48
[21:58:02] <paladox>	 yes
[21:58:24] <RhinosF1>	 but yes when Reception123 ran puppet it should have been spotted
[21:58:39] <RhinosF1>	 We had quite a few issues though so it might have been drowned out
[21:59:14] * RhinosF1 wonders if there's a way we can monitor installed dependencies
[21:59:45] <paladox>	 we shouldn't be using npm in production so that would at least resolve something...
[22:00:03] <paladox>	 you can blame me for introducing npm into production tho
[22:01:03] <icinga-miraheze>	 PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.81, 1.67, 1.42
[22:01:08] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.28, 5.47, 6.50
[22:02:55] <RhinosF1>	 We shouldn't 'blame' each other really
[22:03:09] <RhinosF1>	 We should look at what we can do to avoid this in the future
[22:03:13] <icinga-miraheze>	 RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.20, 1.46, 1.37
[22:03:29] <RhinosF1>	 monitoring dependices is something on my low list anyway
[22:04:08] <RhinosF1>	 But I'm on a drive to scrap a lot of our tech debt because there's heaps of it and make the work we do far more efficient
[22:04:13] <paladox>	 https://www.irccloud.com/pastebin/sLmJgu2f/
[22:04:32] <RhinosF1>	 That means?
[22:05:34] <paladox>	 that's ram usage for php-fpm
[22:05:54] <RhinosF1>	 Ok
[22:06:15] <RhinosF1>	 Looking at the graphs they always use the max
[22:06:24] <RhinosF1>	 Something just tipped it very slightly over
[22:06:37] <RhinosF1>	 So OOM killer decided to say hi
[22:06:55] <RhinosF1>	 I mean making it restart properly would also be nice
[22:07:04] <RhinosF1>	 Because then it becomes less of an issue
[22:07:29] <RhinosF1>	 (Yes it oom'ing is bad but it can start up in seconds so we'd minimise impact)
[22:08:05] <RhinosF1>	 It doesn't look like it even attempted to start
[22:08:08] <paladox>	 php-fpm is using 2.9g of ram
[22:08:09] <paladox>	 :wow:
[22:08:27] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.59, 3.37
[22:08:36] <JohnLewis>	 Is 7.3 available on bullseye or is it only 7.4?
[22:09:33] <RhinosF1>	 JohnLewis: on the Debian official repos just 7.4
[22:11:54] <JohnLewis>	 Something has to be causing a memory leak in 7.4 only
[22:12:08] <JohnLewis>	 It shouldn’t OOM thing quickly
[22:12:33] <RhinosF1>	 I guessed it's something changed in 7.4 a while back
[22:12:38] <RhinosF1>	 What is my issue
[22:12:42] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 3.04, 3.21
[22:12:44] <RhinosF1>	 Then how to mitigate the impact
[22:14:03] <JohnLewis>	 If only a traffic serving server OOM, it makes me thing it’s not core but an extension that’s the cause potentially, one not being actively used on test3
[22:14:29] <JohnLewis>	 Otherwise, if it is core - test3 hasn’t been thoroughly tested on bullseye before the decision to push this into prod was made
[22:17:09] <RhinosF1>	 JohnLewis: the fact it takes 30 minutes would probably indicate it's not something commonly used
[22:17:39] <RhinosF1>	 Or like you say we'd have seen it quicker
[22:17:46] <RhinosF1>	 But I have to sleep
[22:25:02] <JohnLewis>	 What are the errors for https://phabricator.miraheze.org/T8029? Seems obstructive to not transfer them to the bug report if they’re not security related
[22:25:03] <MirahezeBot>	 [url] ⚓ T8029 jacknjellifysbfdiwiki resetWiki caches gives various errors | phabricator.miraheze.org
[22:43:43] <MH-Discord>	 <el komodos drago> Hi, when I log in every time I navigate to a new page I get logged out again.
[22:43:49] <MH-Discord>	 <Agent> Same ^
[22:43:56] <MH-Discord>	 <Agent> Have been having this issue for about a day or two
[22:44:12] <MH-Discord>	 <el komodos drago> It has started for me over the course of today
[22:44:19] <MH-Discord>	 <Ryjack> Me too
[22:44:47] <MH-Discord>	 <el komodos drago> I think when it started I would get a few pages before being logged out but I could be mistaken.
[22:44:52] <MH-Discord>	 <Agent> At least 2 users have also reported similar errors where they either can't login or are logged out frequently
[22:45:01] <MH-Discord>	 <Agent> I get logged out every hour or so
[22:45:48] <MH-Discord>	 <el komodos drago> Now I am getting this message:
[22:45:49] <MH-Discord>	 <el komodos drago> https://cdn.discordapp.com/attachments/808001911868489748/887106820492132382/unknown.png
[22:46:19] <MH-Discord>	 <el komodos drago> Also I keep getting the cookie banner:
[22:46:20] <MH-Discord>	 <el komodos drago> https://cdn.discordapp.com/attachments/808001911868489748/887106949404057630/unknown.png
[22:46:44] <MH-Discord>	 <el komodos drago> I click 'OK' but it just reappears.
[22:46:47] <MH-Discord>	 <Agent> indeed, other users have experienced that error lately ^
[22:48:02] <MH-Discord>	 <el komodos drago> I am going to sleep shortly so I can't respond to any requests for further information until the morning.
[23:02:35] <paladox>	 !log depool mw8 - as php ooms
[23:02:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:06:13] <icinga-miraheze>	 PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8
[23:06:33] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8
[23:06:34] <icinga-miraheze>	 PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8
[23:07:25] <icinga-miraheze>	 PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8
[23:08:03] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.14, 6.59, 5.60
[23:09:12] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.36, 7.02, 6.02
[23:10:11] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.68, 7.27, 5.98
[23:12:17] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.68, 6.85, 6.01
[23:13:27] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.33, 7.27, 6.46
[23:14:26] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.21, 6.34, 5.94
[23:15:30] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.67, 6.17, 6.15
[23:19:55] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.66, 6.63, 6.12
[23:22:12] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.50, 8.26, 6.82
[23:22:41] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.35, 7.75, 6.71
[23:23:33] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 12.51, 9.29, 7.29
[23:55:45] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.78, 7.47, 7.94
[23:57:35] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.47, 3.25
[23:57:51] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.45, 7.77, 7.99
[23:59:52] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.26, 3.27, 3.20