[00:02:16] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:03:27] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:05:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.16, 9.08, 7.27 [00:05:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.78, 7.65, 6.61 [00:07:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.86, 7.27, 6.60 [00:09:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.98, 7.54, 7.06 [00:09:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.61, 6.30, 6.32 [00:10:48] [02mediawiki] 07dependabot[bot] opened pull request 03#1925: Bump extensions/PageForms from `ba1db1a` to `81fa60c` - 13https://git.io/JsF9y [00:10:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/PageForms-81fa60c [+0/-0/±1] 13https://git.io/JsF9S [00:10:51] [02miraheze/mediawiki] 07dependabot[bot] 0393dbee8 - Bump extensions/PageForms from `ba1db1a` to `81fa60c` [00:10:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#1925: Bump extensions/PageForms from `ba1db1a` to `81fa60c` - 13https://git.io/JsF9y [00:10:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#1925: Bump extensions/PageForms from `ba1db1a` to `81fa60c` - 13https://git.io/JsF9y [00:10:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/PageForms-81fa60c - 13https://git.io/vbL5b [00:10:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/Citizen-321f9a7 [+0/-0/±1] 13https://git.io/JsF99 [00:10:59] [02miraheze/mediawiki] 07dependabot[bot] 0350c44ef - Bump skins/Citizen from `b698a20` to `321f9a7` [00:11:00] [02mediawiki] 07dependabot[bot] opened pull request 03#1926: Bump skins/Citizen from `b698a20` to `321f9a7` - 13https://git.io/JsF9H [00:11:02] [02mediawiki] 07dependabot[bot] commented on pull request 03#1874: Bump extensions/PageForms from `ba1db1a` to `7dcf4d4` - 13https://git.io/JsF9Q [00:11:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#1926: Bump skins/Citizen from `b698a20` to `321f9a7` - 13https://git.io/JsF9H [00:11:05] [02mediawiki] 07dependabot[bot] labeled pull request 03#1926: Bump skins/Citizen from `b698a20` to `321f9a7` - 13https://git.io/JsF9H [00:11:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/Citizen-321f9a7 - 13https://git.io/vbL5b [00:11:08] [02mediawiki] 07dependabot[bot] closed pull request 03#1874: Bump extensions/PageForms from `ba1db1a` to `7dcf4d4` - 13https://git.io/Js4RK [00:11:09] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/PageForms-7dcf4d4 - 13https://git.io/vbL5b [00:11:11] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/PageForms-7dcf4d4 [00:11:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/AJAXPoll-b36f7c1 [+0/-0/±1] 13https://git.io/JsF97 [00:11:14] [02miraheze/mediawiki] 07dependabot[bot] 035083a6d - Bump extensions/AJAXPoll from `3830cb4` to `b36f7c1` [00:11:15] [02mediawiki] 07dependabot[bot] opened pull request 03#1927: Bump extensions/AJAXPoll from `3830cb4` to `b36f7c1` - 13https://git.io/JsF95 [00:11:17] [02mediawiki] 07dependabot[bot] commented on pull request 03#1906: Bump skins/Citizen from `2653336` to `11ee87c` - 13https://git.io/JsF9d [00:11:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/AJAXPoll-b36f7c1 - 13https://git.io/vbL5b [00:11:20] [02mediawiki] 07dependabot[bot] labeled pull request 03#1927: Bump extensions/AJAXPoll from `3830cb4` to `b36f7c1` - 13https://git.io/JsF95 [00:11:21] [02mediawiki] 07dependabot[bot] labeled pull request 03#1927: Bump extensions/AJAXPoll from `3830cb4` to `b36f7c1` - 13https://git.io/JsF95 [00:11:23] [02mediawiki] 07dependabot[bot] closed pull request 03#1906: Bump skins/Citizen from `2653336` to `11ee87c` - 13https://git.io/Js40Q [00:11:24] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Citizen-11ee87c - 13https://git.io/vbL5b [00:11:26] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Citizen-11ee87c [00:11:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#1865: Bump extensions/AJAXPoll from `3830cb4` to `137a43f` - 13https://git.io/JsF9F [00:11:29] [02mediawiki] 07dependabot[bot] closed pull request 03#1865: Bump extensions/AJAXPoll from `3830cb4` to `137a43f` - 13https://git.io/Js4RG [00:11:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/MediaWikiChat-d80506a [+0/-0/±1] 13https://git.io/JsF9b [00:11:32] [02miraheze/mediawiki] 07dependabot[bot] 03ce89954 - Bump extensions/MediaWikiChat from `ca65d0d` to `d80506a` [00:11:33] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/AJAXPoll-137a43f - 13https://git.io/vbL5b [00:11:35] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/AJAXPoll-137a43f [00:11:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/MediaWikiChat-d80506a - 13https://git.io/vbL5b [00:11:38] [02mediawiki] 07dependabot[bot] opened pull request 03#1928: Bump extensions/MediaWikiChat from `ca65d0d` to `d80506a` - 13https://git.io/JsF9N [00:11:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#1928: Bump extensions/MediaWikiChat from `ca65d0d` to `d80506a` - 13https://git.io/JsF9N [00:11:41] [02mediawiki] 07dependabot[bot] labeled pull request 03#1928: Bump extensions/MediaWikiChat from `ca65d0d` to `d80506a` - 13https://git.io/JsF9N [00:11:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/PollNY-f82cb41 [+0/-0/±1] 13https://git.io/JsF9A [00:11:44] [02miraheze/mediawiki] 07dependabot[bot] 0345cba70 - Bump extensions/PollNY from `4d7b0e5` to `f82cb41` [00:11:45] [02mediawiki] 07dependabot[bot] opened pull request 03#1929: Bump extensions/PollNY from `4d7b0e5` to `f82cb41` - 13https://git.io/JsF9x [00:11:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/PollNY-f82cb41 - 13https://git.io/vbL5b [00:11:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#1929: Bump extensions/PollNY from `4d7b0e5` to `f82cb41` - 13https://git.io/JsF9x [00:11:50] [02mediawiki] 07dependabot[bot] labeled pull request 03#1929: Bump extensions/PollNY from `4d7b0e5` to `f82cb41` - 13https://git.io/JsF9x [00:11:51] [02mediawiki] 07dependabot[bot] opened pull request 03#1930: Bump extensions/GlobalUsage from `094acd3` to `19974c6` - 13https://git.io/JsF9h [00:11:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#1930: Bump extensions/GlobalUsage from `094acd3` to `19974c6` - 13https://git.io/JsF9h [00:11:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#1930: Bump extensions/GlobalUsage from `094acd3` to `19974c6` - 13https://git.io/JsF9h [00:11:56] [02mediawiki] 07dependabot[bot] commented on pull request 03#1852: Bump extensions/GlobalUsage from `094acd3` to `4c52edd` - 13https://git.io/JsF9j [00:11:57] [02mediawiki] 07dependabot[bot] closed pull request 03#1852: Bump extensions/GlobalUsage from `094acd3` to `4c52edd` - 13https://git.io/Js4B9 [00:11:59] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/GlobalUsage-4c52edd - 13https://git.io/vbL5b [00:12:00] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/GlobalUsage-4c52edd [00:12:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-9c81e9b [+0/-0/±1] 13https://git.io/JsFHe [00:12:03] [02miraheze/mediawiki] 07dependabot[bot] 036743613 - Bump extensions/SnapProjectEmbed from `cf18475` to `9c81e9b` [00:12:05] [02mediawiki] 07dependabot[bot] opened pull request 03#1931: Bump extensions/SnapProjectEmbed from `cf18475` to `9c81e9b` - 13https://git.io/JsFHv [00:12:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-9c81e9b - 13https://git.io/vbL5b [00:12:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#1931: Bump extensions/SnapProjectEmbed from `cf18475` to `9c81e9b` - 13https://git.io/JsFHv [00:12:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#1931: Bump extensions/SnapProjectEmbed from `cf18475` to `9c81e9b` - 13https://git.io/JsFHv [00:12:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/IncidentReporting-d03aa59 [+0/-0/±1] 13https://git.io/JsFHf [00:12:12] [02miraheze/mediawiki] 07dependabot[bot] 03a2f7958 - Bump extensions/IncidentReporting from `85c5618` to `d03aa59` [00:12:14] [02mediawiki] 07dependabot[bot] opened pull request 03#1932: Bump extensions/IncidentReporting from `85c5618` to `d03aa59` - 13https://git.io/JsFHJ [00:12:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/IncidentReporting-d03aa59 - 13https://git.io/vbL5b [00:12:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#1932: Bump extensions/IncidentReporting from `85c5618` to `d03aa59` - 13https://git.io/JsFHJ [00:12:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#1932: Bump extensions/IncidentReporting from `85c5618` to `d03aa59` - 13https://git.io/JsFHJ [00:12:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/YouTube-80b7c65 [+0/-0/±1] 13https://git.io/JsFHU [00:12:21] [02miraheze/mediawiki] 07dependabot[bot] 033624969 - Bump extensions/YouTube from `0941937` to `80b7c65` [00:12:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/YouTube-80b7c65 - 13https://git.io/vbL5b [00:12:24] [02mediawiki] 07dependabot[bot] opened pull request 03#1933: Bump extensions/YouTube from `0941937` to `80b7c65` - 13https://git.io/JsFHT [00:12:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#1933: Bump extensions/YouTube from `0941937` to `80b7c65` - 13https://git.io/JsFHT [00:12:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#1933: Bump extensions/YouTube from `0941937` to `80b7c65` - 13https://git.io/JsFHT [00:12:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/Truglass-d48ad91 [+0/-0/±1] 13https://git.io/JsFHk [00:12:30] [02miraheze/mediawiki] 07dependabot[bot] 03126d802 - Bump skins/Truglass from `cc8fe11` to `d48ad91` [00:12:32] [02mediawiki] 07dependabot[bot] opened pull request 03#1934: Bump skins/Truglass from `cc8fe11` to `d48ad91` - 13https://git.io/JsFHI [00:12:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/Truglass-d48ad91 - 13https://git.io/vbL5b [00:12:35] [02mediawiki] 07dependabot[bot] labeled pull request 03#1934: Bump skins/Truglass from `cc8fe11` to `d48ad91` - 13https://git.io/JsFHI [00:12:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#1934: Bump skins/Truglass from `cc8fe11` to `d48ad91` - 13https://git.io/JsFHI [00:12:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Report-0a98649 [+0/-0/±1] 13https://git.io/JsFHt [00:12:39] [02miraheze/mediawiki] 07dependabot[bot] 034c2d82f - Bump extensions/Report from `f828dc6` to `0a98649` [00:12:41] [02mediawiki] 07dependabot[bot] opened pull request 03#1935: Bump extensions/Report from `f828dc6` to `0a98649` - 13https://git.io/JsFHq [00:12:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Report-0a98649 - 13https://git.io/vbL5b [00:12:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#1935: Bump extensions/Report from `f828dc6` to `0a98649` - 13https://git.io/JsFHq [00:12:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#1935: Bump extensions/Report from `f828dc6` to `0a98649` - 13https://git.io/JsFHq [00:12:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Video-7675eae [+0/-0/±1] 13https://git.io/JsFHm [00:12:48] [02miraheze/mediawiki] 07dependabot[bot] 03fa369d9 - Bump extensions/Video from `28a3af3` to `7675eae` [00:12:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Video-7675eae - 13https://git.io/vbL5b [00:12:51] [02mediawiki] 07dependabot[bot] opened pull request 03#1936: Bump extensions/Video from `28a3af3` to `7675eae` - 13https://git.io/JsFHY [00:12:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#1936: Bump extensions/Video from `28a3af3` to `7675eae` - 13https://git.io/JsFHY [00:12:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#1936: Bump extensions/Video from `28a3af3` to `7675eae` - 13https://git.io/JsFHY [00:12:56] [02mediawiki] 07dependabot[bot] commented on pull request 03#1881: Bump extensions/Video from `28a3af3` to `b41fdf1` - 13https://git.io/JsFHO [00:12:57] [02mediawiki] 07dependabot[bot] closed pull request 03#1881: Bump extensions/Video from `28a3af3` to `b41fdf1` - 13https://git.io/Js4Rb [00:12:59] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Video-b41fdf1 [00:13:00] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Video-b41fdf1 - 13https://git.io/vbL5b [00:13:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Cargo-0ccd967 [+0/-0/±1] 13https://git.io/JsFH3 [00:13:03] [02miraheze/mediawiki] 07dependabot[bot] 03f24dd16 - Bump extensions/Cargo from `8fc3204` to `0ccd967` [00:13:05] [02mediawiki] 07dependabot[bot] opened pull request 03#1937: Bump extensions/Cargo from `8fc3204` to `0ccd967` - 13https://git.io/JsFHs [00:13:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Cargo-0ccd967 - 13https://git.io/vbL5b [00:13:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#1937: Bump extensions/Cargo from `8fc3204` to `0ccd967` - 13https://git.io/JsFHs [00:13:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#1937: Bump extensions/Cargo from `8fc3204` to `0ccd967` - 13https://git.io/JsFHs [00:13:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/SpriteSheet-cdb73e7 [+0/-0/±1] 13https://git.io/JsFHG [00:13:12] [02miraheze/mediawiki] 07dependabot[bot] 03507520d - Bump extensions/SpriteSheet from `594d6da` to `cdb73e7` [00:13:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/SpriteSheet-cdb73e7 - 13https://git.io/vbL5b [00:13:15] [02mediawiki] 07dependabot[bot] opened pull request 03#1938: Bump extensions/SpriteSheet from `594d6da` to `cdb73e7` - 13https://git.io/JsFHZ [00:13:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#1938: Bump extensions/SpriteSheet from `594d6da` to `cdb73e7` - 13https://git.io/JsFHZ [00:13:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#1938: Bump extensions/SpriteSheet from `594d6da` to `cdb73e7` - 13https://git.io/JsFHZ [00:13:20] [02mediawiki] 07dependabot[bot] commented on pull request 03#1899: Bump extensions/SpriteSheet from `594d6da` to `c42e3a1` - 13https://git.io/JsFHn [00:13:21] [02mediawiki] 07dependabot[bot] closed pull request 03#1899: Bump extensions/SpriteSheet from `594d6da` to `c42e3a1` - 13https://git.io/Js40V [00:13:23] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/SpriteSheet-c42e3a1 [00:13:24] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/SpriteSheet-c42e3a1 - 13https://git.io/vbL5b [00:13:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/CheckUser-025d552 [+0/-0/±1] 13https://git.io/JsFHc [00:13:27] [02miraheze/mediawiki] 07dependabot[bot] 030497f01 - Bump extensions/CheckUser from `528b959` to `025d552` [00:13:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/CheckUser-025d552 - 13https://git.io/vbL5b [00:13:30] [02mediawiki] 07dependabot[bot] opened pull request 03#1939: Bump extensions/CheckUser from `528b959` to `025d552` - 13https://git.io/JsFHC [00:13:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#1939: Bump extensions/CheckUser from `528b959` to `025d552` - 13https://git.io/JsFHC [00:13:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#1939: Bump extensions/CheckUser from `528b959` to `025d552` - 13https://git.io/JsFHC [00:13:35] [02mediawiki] 07dependabot[bot] commented on pull request 03#1859: Bump extensions/CheckUser from `528b959` to `12bc076` - 13https://git.io/JsFHW [00:13:36] [02mediawiki] 07dependabot[bot] closed pull request 03#1859: Bump extensions/CheckUser from `528b959` to `12bc076` - 13https://git.io/Js4Rf [00:13:38] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/CheckUser-12bc076 - 13https://git.io/vbL5b [00:13:39] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/CheckUser-12bc076 [00:13:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/QuizGame-fbc2ecb [+0/-0/±1] 13https://git.io/JsFHl [00:13:42] [02miraheze/mediawiki] 07dependabot[bot] 031f1f4b0 - Bump extensions/QuizGame from `1a9fbc3` to `fbc2ecb` [00:13:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/QuizGame-fbc2ecb - 13https://git.io/vbL5b [00:13:45] [02mediawiki] 07dependabot[bot] opened pull request 03#1940: Bump extensions/QuizGame from `1a9fbc3` to `fbc2ecb` - 13https://git.io/JsFH8 [00:13:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#1940: Bump extensions/QuizGame from `1a9fbc3` to `fbc2ecb` - 13https://git.io/JsFH8 [00:13:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#1940: Bump extensions/QuizGame from `1a9fbc3` to `fbc2ecb` - 13https://git.io/JsFH8 [00:13:50] [02mediawiki] 07dependabot[bot] commented on pull request 03#1913: Bump extensions/QuizGame from `1a9fbc3` to `5ca61d5` - 13https://git.io/JsFH4 [00:13:51] [02mediawiki] 07dependabot[bot] closed pull request 03#1913: Bump extensions/QuizGame from `1a9fbc3` to `5ca61d5` - 13https://git.io/Js4EI [00:13:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/Cosmos-60d2e29 [+0/-0/±1] 13https://git.io/JsFHB [00:13:54] [02miraheze/mediawiki] 07dependabot[bot] 03efcef45 - Bump skins/Cosmos from `e8a77e6` to `60d2e29` [00:13:56] [02mediawiki] 07dependabot[bot] opened pull request 03#1941: Bump skins/Cosmos from `e8a77e6` to `60d2e29` - 13https://git.io/JsFHR [00:13:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/QuizGame-5ca61d5 - 13https://git.io/vbL5b [00:13:59] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/QuizGame-5ca61d5 [00:14:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/Cosmos-60d2e29 - 13https://git.io/vbL5b [00:14:02] [02mediawiki] 07dependabot[bot] labeled pull request 03#1941: Bump skins/Cosmos from `e8a77e6` to `60d2e29` - 13https://git.io/JsFHR [00:14:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#1941: Bump skins/Cosmos from `e8a77e6` to `60d2e29` - 13https://git.io/JsFHR [00:14:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/VoteNY-8e81fcb [+0/-0/±1] 13https://git.io/JsFH0 [00:14:06] [02miraheze/mediawiki] 07dependabot[bot] 03bdfa1ec - Bump extensions/VoteNY from `c7a829a` to `8e81fcb` [00:14:08] [02mediawiki] 07dependabot[bot] opened pull request 03#1942: Bump extensions/VoteNY from `c7a829a` to `8e81fcb` - 13https://git.io/JsFHE [00:14:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/VoteNY-8e81fcb - 13https://git.io/vbL5b [00:14:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/GlobalUsage-19974c6 [+0/-0/±1] 13https://git.io/JsFHu [00:14:12] [02miraheze/mediawiki] 07dependabot[bot] 033cf501f - Bump extensions/GlobalUsage from `094acd3` to `19974c6` [00:14:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/GlobalUsage-19974c6 - 13https://git.io/vbL5b [00:14:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#1942: Bump extensions/VoteNY from `c7a829a` to `8e81fcb` - 13https://git.io/JsFHE [00:14:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#1942: Bump extensions/VoteNY from `c7a829a` to `8e81fcb` - 13https://git.io/JsFHE [00:14:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/PDFEmbed-22ea240 [+0/-0/±1] 13https://git.io/JsFHz [00:14:20] [02miraheze/mediawiki] 07dependabot[bot] 03883eff0 - Bump extensions/PDFEmbed from `5c1efbf` to `22ea240` [00:14:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/PDFEmbed-22ea240 - 13https://git.io/vbL5b [00:14:23] [02mediawiki] 07dependabot[bot] opened pull request 03#1943: Bump extensions/PDFEmbed from `5c1efbf` to `22ea240` - 13https://git.io/JsFHg [00:14:24] [02mediawiki] 07dependabot[bot] labeled pull request 03#1943: Bump extensions/PDFEmbed from `5c1efbf` to `22ea240` - 13https://git.io/JsFHg [00:14:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#1943: Bump extensions/PDFEmbed from `5c1efbf` to `22ea240` - 13https://git.io/JsFHg [00:14:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/WikiDiscover-d88811a [+0/-0/±1] 13https://git.io/JsFHa [00:14:29] [02miraheze/mediawiki] 07dependabot[bot] 033bcb78d - Bump extensions/WikiDiscover from `2be49d8` to `d88811a` [00:14:30] [02mediawiki] 07dependabot[bot] opened pull request 03#1944: Bump extensions/WikiDiscover from `2be49d8` to `d88811a` - 13https://git.io/JsFHV [00:14:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/WikiDiscover-d88811a - 13https://git.io/vbL5b [00:14:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#1944: Bump extensions/WikiDiscover from `2be49d8` to `d88811a` - 13https://git.io/JsFHV [00:14:35] [02mediawiki] 07dependabot[bot] labeled pull request 03#1944: Bump extensions/WikiDiscover from `2be49d8` to `d88811a` - 13https://git.io/JsFHV [00:14:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/RottenLinks-41e13dd [+0/-0/±1] 13https://git.io/JsFHr [00:14:38] [02miraheze/mediawiki] 07dependabot[bot] 03ace49d6 - Bump extensions/RottenLinks from `c5a8212` to `41e13dd` [00:14:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/RottenLinks-41e13dd - 13https://git.io/vbL5b [00:14:41] [02mediawiki] 07dependabot[bot] opened pull request 03#1945: Bump extensions/RottenLinks from `c5a8212` to `41e13dd` - 13https://git.io/JsFHo [00:14:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#1945: Bump extensions/RottenLinks from `c5a8212` to `41e13dd` - 13https://git.io/JsFHo [00:14:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#1945: Bump extensions/RottenLinks from `c5a8212` to `41e13dd` - 13https://git.io/JsFHo [00:14:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/MatomoAnalytics-b59c519 [+0/-0/±1] 13https://git.io/JsFHK [00:14:47] [02miraheze/mediawiki] 07dependabot[bot] 03e27c1ba - Bump extensions/MatomoAnalytics from `1d6d1f2` to `b59c519` [00:14:48] [02mediawiki] 07dependabot[bot] opened pull request 03#1946: Bump extensions/MatomoAnalytics from `1d6d1f2` to `b59c519` - 13https://git.io/JsFH6 [00:14:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/MatomoAnalytics-b59c519 - 13https://git.io/vbL5b [00:14:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#1946: Bump extensions/MatomoAnalytics from `1d6d1f2` to `b59c519` - 13https://git.io/JsFH6 [00:14:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#1946: Bump extensions/MatomoAnalytics from `1d6d1f2` to `b59c519` - 13https://git.io/JsFH6 [00:14:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/GlobalNewFiles-e5e58cc [+0/-0/±1] 13https://git.io/JsFHi [00:14:56] [02miraheze/mediawiki] 07dependabot[bot] 03576d6e7 - Bump extensions/GlobalNewFiles from `d831a1a` to `e5e58cc` [00:14:57] [02mediawiki] 07dependabot[bot] opened pull request 03#1947: Bump extensions/GlobalNewFiles from `d831a1a` to `e5e58cc` - 13https://git.io/JsFHP [00:14:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/GlobalNewFiles-e5e58cc - 13https://git.io/vbL5b [00:15:00] [02mediawiki] 07dependabot[bot] labeled pull request 03#1947: Bump extensions/GlobalNewFiles from `d831a1a` to `e5e58cc` - 13https://git.io/JsFHP [00:15:02] [02mediawiki] 07dependabot[bot] labeled pull request 03#1947: Bump extensions/GlobalNewFiles from `d831a1a` to `e5e58cc` - 13https://git.io/JsFHP [00:15:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.23, 6.13, 6.67 [00:15:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/CodeMirror-84846ec [+0/-0/±1] 13https://git.io/JsFHX [00:15:05] [02miraheze/mediawiki] 07dependabot[bot] 03610d838 - Bump extensions/CodeMirror from `a4ebaae` to `84846ec` [00:15:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/CodeMirror-84846ec - 13https://git.io/vbL5b [00:15:08] [02mediawiki] 07dependabot[bot] opened pull request 03#1948: Bump extensions/CodeMirror from `a4ebaae` to `84846ec` - 13https://git.io/JsFH1 [00:15:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#1948: Bump extensions/CodeMirror from `a4ebaae` to `84846ec` - 13https://git.io/JsFH1 [00:15:11] [02mediawiki] 07dependabot[bot] labeled pull request 03#1948: Bump extensions/CodeMirror from `a4ebaae` to `84846ec` - 13https://git.io/JsFH1 [00:15:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ReplaceText-7c4fbd1 [+0/-0/±1] 13https://git.io/JsFHM [00:15:14] [02miraheze/mediawiki] 07dependabot[bot] 037f8c6e8 - Bump extensions/ReplaceText from `feb8a9c` to `7c4fbd1` [00:15:15] [02mediawiki] 07dependabot[bot] opened pull request 03#1949: Bump extensions/ReplaceText from `feb8a9c` to `7c4fbd1` - 13https://git.io/JsFHD [00:15:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ReplaceText-7c4fbd1 - 13https://git.io/vbL5b [00:15:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#1949: Bump extensions/ReplaceText from `feb8a9c` to `7c4fbd1` - 13https://git.io/JsFHD [00:15:20] [02mediawiki] 07dependabot[bot] labeled pull request 03#1949: Bump extensions/ReplaceText from `feb8a9c` to `7c4fbd1` - 13https://git.io/JsFHD [00:15:21] [02mediawiki] 07dependabot[bot] commented on pull request 03#1864: Bump extensions/ReplaceText from `feb8a9c` to `f7f0ddb` - 13https://git.io/JsFHy [00:15:23] [02mediawiki] 07dependabot[bot] closed pull request 03#1864: Bump extensions/ReplaceText from `feb8a9c` to `f7f0ddb` - 13https://git.io/Js4R3 [00:15:24] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/ReplaceText-f7f0ddb [00:15:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/RemovePII-8eb89df [+0/-0/±1] 13https://git.io/JsFHS [00:15:27] [02miraheze/mediawiki] 07dependabot[bot] 03cc4e291 - Bump extensions/RemovePII from `738036f` to `8eb89df` [00:15:29] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/ReplaceText-f7f0ddb - 13https://git.io/vbL5b [00:15:30] [02mediawiki] 07dependabot[bot] opened pull request 03#1950: Bump extensions/RemovePII from `738036f` to `8eb89df` - 13https://git.io/JsFH9 [00:15:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/RemovePII-8eb89df - 13https://git.io/vbL5b [00:15:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#1950: Bump extensions/RemovePII from `738036f` to `8eb89df` - 13https://git.io/JsFH9 [00:15:35] [02mediawiki] 07dependabot[bot] labeled pull request 03#1950: Bump extensions/RemovePII from `738036f` to `8eb89df` - 13https://git.io/JsFH9 [00:15:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/AutoCreateCategoryPages-905e85a [+0/-0/±1] 13https://git.io/JsFHH [00:15:38] [02miraheze/mediawiki] 07dependabot[bot] 03c25e786 - Bump extensions/AutoCreateCategoryPages from `463effd` to `905e85a` [00:15:39] [02mediawiki] 07dependabot[bot] opened pull request 03#1951: Bump extensions/AutoCreateCategoryPages from `463effd` to `905e85a` - 13https://git.io/JsFH7 [00:15:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/AutoCreateCategoryPages-905e85a - 13https://git.io/vbL5b [00:15:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#1951: Bump extensions/AutoCreateCategoryPages from `463effd` to `905e85a` - 13https://git.io/JsFH7 [00:15:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#1951: Bump extensions/AutoCreateCategoryPages from `463effd` to `905e85a` - 13https://git.io/JsFH7 [00:15:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/cldr-3eadcc4 [+0/-0/±1] 13https://git.io/JsFHd [00:15:47] [02miraheze/mediawiki] 07dependabot[bot] 0373bb4b6 - Bump extensions/cldr from `4b52abe` to `3eadcc4` [00:15:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/cldr-3eadcc4 - 13https://git.io/vbL5b [00:15:49] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [00:15:50] [02mediawiki] 07dependabot[bot] opened pull request 03#1952: Bump extensions/cldr from `4b52abe` to `3eadcc4` - 13https://git.io/JsFHF [00:15:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#1952: Bump extensions/cldr from `4b52abe` to `3eadcc4` - 13https://git.io/JsFHF [00:15:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#1952: Bump extensions/cldr from `4b52abe` to `3eadcc4` - 13https://git.io/JsFHF [00:15:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-3e3edf0 [+0/-0/±1] 13https://git.io/JsFHb [00:15:56] [02miraheze/mediawiki] 07dependabot[bot] 03973a565 - Bump extensions/ApprovedRevs from `69f6dc7` to `3e3edf0` [00:15:57] [02mediawiki] 07dependabot[bot] opened pull request 03#1953: Bump extensions/ApprovedRevs from `69f6dc7` to `3e3edf0` - 13https://git.io/JsFHN [00:15:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-3e3edf0 - 13https://git.io/vbL5b [00:16:00] [02mediawiki] 07dependabot[bot] labeled pull request 03#1953: Bump extensions/ApprovedRevs from `69f6dc7` to `3e3edf0` - 13https://git.io/JsFHN [00:16:02] [02mediawiki] 07dependabot[bot] labeled pull request 03#1953: Bump extensions/ApprovedRevs from `69f6dc7` to `3e3edf0` - 13https://git.io/JsFHN [00:16:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/CommentStreams-4686df9 [+0/-0/±1] 13https://git.io/JsFHA [00:16:05] [02miraheze/mediawiki] 07dependabot[bot] 03f5b68df - Bump extensions/CommentStreams from `ed3fc73` to `4686df9` [00:16:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/CommentStreams-4686df9 - 13https://git.io/vbL5b [00:16:08] [02mediawiki] 07dependabot[bot] opened pull request 03#1954: Bump extensions/CommentStreams from `ed3fc73` to `4686df9` - 13https://git.io/JsFHx [00:16:09] [02mediawiki] 07dependabot[bot] commented on pull request 03#1905: Bump extensions/ApprovedRevs from `69f6dc7` to `c01d76a` - 13https://git.io/JsFHp [00:16:11] [02mediawiki] 07dependabot[bot] labeled pull request 03#1954: Bump extensions/CommentStreams from `ed3fc73` to `4686df9` - 13https://git.io/JsFHx [00:16:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#1954: Bump extensions/CommentStreams from `ed3fc73` to `4686df9` - 13https://git.io/JsFHx [00:16:14] [02mediawiki] 07dependabot[bot] closed pull request 03#1905: Bump extensions/ApprovedRevs from `69f6dc7` to `c01d76a` - 13https://git.io/Js409 [00:16:15] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-c01d76a - 13https://git.io/vbL5b [00:16:17] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-c01d76a [00:16:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/GettingStarted-4a65130 [+0/-0/±1] 13https://git.io/JsFHh [00:16:20] [02miraheze/mediawiki] 07dependabot[bot] 03e742a25 - Bump extensions/GettingStarted from `5ec1c75` to `4a65130` [00:16:21] [02mediawiki] 07dependabot[bot] commented on pull request 03#1872: Bump extensions/CommentStreams from `ed3fc73` to `52be0c2` - 13https://git.io/JsFHj [00:16:23] [02mediawiki] 07dependabot[bot] opened pull request 03#1955: Bump extensions/GettingStarted from `5ec1c75` to `4a65130` - 13https://git.io/JsFQe [00:16:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/GettingStarted-4a65130 - 13https://git.io/vbL5b [00:16:26] [02mediawiki] 07dependabot[bot] closed pull request 03#1872: Bump extensions/CommentStreams from `ed3fc73` to `52be0c2` - 13https://git.io/Js4Ra [00:16:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#1955: Bump extensions/GettingStarted from `5ec1c75` to `4a65130` - 13https://git.io/JsFQe [00:16:29] [02mediawiki] 07dependabot[bot] labeled pull request 03#1955: Bump extensions/GettingStarted from `5ec1c75` to `4a65130` - 13https://git.io/JsFQe [00:16:30] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/CommentStreams-52be0c2 [00:16:32] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/CommentStreams-52be0c2 - 13https://git.io/vbL5b [00:16:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/BlogPage-33af4de [+0/-0/±1] 13https://git.io/JsFQf [00:16:35] [02miraheze/mediawiki] 07dependabot[bot] 03daac5b9 - Bump extensions/BlogPage from `1672d46` to `33af4de` [00:16:36] [02mediawiki] 07dependabot[bot] opened pull request 03#1956: Bump extensions/BlogPage from `1672d46` to `33af4de` - 13https://git.io/JsFQJ [00:16:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/BlogPage-33af4de - 13https://git.io/vbL5b [00:16:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#1956: Bump extensions/BlogPage from `1672d46` to `33af4de` - 13https://git.io/JsFQJ [00:16:41] [02mediawiki] 07dependabot[bot] labeled pull request 03#1956: Bump extensions/BlogPage from `1672d46` to `33af4de` - 13https://git.io/JsFQJ [00:16:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/SocialProfile-f70d79e [+0/-0/±1] 13https://git.io/JsFQU [00:16:44] [02miraheze/mediawiki] 07dependabot[bot] 035cf411e - Bump extensions/SocialProfile from `b258833` to `f70d79e` [00:16:45] [02mediawiki] 07dependabot[bot] opened pull request 03#1957: Bump extensions/SocialProfile from `b258833` to `f70d79e` - 13https://git.io/JsFQT [00:16:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-f70d79e - 13https://git.io/vbL5b [00:16:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#1957: Bump extensions/SocialProfile from `b258833` to `f70d79e` - 13https://git.io/JsFQT [00:16:50] [02mediawiki] 07dependabot[bot] labeled pull request 03#1957: Bump extensions/SocialProfile from `b258833` to `f70d79e` - 13https://git.io/JsFQT [00:16:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ArticleToCategory2-ce3b839 [+0/-0/±1] 13https://git.io/JsFQk [00:16:53] [02miraheze/mediawiki] 07dependabot[bot] 03f1cc2f8 - Bump extensions/ArticleToCategory2 from `9fa7338` to `ce3b839` [00:16:54] [02mediawiki] 07dependabot[bot] opened pull request 03#1958: Bump extensions/ArticleToCategory2 from `9fa7338` to `ce3b839` - 13https://git.io/JsFQI [00:16:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ArticleToCategory2-ce3b839 - 13https://git.io/vbL5b [00:16:57] [02mediawiki] 07dependabot[bot] labeled pull request 03#1958: Bump extensions/ArticleToCategory2 from `9fa7338` to `ce3b839` - 13https://git.io/JsFQI [00:16:59] [02mediawiki] 07dependabot[bot] labeled pull request 03#1958: Bump extensions/ArticleToCategory2 from `9fa7338` to `ce3b839` - 13https://git.io/JsFQI [00:17:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-282a40e [+0/-0/±1] 13https://git.io/JsFQq [00:17:02] [02miraheze/mediawiki] 07dependabot[bot] 033263dea - Bump extensions/UniversalLanguageSelector from `b556ff5` to `282a40e` [00:17:03] [02mediawiki] 07dependabot[bot] opened pull request 03#1959: Bump extensions/UniversalLanguageSelector from `b556ff5` to `282a40e` - 13https://git.io/JsFQm [00:17:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-282a40e - 13https://git.io/vbL5b [00:17:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#1959: Bump extensions/UniversalLanguageSelector from `b556ff5` to `282a40e` - 13https://git.io/JsFQm [00:17:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#1959: Bump extensions/UniversalLanguageSelector from `b556ff5` to `282a40e` - 13https://git.io/JsFQm [00:17:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Maps-eb95ae6 [+0/-0/±1] 13https://git.io/JsFQY [00:17:11] [02miraheze/mediawiki] 07dependabot[bot] 030d2a0ed - Bump extensions/Maps from `11d8882` to `eb95ae6` [00:17:12] [02mediawiki] 07dependabot[bot] opened pull request 03#1960: Bump extensions/Maps from `11d8882` to `eb95ae6` - 13https://git.io/JsFQO [00:17:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Maps-eb95ae6 - 13https://git.io/vbL5b [00:17:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#1960: Bump extensions/Maps from `11d8882` to `eb95ae6` - 13https://git.io/JsFQO [00:17:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#1960: Bump extensions/Maps from `11d8882` to `eb95ae6` - 13https://git.io/JsFQO [00:24:35] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:31:25] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:35:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.60, 6.59, 6.07 [00:37:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.53, 6.09, 5.94 [00:38:11] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [00:46:55] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [01:05:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.46, 7.03, 5.86 [01:07:46] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.40, 6.55, 5.84 [01:41:14] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [01:55:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.91, 6.23, 5.66 [01:57:01] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.18, 6.83, 5.92 [01:59:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 18.76, 10.95, 7.25 [02:00:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.01, 10.12, 7.37 [02:00:03] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [02:00:12] PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:00:18] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.02, 9.53, 6.75 [02:00:24] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is CRITICAL: CRITICAL - NGINX Error Rate is 95% [02:00:42] PROBLEM - cp13 HTTPS on cp13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:47] PROBLEM - cp12 HTTPS on cp12 is CRITICAL: HTTP CRITICAL - No data received from host [02:00:49] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Cannot make SSL connection. [02:00:54] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.76, 1.60, 1.22 [02:01:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [02:01:30] PROBLEM - wiki.scrungecraft.gg - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:01:42] PROBLEM - planetnomads.speedbomb.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:01:43] PROBLEM - icinga.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:01:47] PROBLEM - rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:01:48] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:01:48] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:01:58] PROBLEM - wiki.globasa.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:00] PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:04] PROBLEM - wiki.candela.digital - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:10] !sre 502 Bad Gateway and 500 Internal Server Error on `metawiki` [02:02:11] PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:11] PROBLEM - wiki.yumeka.icu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:12] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:16] PROBLEM - egoish.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:29] PROBLEM - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [02:02:31] PROBLEM - christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:31] PROBLEM - wiki.thedev.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:33] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [02:02:34] PROBLEM - wiki.rpointer.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:34] PROBLEM - electowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:42] PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:45] PROBLEM - erikapedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:51] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:02:55] PROBLEM - cp11 Stunnel Http for mw8 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:02:56] PROBLEM - kkutu.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:12] PROBLEM - wiki.jacksonheights.nyc - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:16] PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:03:17] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:19] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:19] PROBLEM - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:21] PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:24] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 4.85, 2.59, 1.62 [02:03:24] PROBLEM - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:28] PROBLEM - programming.red - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:28] PROBLEM - www.zenbuddhism.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:29] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:29] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:34] PROBLEM - puardian.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:37] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:38] PROBLEM - wiki.thefactoryhka.com.pa - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:42] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:46] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:49] PROBLEM - es.nocyclo.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:51] PROBLEM - wiki.usagihime.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:51] PROBLEM - www.rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:52] PROBLEM - cp10 Stunnel Http for mw8 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:03:52] PROBLEM - animatedfeet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:53] PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:57] PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:58] PROBLEM - wiki.jumpbound.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:58] PROBLEM - wiki.helioss.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:03:59] PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:02] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:08] PROBLEM - wiki.xfbs.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:09] PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:04:37] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:04:38] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:41] PROBLEM - oecumene.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:41] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:42] PROBLEM - www.wikilukas.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:44] PROBLEM - cp12 APT on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:04:44] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:56] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:57] PROBLEM - wiki.ostans.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:04:59] PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:01] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:02] PROBLEM - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:04] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:06] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:06] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:06] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:07] PROBLEM - www.fossilfighterswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:08] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:15] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:15] PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:15] PROBLEM - bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:15] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:15] PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:21] RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 9.192 second response time [02:05:25] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:26] PROBLEM - mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:05:30] RECOVERY - cp11 Stunnel Http for mw8 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 7.158 second response time [02:05:47] RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 3.355 second response time [02:06:02] PROBLEM - idleon.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:06:02] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:06:05] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20887 bytes in 3.716 second response time [02:06:09] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:06:18] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:06:18] RECOVERY - cp10 Stunnel Http for mw8 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 3.898 second response time [02:06:19] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:06:26] PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:06:26] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:06:29] PROBLEM - wikibeats.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:06:36] RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 8.684 second response time [02:07:03] RECOVERY - cp12 APT on cp12 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [02:07:11] PROBLEM - wiki.velaan.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:19] PROBLEM - petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:20] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:22] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.37, 1.69, 0.91 [02:07:23] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:25] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:29] PROBLEM - wiki.arnab.at - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:32] PROBLEM - wiki.ct777.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:32] PROBLEM - www.erikapedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:32] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:36] PROBLEM - lama.madooa.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:07:44] PROBLEM - pa.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:08:28] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:08:28] PROBLEM - miraheze.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:08:28] PROBLEM - www.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:08:33] PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:08:36] RECOVERY - wiki.scrungecraft.gg - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scrungecraft.gg' will expire on Tue 29 Jun 2021 18:43:12 GMT +0000. [02:08:36] PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:08:43] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:08:44] !sre [02:08:56] PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:09:01] RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Sat 07 Aug 2021 03:45:48 GMT +0000. [02:09:06] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:09:08] RECOVERY - wiki.globasa.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Sun 11 Jul 2021 16:41:59 GMT +0000. [02:09:20] RECOVERY - egoish.org - LetsEncrypt on sslhost is OK: OK - Certificate 'egoish.org' will expire on Wed 07 Jul 2021 14:29:21 GMT +0000. [02:09:21] PROBLEM - cp13 Stunnel Http for mw10 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:09:33] PROBLEM - cp10 Stunnel Http for mw10 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:09:42] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:09:43] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.15, 1.54, 0.97 [02:09:47] PROBLEM - matttest.mtwiki.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:09:47] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:09:48] PROBLEM - wiki.umbrel.tech - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:09:48] PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [02:10:05] PROBLEM - cp11 Stunnel Http for mw10 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:10] RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Sat 07 Aug 2021 04:56:01 GMT +0000. [02:10:18] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Sat 07 Aug 2021 14:10:59 GMT +0000. [02:10:18] RECOVERY - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.umbranet.mandelbrotgames.com' will expire on Tue 06 Jul 2021 03:47:45 GMT +0000. [02:10:20] RECOVERY - wiki.jacksonheights.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jacksonheights.nyc' will expire on Sat 07 Aug 2021 01:23:05 GMT +0000. [02:10:25] RECOVERY - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dc-multiverse.dcwikis.com' will expire on Sat 07 Aug 2021 08:12:24 GMT +0000. [02:10:34] PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:34] RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sat 07 Aug 2021 05:30:44 GMT +0000. [02:10:35] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Sat 07 Aug 2021 01:11:15 GMT +0000. [02:10:41] RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info' will expire on Fri 20 Aug 2021 22:04:48 GMT +0000. [02:10:45] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sat 07 Aug 2021 05:56:13 GMT +0000. [02:10:46] RECOVERY - wiki.astralprojections.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Sat 14 Aug 2021 18:06:42 GMT +0000. [02:10:47] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Sat 07 Aug 2021 09:16:22 GMT +0000. [02:10:47] @dmehus all wikis are down [02:10:49] RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sat 07 Aug 2021 11:02:49 GMT +0000. [02:10:54] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Sat 07 Aug 2021 10:42:34 GMT +0000. [02:10:57] RECOVERY - puardian.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'puardian.xyz' will expire on Tue 27 Jul 2021 09:14:30 GMT +0000. [02:10:58] RECOVERY - es.nocyclo.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'es.nocyclo.tk' will expire on Sat 07 Aug 2021 01:23:47 GMT +0000. [02:10:58] Cocopuff2018, thanks :) [02:10:59] RECOVERY - wiki.thefactoryhka.com.pa - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thefactoryhka.com.pa' will expire on Fri 09 Jul 2021 05:41:36 GMT +0000. [02:11:00] RECOVERY - www.rangpurpedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'rangpurpedia.xyz' will expire on Fri 02 Jul 2021 06:08:10 GMT +0000. [02:11:01] RECOVERY - animatedfeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [02:11:01] RECOVERY - wiki.usagihime.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.usagihime.ml' will expire on Sat 07 Aug 2021 03:09:56 GMT +0000. [02:11:01] RECOVERY - wiki.meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Sat 21 Aug 2021 14:09:26 GMT +0000. [02:11:02] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Sat 07 Aug 2021 03:34:59 GMT +0000. [02:11:03] I'll update the task [02:11:09] RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Sat 07 Aug 2021 05:50:29 GMT +0000. [02:11:09] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Sat 07 Aug 2021 01:39:47 GMT +0000. [02:11:10] RECOVERY - wiki.jumpbound.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jumpbound.com' will expire on Wed 11 Aug 2021 04:43:11 GMT +0000. [02:11:18] RECOVERY - wiki.helioss.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.helioss.co' will expire on Sat 07 Aug 2021 02:54:20 GMT +0000. [02:11:18] RECOVERY - wiki.xfbs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.xfbs.net' will expire on Sat 07 Aug 2021 19:05:17 GMT +0000. [02:11:29] ``` Doug [02:11:29] SENDOPS #miraheze-sre see #miraheze-sre - 502 Bad Gateways and 500 Internal Server Errors on all or most wikis going back 10 minutes or more. Phabricator task is https://phabricator.miraheze.org/T7352, and cause should be investigated [02:11:29] 19:10 [02:11:29] The memo has been successfully sent to 6 operators on #miraheze-sre.``` [02:11:29] Cocopuff2018: ^ [02:11:30] [ ⚓ T7352 502 Bad Gateway and 500 Internal Server Error on metawiki ] - phabricator.miraheze.org [02:11:36] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Tue 10 Aug 2021 19:08:35 GMT +0000. [02:11:45] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sun 08 Aug 2021 05:05:42 GMT +0000. [02:11:45] RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate 'oecumene.org' will expire on Sat 07 Aug 2021 10:23:41 GMT +0000. [02:11:49] RECOVERY - www.wikilukas.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wikilukas.tk' will expire on Sat 07 Aug 2021 19:12:21 GMT +0000. [02:11:52] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Fri 06 Aug 2021 23:54:15 GMT +0000. [02:11:53] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [02:12:02] RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Fri 20 Aug 2021 22:59:52 GMT +0000. [02:12:05] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Thu 29 Jul 2021 12:53:14 GMT +0000. [02:12:08] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Sat 07 Aug 2021 02:55:10 GMT +0000. [02:12:12] RECOVERY - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitaldesignhq.com' will expire on Mon 09 Aug 2021 20:08:17 GMT +0000. [02:12:16] RECOVERY - wiki.ostans.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ostans.com' will expire on Sat 07 Aug 2021 08:01:07 GMT +0000. [02:12:17] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Sat 07 Aug 2021 10:09:02 GMT +0000. [02:12:21] RECOVERY - www.fossilfighterswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com' will expire on Sun 20 Jun 2021 19:36:06 GMT +0000. [02:12:27] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Sat 07 Aug 2021 08:19:02 GMT +0000. [02:12:29] RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Sat 07 Aug 2021 10:38:37 GMT +0000. [02:12:29] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 21 Aug 2021 00:11:35 GMT +0000. [02:12:29] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Sat 07 Aug 2021 08:35:47 GMT +0000. [02:12:35] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Sat 07 Aug 2021 18:22:20 GMT +0000. [02:12:35] RECOVERY - wiki.slimelab.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimelab.net' will expire on Tue 29 Jun 2021 13:29:07 GMT +0000. [02:12:35] RECOVERY - bots.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Thu 24 Jun 2021 11:21:27 GMT +0000. [02:12:35] RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Sat 07 Aug 2021 03:27:18 GMT +0000. [02:12:35] RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Fri 20 Aug 2021 22:13:22 GMT +0000. [02:12:36] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Sat 07 Aug 2021 08:58:20 GMT +0000. [02:12:42] RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Sat 07 Aug 2021 10:09:02 GMT +0000. [02:12:51] RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 4.632 second response time [02:12:55] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.60, 1.74, 1.76 [02:12:56] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2271 bytes in 5.781 second response time [02:13:04] RECOVERY - idleon.info - LetsEncrypt on sslhost is OK: OK - Certificate 'idleon.info' will expire on Sat 07 Aug 2021 10:14:05 GMT +0000. [02:13:05] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Sat 07 Aug 2021 07:36:14 GMT +0000. [02:13:16] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sat 07 Aug 2021 00:37:14 GMT +0000. [02:13:19] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Sat 07 Aug 2021 09:03:51 GMT +0000. [02:13:20] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Sat 07 Aug 2021 01:15:48 GMT +0000. [02:13:21] RECOVERY - wikibeats.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikibeats.net' will expire on Sat 07 Aug 2021 18:43:31 GMT +0000. [02:13:34] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Sat 07 Aug 2021 08:35:47 GMT +0000. [02:13:36] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Sat 07 Aug 2021 06:47:52 GMT +0000. [02:13:42] RECOVERY - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is OK: OK - NGINX Error Rate is 9% [02:13:55] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 24% [02:14:01] RECOVERY - wiki.velaan.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.velaan.org' will expire on Sat 07 Aug 2021 08:06:51 GMT +0000. [02:14:20] RECOVERY - wiki.arnab.at - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.arnab.at' will expire on Sat 07 Aug 2021 07:26:41 GMT +0000. [02:14:21] RECOVERY - petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.petrawiki.org' will expire on Fri 25 Jun 2021 05:54:36 GMT +0000. [02:14:22] RECOVERY - www.erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com' will expire on Fri 20 Aug 2021 22:28:16 GMT +0000. [02:14:22] RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Sat 07 Aug 2021 00:58:24 GMT +0000. [02:14:22] RECOVERY - wiki.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ct777.cf' will expire on Fri 02 Jul 2021 05:32:35 GMT +0000. [02:14:26] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is WARNING: WARNING - NGINX Error Rate is 49% [02:14:27] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Fri 06 Aug 2021 23:39:26 GMT +0000. [02:14:27] @dmehus look like none available [02:14:29] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Sat 07 Aug 2021 10:48:18 GMT +0000. [02:14:33] RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pa.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:45:02 GMT +0000. [02:14:34] RECOVERY - lama.madooa.org - LetsEncrypt on sslhost is OK: OK - Certificate 'lama.madooa.org' will expire on Sat 07 Aug 2021 08:51:42 GMT +0000. [02:14:47] RECOVERY - cp13 HTTPS on cp13 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2272 bytes in 0.568 second response time [02:14:56] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.41, 1.27, 1.58 [02:14:58] Cocopuff2018, okay, thank you. I've updated the task. No one with server access is available at the moment [02:15:20] RECOVERY - history.sdtef.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Sat 07 Aug 2021 03:36:43 GMT +0000. [02:15:20] RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Fri 30 Jul 2021 05:01:06 GMT +0000. [02:15:22] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Sat 07 Aug 2021 07:44:17 GMT +0000. [02:15:22] RECOVERY - www.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Fri 25 Jun 2021 05:43:40 GMT +0000. [02:15:31] RECOVERY - rangpurpedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'rangpurpedia.xyz' will expire on Fri 02 Jul 2021 06:08:10 GMT +0000. [02:15:34] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Wed 30 Jun 2021 11:38:37 GMT +0000. [02:15:39] RECOVERY - planetnomads.speedbomb.de - LetsEncrypt on sslhost is OK: OK - Certificate 'planetnomads.speedbomb.de' will expire on Sat 07 Aug 2021 04:19:27 GMT +0000. [02:15:41] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Sat 07 Aug 2021 07:05:38 GMT +0000. [02:15:44] RECOVERY - icinga.bots.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate '*.bots.miraheze.wiki' will expire on Tue 17 Aug 2021 04:56:02 GMT +0000. [02:15:49] RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15299 bytes in 9.362 second response time [02:15:54] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Sat 07 Aug 2021 03:00:47 GMT +0000. [02:15:54] RECOVERY - wiki.yumeka.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.icu' will expire on Sun 13 Jun 2021 13:30:44 GMT +0000. [02:15:58] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Sat 07 Aug 2021 07:54:14 GMT +0000. [02:16:00] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Sat 07 Aug 2021 07:38:27 GMT +0000. [02:16:06] RECOVERY - mr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mr.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:44:14 GMT +0000. [02:16:10] RECOVERY - cp13 Stunnel Http for mw10 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15299 bytes in 9.062 second response time [02:16:12] RECOVERY - wiki.candela.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.candela.digital' will expire on Sat 07 Aug 2021 03:42:24 GMT +0000. [02:16:23] RECOVERY - cp10 Stunnel Http for mw10 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 8.150 second response time [02:16:23] Wikis still down [02:16:23] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 9.487 second response time [02:16:26] RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Fri 20 Aug 2021 22:34:45 GMT +0000. [02:16:27] RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 22% [02:16:28] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Sat 07 Aug 2021 03:28:42 GMT +0000. [02:16:28] RECOVERY - wiki.rpointer.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rpointer.ml' will expire on Sat 24 Jul 2021 12:18:00 GMT +0000. [02:16:31] RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Sat 07 Aug 2021 10:12:24 GMT +0000. [02:16:34] PROBLEM - wiki.umbrel.tech - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.umbrel.tech' expires in 8 day(s) (Tue 01 Jun 2021 18:51:21 GMT +0000). [02:16:37] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:45:41 GMT +0000. [02:16:37] RECOVERY - matttest.mtwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'matttest.mtwiki.cf' will expire on Tue 06 Jul 2021 03:50:41 GMT +0000. [02:16:43] RECOVERY - lg.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lg.gyaanipedia.com' will expire on Sat 07 Aug 2021 01:59:00 GMT +0000. [02:16:44] RECOVERY - kkutu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kkutu.wiki' will expire on Sat 07 Aug 2021 04:15:13 GMT +0000. [02:16:48] RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2274 bytes in 7.984 second response time [02:16:52] RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Sat 07 Aug 2021 04:18:14 GMT +0000. [02:16:54] RECOVERY - wiki.thedev.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thedev.gq' will expire on Tue 17 Aug 2021 05:08:15 GMT +0000. [02:16:56] RECOVERY - erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com' will expire on Fri 20 Aug 2021 22:28:16 GMT +0000. [02:17:50] PROBLEM - wiki.invinciblewellbeing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.invinciblewellbeing.com could not be found [02:17:54] PROBLEM - www.winenjoy.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.winenjoy.net could not be found [02:17:54] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [02:17:59] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [02:18:06] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:18:08] @dmehus look like we gonna have to wait till sre wakes up [02:18:19] PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found [02:18:24] PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gesamtschule-nordkirchen.de could not be found [02:18:25] PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikien.wildterra2.com could not be found [02:18:26] PROBLEM - biblestrength.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for biblestrength.net could not be found [02:18:28] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [02:18:29] PROBLEM - wiki.wilderyogi.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wilderyogi.eu could not be found [02:18:29] PROBLEM - translate.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for translate.petrawiki.org could not be found [02:18:30] PROBLEM - aman.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for aman.awiki.org could not be found [02:18:40] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fbpml.org could not be found [02:18:44] PROBLEM - famedata.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for famedata.org could not be found [02:18:47] Cocopuff2018, I can access `metawiki`, but load times are atrocious [02:18:53] and it's hit or miss [02:19:06] that'll be another 3 hours before Reception123 is up [02:19:10] RECOVERY - cp11 Stunnel Http for mw10 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 9.637 second response time [02:19:40] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:19:58] PROBLEM - wiki.joust.ro - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.joust.ro could not be found [02:19:59] PROBLEM - wikilukas.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikilukas.tk could not be found [02:20:02] PROBLEM - wiki.s23.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.s23.org could not be found [02:20:05] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [02:20:06] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20889 bytes in 2.046 second response time [02:20:09] PROBLEM - wiki.xysspon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.xysspon.com could not be found [02:20:09] @dmehus can you load up any other wikis besides meta? [02:20:10] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.triplescripts.org could not be found [02:20:17] @dmehus can you load up any other wikis besides meta? [02:20:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.80, 20.15, 15.99 [02:21:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:22:19] @dhemus looks like its back for me [02:22:25] Cocopuff2018, let me try [02:22:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.21, 22.59, 17.33 [02:22:49] k. [02:23:07] yes, but still slow load times [02:23:18] It seems to be improving, but not great overall [02:24:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.93, 20.44, 17.15 [02:24:35] RECOVERY - wiki.invinciblewellbeing.com - reverse DNS on sslhost is OK: rDNS OK - wiki.invinciblewellbeing.com reverse DNS resolves to cp13.miraheze.org [02:24:36] gonna have to wait for sre [02:24:41] RECOVERY - www.winenjoy.net - reverse DNS on sslhost is OK: rDNS OK - www.winenjoy.net reverse DNS resolves to cp13.miraheze.org [02:24:42] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: rDNS OK - storytime.jdstroy.cf reverse DNS resolves to cp13.miraheze.org [02:24:54] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: rDNS OK - www.lab612.at reverse DNS resolves to cp13.miraheze.org [02:25:08] RECOVERY - wikien.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikien.wildterra2.com reverse DNS resolves to cp13.miraheze.org [02:25:15] RECOVERY - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is OK: rDNS OK - wiki.gesamtschule-nordkirchen.de reverse DNS resolves to cp13.miraheze.org [02:25:15] RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp13.miraheze.org [02:25:18] RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [02:25:18] RECOVERY - biblestrength.net - reverse DNS on sslhost is OK: rDNS OK - biblestrength.net reverse DNS resolves to cp13.miraheze.org [02:25:21] RECOVERY - translate.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - translate.petrawiki.org reverse DNS resolves to cp13.miraheze.org [02:25:28] RECOVERY - wiki.wilderyogi.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.wilderyogi.eu reverse DNS resolves to cp13.miraheze.org [02:25:30] RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fbpml.org reverse DNS resolves to cp13.miraheze.org [02:25:31] RECOVERY - aman.awiki.org - reverse DNS on sslhost is OK: rDNS OK - aman.awiki.org reverse DNS resolves to cp13.miraheze.org [02:25:38] RECOVERY - famedata.org - reverse DNS on sslhost is OK: rDNS OK - famedata.org reverse DNS resolves to cp13.miraheze.org [02:26:02] Cocopuff2018, yep. Task updated hopefully for the final time till SRE examines it [02:26:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.14, 18.09, 16.71 [02:26:41] RECOVERY - wikilukas.tk - reverse DNS on sslhost is OK: rDNS OK - wikilukas.tk reverse DNS resolves to cp13.miraheze.org [02:26:51] RECOVERY - wiki.joust.ro - reverse DNS on sslhost is OK: rDNS OK - wiki.joust.ro reverse DNS resolves to cp13.miraheze.org [02:26:55] RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: rDNS OK - wiki.triplescripts.org reverse DNS resolves to cp13.miraheze.org [02:26:58] RECOVERY - wiki.xysspon.com - reverse DNS on sslhost is OK: rDNS OK - wiki.xysspon.com reverse DNS resolves to cp13.miraheze.org [02:26:59] RECOVERY - wiki.s23.org - reverse DNS on sslhost is OK: rDNS OK - wiki.s23.org reverse DNS resolves to cp13.miraheze.org [02:27:02] RECOVERY - wiki.climatechange.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.climatechange.ai reverse DNS resolves to cp13.miraheze.org [02:29:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.43, 5.73, 7.94 [02:31:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.22, 6.00, 7.84 [02:31:44] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:33:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 6.44, 7.87 [02:35:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 7.23, 8.01 [02:37:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.87, 7.38, 7.96 [02:41:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.07, 5.37, 6.58 [02:43:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.61, 5.38, 6.65 [02:43:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.81, 6.73, 7.76 [02:45:27] Sorry I was away also. Is everything working now? Matomo and Login says those two are back up now, is everything working? dmehus, Cocopuff2018 ? [02:51:12] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.77, 1.36, 1.20 [02:51:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.65, 5.66, 6.58 [02:53:12] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.83, 1.17, 1.15 [02:54:05] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:55:19] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.05, 5.76, 6.65 [02:57:12] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.27, 1.72, 1.41 [02:59:12] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.99, 1.46, 1.35 [02:59:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 6.78, 6.85 [03:00:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.03, 6.93, 6.67 [03:02:36] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.07, 6.66, 6.61 [03:03:19] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.97, 7.53, 7.12 [03:06:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.13, 6.83, 6.73 [03:07:39] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [03:08:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.70, 6.09, 6.47 [03:09:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.02, 7.65, 7.38 [03:21:20] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.79, 6.08, 6.70 [03:26:52] Crepusculum, seems to be, as far as I know. Should be investigated, though, when Paladox and Southparkfan are around [03:28:33] dmehus: Yeah definitely should be investigated. Seems to be SSL or DNS related maybe. [03:41:06] Crepusculum, yeah, maybe. I'm thinking the cache proxies OOM'ed [03:43:33] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [03:45:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jsb4v [03:45:12] [02miraheze/services] 07MirahezeSSLBot 038cae525 - BOT: Updating services config for wikis [03:49:39] dmehus: even Matomo went down, I'm not positive how that one really works but would that cause Matomo to be down also? [04:18:23] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [04:34:25] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [04:43:54] Crepusculum, yeah, I mean, Nginx is the web server, so if that OOM'ed, in theory, Matomo could stay running, assuming it's a separate process; it just wouldn't stay accessible. But maybe that suggests it's broader than Nginx/Varnish, and was a cache proxy issue or maybe something OOM'ed and brought down the whole cloud server? [04:47:55] dmehus: Yeah I thought it might've been the whole cloud server tbh, but if that happened we would have had way more alerts I think. Something caused it, and that definitely needs looked into as it was a total outage. I'm just glad it self resolved otherwise we'd still be down as I have access to handle issues beyond the MediaWiki servers, so couldn't have fixed it anyways, and pal_adox and SPF seemed to be away, and [04:47:55] Reception_123 was likely asleep. (Put underscores in usernames so I don't ping them) [04:47:58] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [04:49:52] Crepusculum, yeah, though there were quite a few alerts, I think, but yeah, definitely needs SRE infrastructure team to investigate thoroughly [04:50:45] lol I mean underscores work, or we could invent nicknames for them...like pdox, SPF, and Reception [04:51:25] Yeah lol could've. [04:52:45] dmehus: And yeah definitely needs investigating for what happened shouldn't have, no one was doing anything on any servers at the time. It lasted 14 minutes (Matomo outage lasted 4 minutes is all though) [05:07:04] Crepusculum, hrm, I think it lasted a little longer than that. It started 3 minutes before 7 pm my time and went at least until 7:14 pm and wasn't normalized in terms of page load times until 7:20-7:25 pm [05:09:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JsbPk [05:09:10] [02miraheze/mw-config] 07Universal-Omega 03eb1e3e2 - Remove two deprecated password policy checks [05:09:11] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [05:09:13] [02mw-config] 07Universal-Omega opened pull request 03#3916: Remove two deprecated password policy checks - 13https://git.io/JsbPL [05:10:15] miraheze/mw-config - Universal-Omega the build passed. [05:10:22] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:10:26] dmehus: not according to status.miraheze.wiki . [05:16:09] Crepusculum, hrm, I'm just going by when I and others couldn't access Miraheze wikis and we couldn't access them from 6:57 pm through 7:14 pm my time, though there may have been some intermittent access in there [05:18:15] dmehus: According to status.miraheze.wiki it started at 7:00 and was back at 7:14 (your time) but maybe it is off by 3 minutes? [05:32:51] Crepusculum, yeah, that is about right, though it was definitely started 1-3 minutes before 7 pm. And by back it wasn't fully back. 502 errors did intermittently persist for another several minutes after 7:14 pm [05:35:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 6.22, 5.41 [05:37:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.75, 5.97, 5.40 [05:39:50] Morning, RhinosF1, and Reception123 :) [05:40:17] hi. I've seen the error but so far have no idea how it happened [05:40:18] dmehus: saw your task [05:40:29] Reception123: logs? [05:40:30] checked OVH and it doesn't seem to be a problem from there [05:40:33] Reception123, RhinosF1: ack, yeah I don't know [05:40:35] On cp12/13 [05:41:02] was it only cp12/cp13? If so why wouldn't the other ones just take over? [05:41:04] My working theory is Nginx or Varnish OOM'ed badly, and brought down cache proxies and cloud server [05:41:48] while good in theory, the kernel log shows no OOMs on any cp servers in the past 8 hours [05:41:53] Reception123: they should have [05:41:54] oh :( [05:41:59] hrm [05:42:00] unless I've done something wrong [05:42:00] Check varnish logs [05:42:11] It doesn't look like an oom [05:42:27] * dmehus hopes it was a DoS attack [05:42:44] * dmehus hopes he didn't just get RhinosF1 into panic mode with that thought bubble [05:43:21] Reception123: cp3 shows faults [05:43:25] yeah [05:43:34] Just didn't see it in Doug's icinga alerts [05:43:41] I do think cp3 was a major source of the problem, but it wasn't the only one [05:43:50] https://www.irccloud.com/pastebin/6fz29FIy/ [05:43:51] [ Snippet | IRCCloud ] - www.irccloud.com [05:43:54] yeah that might've come later [05:44:15] nothing that tells us anything new really [05:44:21] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [05:44:22] oh [05:44:43] Reception123: that means mw at fault [05:45:06] Oh f [05:45:11] That's some load [05:45:18] yeah [05:45:24] there were major load issues at the time [05:45:48] and GDNSD servers were down according to the icinga logs [05:46:06] there does seem to be a strange OOM error on mw8/mw9 then but the time doesn't match [05:46:11] oh [05:46:16] what time, Reception123? [05:46:16] one is way before the error and one is after [05:46:28] 23:52 UTC and 03:00 UTC [05:46:41] and the error would've been at 2:00 UTC from what I've seen [05:46:41] I got my first 502 or 500 (can't remember which) at 6:57 my time [05:46:52] dmehus: gdnsd means all cps went down [05:47:00] RhinosF1, ah [05:47:12] Reception123, oh that's a good hour before [05:47:24] yeah, so I'd say that's not really related [05:47:52] possibly, but not necessarily. Hard to say, I guess [05:48:03] Reception123: paste the log [05:48:19] the varnish one you mean? [05:48:38] No the strange OOM [05:48:41] ah [05:49:05] convert invoked oom-killer: gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0 [05:49:10] dmehus: major load issues is an understatement [05:49:19] 23:52 UTC on mw8 and 03:00 UTC on mw9 [05:49:21] RhinosF1, heh yeah [05:49:57] Reception123: yeah that not helping [05:50:15] yeah, though I'm unsure what else it could be [05:50:31] heh I wish paladox or Southparkfan were around :D [05:51:26] Reception123: check /etc/cron.d/ [05:51:37] Look if anything matches the timings [05:52:50] I mean there's not much in there, nothing that matches [05:53:05] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:53:23] Reception123: I was hoping a task in there would have started the right time [05:53:41] Crepusculum, paladox, Southparkfan: I'm now out of ideas [05:53:59] well there's only puppet and a php sessionclean [05:54:00] Because I can't do top on something that's already happened [05:56:25] Yeah I don't know what could've caused it either. I tried looking a bit earlier after it happened but didn't find anything that could cause it. I was away when it happened though so was about 45 minutes after. [05:57:37] well I think even if people were there during that time there wouldn't have been much one could've done since even after we can't figure out what caused it [05:58:26] I'm at a loss now [06:06:39] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [06:18:57] Reception123: we had a spike of uncached requests [06:19:07] So now we need to look at what was a cache miss then [06:19:17] hmm [06:20:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 6.29, 5.79 [06:20:56] Reception123: does graylog cache proxy logs show cache miss / hit [06:21:02] On requests [06:22:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.32, 6.05, 5.75 [06:22:21] varnish isn't on graylog [06:22:39] Under nginx logs [06:22:44] Probably [06:22:55] Reception123, ah we haven't centralized the Varnish logs, I gather? [06:23:12] yeah [06:23:19] ah [06:23:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.80, 6.24, 5.31 [06:24:01] searched for 'miss' and there doesn't seem to be anything [06:25:20] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.13, 7.39, 6.10 [06:25:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.27, 7.43, 5.85 [06:30:34] Reception123: if it's there it'll be a filter [06:30:41] cache_hit or something [06:35:48] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:37:17] Reception123: varnishtop -i BereqURL [06:38:17] But that might only have worked in the moment [06:39:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.83, 7.17, 7.23 [06:43:48] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.98, 6.06, 6.77 [06:45:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.48, 7.43, 7.90 [06:47:56] Reception123: we're gonna have to manually read logs [06:47:59] Well you are [06:49:09] RhinosF1, like access logs, or system logs? [06:49:22] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [06:49:35] dmehus: varnish ones [06:51:14] I can look a bit now but I won't really be around soon until the afternoon [06:51:19] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.53, 5.52, 6.80 [06:51:39] Reception123: okay [06:51:51] I wish I could tell you there was an easy way [06:51:57] But even wikimedia sre don't have one [06:52:15] RhinosF1, ah [07:04:59] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:11:37] PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (3 critical updates). [07:12:26] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates). [07:12:53] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (3 critical updates). [07:13:03] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (2 critical updates). [07:16:05] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (3 critical updates). [07:19:54] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (2 critical updates). [07:21:03] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 108 packages available for upgrade (3 critical updates). [07:21:04] PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [07:21:16] PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [07:21:20] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 73 packages available for upgrade (2 critical updates). [07:21:43] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 73 packages available for upgrade (2 critical updates). [07:22:14] PROBLEM - test4 APT on test4 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [07:22:41] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (2 critical updates). [07:23:13] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 51 packages available for upgrade (2 critical updates). [07:23:41] PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (2 critical updates). [07:24:29] PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (2 critical updates). [07:24:53] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (4 critical updates). [07:27:05] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (3 critical updates). [07:27:16] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (2 critical updates). [07:27:38] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [07:31:37] PROBLEM - cp13 APT on cp13 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (2 critical updates). [07:32:27] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (4 critical updates). [07:32:38] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [07:34:29] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates). [07:34:54] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates). [07:35:01] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (4 critical updates). [07:35:19] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates). [07:35:23] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates). [07:35:37] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (4 critical updates). [07:36:05] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [07:36:15] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates). [07:37:36] PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (2 critical updates). [07:38:49] PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates). [07:59:19] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [08:28:25] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:41:59] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [08:57:34] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:11:07] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [09:33:27] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:47:23] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [10:03:24] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [10:53:28] PROBLEM - ping6 on dbbackup1 is CRITICAL: CRITICAL - Destination Unreachable (fe80::f816:3eff:fe94:15f5) [10:57:04] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [11:20:57] RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 91.18 ms [11:26:08] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [12:00:39] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [12:10:12] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±18] 13https://git.io/JsAJ8 [12:10:13] [02miraheze/MirahezeMagic] 07translatewiki 0370789f2 - Localisation updates from https://translatewiki.net. [12:10:14] [ Main page - translatewiki.net ] - translatewiki.net [12:10:15] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JsAJR [12:10:16] [02miraheze/ManageWiki] 07translatewiki 03284a6fe - Localisation updates from https://translatewiki.net. [12:10:17] [ Main page - translatewiki.net ] - translatewiki.net [12:10:18] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JsAJ0 [12:10:19] [02miraheze/CreateWiki] 07translatewiki 0353b1f7b - Localisation updates from https://translatewiki.net. [12:10:20] [ Main page - translatewiki.net ] - translatewiki.net [12:11:11] miraheze/ManageWiki - translatewiki the build passed. [12:11:20] miraheze/MirahezeMagic - translatewiki the build passed. [12:11:22] miraheze/CreateWiki - translatewiki the build passed. [12:16:25] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [12:17:03] here we go with banning azure IPs again [12:30:12] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [12:34:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 6.13, 5.63 [12:36:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.68, 5.82, 5.57 [12:45:58] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [12:52:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 6.76, 6.06 [12:54:15] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.37, 7.05, 6.23 [12:56:10] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.38, 6.63, 6.17 [13:10:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.74, 7.04, 6.56 [13:14:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.69, 6.44, 6.41 [13:20:24] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [13:30:53] PROBLEM - wiki.twilightsignal.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.twilightsignal.com could not be found [13:30:54] PROBLEM - test2.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for test2.miraheze.org could not be found [13:32:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.01, 6.52, 6.08 [13:33:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.19, 7.91, 7.10 [13:34:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.18, 7.60, 6.53 [13:36:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.26, 6.84, 6.19 [13:36:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.87, 7.22, 6.53 [13:37:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.15, 7.43, 7.12 [13:37:35] RECOVERY - wiki.twilightsignal.com - reverse DNS on sslhost is OK: rDNS OK - wiki.twilightsignal.com reverse DNS resolves to cp13.miraheze.org [13:37:47] RECOVERY - test2.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - test2.miraheze.org reverse DNS resolves to cp13.miraheze.org [13:38:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.20, 6.24, 6.05 [13:38:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.12, 6.76, 6.44 [13:46:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.72, 6.60, 6.80 [13:50:49] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.02, 6.84, 6.86 [13:52:50] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.00, 6.65, 6.80 [13:55:34] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JsARx [13:55:36] [02miraheze/puppet] 07Reception123 03e8e6f69 - set test3 branch to REL1_36 [13:55:37] [02puppet] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbiAS [13:55:39] [02puppet] 07Reception123 opened pull request 03#1761: set test3 branch to REL1_36 - 13https://git.io/JsARh [13:56:51] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:04:40] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JsAuT [14:04:41] [02miraheze/dns] 07Reception123 03827c762 - add thestarsareright.org zone [14:04:42] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.64, 7.64, 6.65 [14:05:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.45, 6.78, 6.20 [14:06:42] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 6.87, 6.49 [14:07:00] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.19, 6.77, 6.29 [14:08:52] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/JsAzJ [14:08:53] [02miraheze/ssl] 07Reception123 03d24087b - add thestarsareright.org cert [14:08:55] [02miraheze/ssl] 07Reception123 035c51c5d - add electrowiki.org cert [14:10:37] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [14:12:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.64, 6.96, 6.74 [14:12:46] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.41, 6.37, 6.40 [14:14:03] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.17, 6.43, 6.58 [14:18:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.42, 7.07, 6.40 [14:20:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.10, 6.69, 6.35 [14:22:12] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.71, 8.12, 7.12 [14:23:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.58, 6.72, 6.17 [14:24:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.78, 7.30, 6.69 [14:25:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.21, 6.74 [14:26:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 7.94, 7.31 [14:26:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.32, 6.62, 6.52 [14:27:19] [02miraheze/mw-config] 07Reception123 created branch 03Reception123-patch-1 13https://git.io/JsAVM [14:27:21] [02miraheze/mw-config] 07Reception123 created branch 03Reception123-patch-1-1 13https://git.io/JsAVM [14:27:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.47, 6.63, 6.30 [14:27:22] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [14:27:24] [02mw-config] 07Reception123 created branch 03Reception123-patch-1-1 - 13https://git.io/vbvb3 [14:27:26] [02mw-config] 07Reception123 opened pull request 03#3917: extendedconfirmed protection level for lhmnwiki T7346 - 13https://git.io/JsAVS [14:28:28] miraheze/mw-config - Reception123 the build passed. [14:28:40] [02mw-config] 07Reception123 closed pull request 03#3917: extendedconfirmed protection level for lhmnwiki T7346 - 13https://git.io/JsAVS [14:28:42] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsAwv [14:28:43] [02miraheze/mw-config] 07Reception123 036e98289 - extendedconfirmed protection level for lhmnwiki T7346 (#3917) [14:28:44] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1-1 - 13https://git.io/vbvb3 [14:28:46] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1-1 [14:29:40] miraheze/mw-config - Reception123 the build passed. [14:29:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.56, 6.74, 6.64 [14:30:33] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsAw3 [14:30:35] [02miraheze/mw-config] 07Reception123 030608a31 - fix mistake made in previous PR #3893 [14:31:32] miraheze/mw-config - Reception123 the build passed. [14:32:09] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.45, 8.16, 7.54 [14:33:37] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.04, 8.01, 7.13 [14:34:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 6.92, 6.58 [14:36:53] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [14:37:03] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.34, 7.95, 6.96 [14:38:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.53, 8.60, 7.29 [14:38:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.76, 7.49, 6.92 [14:40:52] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.24, 7.76, 7.08 [14:42:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.57, 7.74, 7.15 [14:43:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.41, 7.97, 7.64 [14:44:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.93, 7.51, 7.30 [14:46:28] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:50:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.89, 7.73, 7.20 [14:51:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.11, 7.05, 7.18 [14:52:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.73, 7.68, 7.25 [14:53:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.54, 7.21, 7.22 [14:55:21] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 7.38, 7.27 [14:56:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.09, 5.69, 6.51 [14:57:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.76, 6.41, 6.93 [14:57:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 7.55, 7.92 [14:59:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.89, 6.02, 6.74 [14:59:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [15:00:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.06, 7.32, 7.21 [15:01:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.69, 7.85, 7.93 [15:02:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.83, 6.96, 6.76 [15:03:16] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.53, 8.75, 7.61 [15:04:54] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.43, 8.75, 7.48 [15:06:50] Reception123: hi [15:06:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.57, 7.89, 7.32 [15:08:52] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [15:09:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.16, 7.67, 7.62 [15:09:58] [02puppet] 07RhinosF1 edited pull request 03#1735: Switch test3 to 1.36 stuff - 13https://git.io/JOeT9 [15:10:30] [02puppet] 07RhinosF1 commented on pull request 03#1761: set test3 branch to REL1_36 - 13https://git.io/JsAXl [15:11:01] Reception123: use mine [15:12:10] ah, good idea so we don't need to manually pull the branch [15:12:30] hello [15:12:34] [02puppet] 07Reception123 closed pull request 03#1761: set test3 branch to REL1_36 - 13https://git.io/JsARh [15:12:46] Reception123: feel free to go whenever [15:12:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.70, 7.46, 7.73 [15:12:56] I will do schema changes this evening [15:13:02] Hi Cocopuff2018 [15:13:04] any updates on special Multi edit [15:13:07] No [15:13:39] well then might not be able to use it anymoe [15:14:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.12, 7.40, 7.92 [15:14:37] will do later today then, when Crepusculum is also around [15:15:13] Reception123: I wouldn't worry about breaking test3 [15:16:19] yeah, that's not a worry. we have the backup worst case even if it would be fine even without it [15:16:28] probably will do it in an hour or so [15:16:42] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.67, 7.89, 7.82 [15:16:43] big outage last night [15:17:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.28, 6.49, 6.75 [15:17:24] !log test3 -> 1.36 in ~1 hour, please don't panic at exceptions until you see a message saying 1.36 schema changes done - See T7117 [15:17:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:17:34] Cocopuff2018: we know [15:17:42] Not sure why yet [15:18:44] RhinosF1: oh and if we could I'd like to fix the Flow issue before we upgrade so we can properly test it after [15:18:56] Reception123: which flow issue? [15:19:10] RhinosF1: exception and the thing is when I try to search the trace ID in graylog nothing shows up [15:19:16] you can get the error by visiting any talk page [15:19:20] like Talk:Test [15:19:26] Reception123: link? [15:19:54] https://test3.miraheze.org/wiki/Talk:Test [15:19:56] [ Unknown topic - Test3 ] - test3.miraheze.org [15:20:03] the wikis seemed to come back up in up in a short period of time [15:20:15] in a short period of time [15:20:34] Reception123: nothing in graylog at all? [15:20:53] no, that's the strange part, I tried to search IDs and nothing shows up [15:21:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.74, 6.84, 6.84 [15:21:44] does miraheze have some kind of backup thing for if the system goes down? [15:22:18] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [15:22:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.24, 7.86, 7.85 [15:22:39] Cocopuff2018: we don't have redundancy [15:22:48] Backups yes to a degree [15:23:06] But you couldn't run off them [15:24:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.16, 7.51, 7.66 [15:24:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.83, 6.55, 6.75 [15:25:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.24, 6.23, 6.79 [15:26:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.47, 6.81, 7.39 [15:26:51] test3 is going to 1.36 already? [15:27:28] yeah, we're preparing for the 1.36 release which shouldn't be in way too long [15:28:04] perhaps by July it should be ready? [15:28:09] I hope so [15:28:34] We'll have Debian upgrades shortly after too but hopefully you won't notice them directly [15:28:56] how long does upgrading take? to finish [15:29:13] 1.36 a while [15:29:21] Normally we do two hours for production [15:30:08] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.17, 7.73, 7.60 [15:30:48] after 1.36 i am sure it will be a few years before 1.37 is released [15:30:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.59, 7.13, 6.89 [15:32:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.72, 7.41, 7.51 [15:32:54] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.62, 6.39, 6.66 [15:34:05] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 7.78, 7.63 [15:34:20] not a few years, usually it's two every year [15:34:47] but because of COVID-19 there was only one last year [15:35:14] The 1.36. 0 stable release is expected to come out in May 2021 [15:35:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.03, 6.87 [15:36:46] found a whole list of task for 1.36 [15:36:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.02, 7.21, 6.97 [15:37:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.54, 6.46, 6.68 [15:38:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.42, 7.35, 7.35 [15:38:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.65, 6.64, 6.79 [15:40:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.28, 7.84, 7.74 [15:40:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.54, 6.94, 7.19 [15:41:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 6.86, 6.83 [15:43:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.10, 7.46, 7.04 [15:43:58] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.02, 8.76, 8.09 [15:44:24] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [15:44:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.36, 7.57, 7.38 [15:44:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.76, 7.23, 6.98 [15:46:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.68, 7.22, 7.28 [15:47:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.58, 7.33, 7.07 [15:49:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.01, 7.60, 7.83 [15:51:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.97, 7.83, 7.27 [15:52:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.60, 6.55, 6.78 [15:53:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.68, 7.66, 7.28 [15:57:52] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [15:58:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.10, 6.12, 6.77 [16:03:18] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 6.91, 6.90 [16:03:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.20, 7.25, 7.26 [16:04:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.40, 6.75, 6.80 [16:04:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.99, 6.83, 6.63 [16:05:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.43, 6.81, 6.88 [16:05:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.40, 7.33, 7.29 [16:08:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.60, 6.06, 6.41 [16:09:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.36, 6.33, 6.65 [16:10:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.28, 6.26, 6.61 [16:12:48] Reception123: ping for puppet stuff [16:13:15] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [16:13:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.45, 6.07, 6.76 [16:20:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.20, 6.13, 6.00 [16:24:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.37, 6.58, 6.24 [16:25:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.69, 6.89, 6.71 [16:27:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.65, 6.66 [16:34:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.14, 8.20, 6.93 [16:35:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.21, 7.27, 6.38 [16:36:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.27, 6.91, 6.35 [16:36:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 7.50, 6.84 [16:37:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.20, 6.82, 6.32 [16:37:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 6.99, 6.73 [16:40:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.30, 6.11, 6.20 [16:41:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 6.66, 6.68 [16:42:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.99, 6.37, 6.60 [16:43:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.93, 6.22, 6.30 [16:49:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.81, 7.72, 6.91 [16:50:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 6.93, 6.50 [16:51:52] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.17, 7.37, 6.93 [16:52:55] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.33, 6.63, 6.44 [16:53:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.11, 7.29, 6.96 [16:55:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.47, 8.04, 7.23 [16:55:52] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.83, 8.00, 7.26 [16:57:01] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.16, 7.71, 7.21 [16:58:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 6.88, 6.60 [16:58:57] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.79, 8.01, 7.38 [16:59:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 7.71, 7.32 [17:00:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.36, 7.39, 7.23 [17:02:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.11, 6.36, 6.44 [17:03:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 8.12, 7.59 [17:05:25] [02mw-config] 07dmehus commented on commit 030608a3109f05488074dd5928d61d2ac44fd36c5f - 13https://git.io/JsxIl [17:05:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.93, 7.46, 7.40 [17:08:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.06, 6.10, 6.64 [17:13:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.16, 6.26, 6.78 [17:20:47] paladox: about the error last night do you have any idea which service might've actually caused the error? [17:20:49] that seems a bit unclear [17:21:13] i gave some hints but it's unclear. [17:21:18] anything in the logs at the time? [17:21:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 7.17, 6.95 [17:21:59] I mean I mostly just checked for OOMs and there wasn't anything related [17:23:54] https://grafana.miraheze.org/d/UzwQVt1mk/php-fpm-appservers?viewPanel=14&orgId=1&var-host=mw11.miraheze.org:9253&from=now-24h&to=now doesn't show it was a php-fpm running out of childs error [17:23:55] [ Grafana ] - grafana.miraheze.org [17:24:21] oh [17:24:22] wait [17:24:24] i misread it [17:24:55] that makes it look like all the php-fpm childs were in use for ~20 minutes [17:26:09] so then you think it's rather the mw* servers that were the cause [17:27:10] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [17:29:53] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.49, 6.63, 6.80 [17:30:18] i mean looking at the graphs for the dbs i see extra activity. [17:35:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.04, 6.83, 6.81 [17:39:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.71, 6.72, 6.75 [17:40:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.57, 7.55, 6.79 [17:40:43] [02puppet] 07Reception123 closed pull request 03#1735: Switch test3 to 1.36 stuff - 13https://git.io/JOeT9 [17:40:45] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jsx3R [17:40:46] [02miraheze/puppet] 07RhinosF1 03dc30497 - Switch test3 to 1.36 stuff (#1735) [17:44:11] Reception123: no it isnt [17:44:16] dmehus: * [17:45:56] RhinosF1, oh referring to my test3wiki edit? [17:46:10] so we're just upgrading test3wiki to MW 1.35.2 then? [17:46:25] dmehus: no we're updating to 1.36 [17:46:28] oh [17:46:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.59, 6.74, 6.63 [17:46:35] well it shows as MW 1.35.2 [17:46:45] Because it's not done yet [17:46:47] when Reception123 updated the test3wiki sitenotice [17:46:48] ah [17:47:02] dmehus: yeah heh it's because I used {{CURRENTVERSION}} [17:47:04] Will he be disabling Puppet on test3 then? [17:47:13] nope, running it! [17:47:26] because the version is set in puppet this time [17:47:33] Reception123, ah, but that should still go by test3wiki's version? [17:47:47] well the version is set to REL1_36 in puppet [17:47:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.35, 6.95, 6.86 [17:47:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.91, 6.93, 6.36 [17:48:00] dmehus: give it time to sync [17:48:29] Reception123, ah, so it's just a matter of needing Puppet to run then [17:48:35] in about 12 minutes or so [17:48:38] heh, so far there's an error [17:48:48] well I'm not going to wait, I ran it now manually :) [17:49:02] but anyway don't expect it to work, RhinosF1 needs to do the SQL changes [17:49:03] can't you make it so test3wiki is isolated from the mw servers as far as Puppet goes? [17:49:12] dmehus: yes, that is exactly what we did! [17:49:13] why is it causing errors on the mw servers? [17:49:18] it's not [17:49:32] oh that's just load errors [17:49:37] yeah, unrelated [17:49:39] but why'd you say there's a problem? [17:50:21] because puppet did fail, but on test3 [17:50:25] * dmehus hopes RhinosF1 is fully awake when he does the schema changes :D [17:50:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.80, 7.00, 6.78 [17:50:38] Reception123, ah, okay, that makes sense then thanks :) [17:50:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.17, 7.36, 6.41 [17:51:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.63, 7.49, 6.67 [17:52:50] !log rhinos@test3:/srv/mediawiki/config$ sudo -u www-data git checkout --track origin/REL1_36 [17:52:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:55] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 7.47, 6.75 [17:54:35] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.060 second response time [17:54:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.16, 7.81, 6.89 [17:55:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.15, 6.62, 6.53 [17:56:34] !log test3: puppet is a snail, 500 expected [17:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:50] Reception123: can you update composer [17:57:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.24, 8.27, 7.52 [17:58:06] urgh, update it? [17:58:36] hopes it's just php composer.phar update [17:58:42] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:58:45] */me [17:58:58] Reception123: wget -O composer.phar https://getcomposer.org/composer-1.phar | php && php composer.phar install --no-dev [17:59:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsxZB [17:59:04] should be [17:59:05] [02miraheze/puppet] 07paladox 03ac1babe - graylog: Upgrade elasticsearch, mongodb and graylog [17:59:32] [02puppet] 07dmehus opened pull request 03#1762: Super Gnomey edit to remove extra space - 13https://git.io/JsxZz [17:59:52] paladox: if you're updating stuff mind also doing Matomo? :) [17:59:56] and not sure if phab needs updating too [18:00:01] oh there's an update for matomo? [18:00:05] yup [18:00:06] sure i can look at that in a min [18:00:10] thanks! [18:00:16] [02puppet] 07paladox closed pull request 03#1762: Super Gnomey edit to remove extra space - 13https://git.io/JsxZz [18:00:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsxZK [18:00:19] [02miraheze/puppet] 07dmehus 03443ecd7 - Super Gnomey edit to remove extra space (#1762) [18:00:23] thanks paladox [18:00:33] Reception123: https://github.com/Universal-Omega/Monaco/pull/3 is merged, probs want to deploy it [18:00:33] !log test3: puppet finished a minute back, Reception123 is updating composer [18:00:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.71, 7.26 [18:00:34] [ Remove unused hook by paladox · Pull Request #3 · Universal-Omega/Monaco · GitHub ] - github.com [18:00:36] to 1.35 + 1.36 [18:00:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:50] paladox: will do [18:00:56] thanks! [18:01:13] PROBLEM - wiki.mussa.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mussa.id could not be found [18:01:23] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JsxZQ [18:01:24] [02miraheze/mediawiki] 07Reception123 031b7e782 - Update DataTransfer [18:01:46] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fc.songcontests.eu could not be found [18:01:51] PROBLEM - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phabdigests.bots.miraheze.wiki could not be found [18:01:52] PROBLEM - www.erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.erikapedia.com could not be found [18:01:52] PROBLEM - miraheze.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ga could not be found [18:01:53] PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found [18:01:54] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [18:02:00] PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found [18:02:00] PROBLEM - wiki.ssij.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssij.se could not be found [18:02:01] PROBLEM - www.tradwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.tradwiki.org could not be found [18:02:05] PROBLEM - ao90.pinho.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ao90.pinho.org could not be found [18:02:06] PROBLEM - es.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.gyaanipedia.com could not be found [18:02:18] Reception123: let me know when composer has ran [18:02:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.93, 7.51, 7.22 [18:02:38] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JsxZj [18:02:39] [02miraheze/mediawiki] 07Reception123 036e56774 - Update Monaco (skin) [18:02:42] RhinosF1: it has just now [18:03:12] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JsxnI [18:03:13] [02miraheze/mediawiki] 07Reception123 035257d95 - Update Monaco (skin) [18:03:20] Reception123: does merge message list & rebuild i18n run? [18:03:44] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 6.58, 3.95, 2.52 [18:03:54] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [18:03:55] RhinosF1: heh, you wish [18:03:55] Fatal error: Uncaught Exception: Unable to open file /srv/mediawiki/w/extensions/RightFunctions/extension.json: filemtime(): stat failed for /srv/mediawiki/w/extensions/RightFunctions/extension.json in /srv/mediawiki/w/includes/registration/ExtensionRegistry.php:177 [18:03:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.33, 8.34, 7.16 [18:04:17] there is indeed no extension.json for that [18:04:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.88, 8.60, 7.66 [18:04:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.64, 8.90, 7.63 [18:04:50] even though there is one here: https://github.com/wikimedia/mediawiki-extensions-RightFunctions/tree/REL1_36 [18:04:51] [ GitHub - wikimedia/mediawiki-extensions-RightFunctions at REL1_36 ] - github.com [18:05:08] so something is up [18:05:40] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.16, 3.85, 2.68 [18:07:39] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.53, 3.31, 2.62 [18:07:40] PROBLEM - wiki.mnzp.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mnzp.xyz could not be found [18:07:42] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seamly.net could not be found [18:07:44] Reception123: looking. I can't remember if that's one I did or not [18:08:29] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: rDNS OK - fc.songcontests.eu reverse DNS resolves to cp13.miraheze.org [18:08:35] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: rDNS OK - wiki.zymonic.com reverse DNS resolves to cp13.miraheze.org [18:08:35] Crepusculum: well it wouldn't be your fault as there is an extension.json in the GH repo [18:08:36] RECOVERY - miraheze.ga - reverse DNS on sslhost is OK: rDNS OK - miraheze.ga reverse DNS resolves to cp13.miraheze.org [18:08:39] just not on the server [18:08:39] RECOVERY - www.erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - www.erikapedia.com reverse DNS resolves to cp13.miraheze.org [18:08:41] RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: rDNS OK - madgenderscience.wiki reverse DNS resolves to cp13.miraheze.org [18:08:46] so it means the server seems to have not pulled the extension correctly [18:08:47] RECOVERY - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phabdigests.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [18:08:48] RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org [18:08:54] RECOVERY - www.tradwiki.org - reverse DNS on sslhost is OK: rDNS OK - www.tradwiki.org reverse DNS resolves to cp13.miraheze.org [18:08:56] RECOVERY - ao90.pinho.org - reverse DNS on sslhost is OK: rDNS OK - ao90.pinho.org reverse DNS resolves to cp13.miraheze.org [18:08:58] RECOVERY - wiki.ssij.se - reverse DNS on sslhost is OK: rDNS OK - wiki.ssij.se reverse DNS resolves to cp13.miraheze.org [18:09:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.73, 20.81, 17.76 [18:09:05] RECOVERY - es.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - es.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org [18:09:18] now is not the time icinga-miraheze [18:10:26] Reception123: try `cd /srv/mediawiki/w/extensions/RightFunctions && sudo -u www-data git fetch && sudo -u www-data git reset --hard origin/REL1_36` [18:10:45] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20928 bytes in 0.709 second response time [18:10:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jsxcy [18:10:52] [02miraheze/puppet] 07paladox 035e63fd1 - matomo: Upgrade to 4.3.0 [18:10:55] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:11:04] Crepusculum: will do, though I hope it's not a wider issue when no extensions worked [18:11:12] as we wouldn't want to deal with that with prod when we do the real upgrade [18:11:30] Yeah it's just to debug [18:11:40] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 2.32, 3.66, 3.02 [18:12:21] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:12:24] Crepusculum: that did it yeah [18:12:46] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:12:55] RhinosF1: Crepusculum MessageFiles/rebuild running [18:13:01] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.33, 19.95, 18.15 [18:13:08] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:13:47] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.35, 3.12, 2.90 [18:13:53] lol, RightFunctions was the first extension Universal_Omega ever installed for me actually, iirc [18:14:05] so he probably missed adding an extension.json file [18:14:13] Great Reception123, maybe it was an issue with puppet pulling correct commit from git? [18:14:15] Is extension.json now mandatory with MW 1.36? [18:14:28] RECOVERY - wiki.mnzp.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.mnzp.xyz reverse DNS resolves to cp13.miraheze.org [18:14:30] RECOVERY - wiki.seamly.net - reverse DNS on sslhost is OK: rDNS OK - wiki.seamly.net reverse DNS resolves to cp13.miraheze.org [18:14:30] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:14:31] No. But every extension on Miraheze should support it. [18:14:34] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:14:35] ^ [18:15:07] rebuild passed 'en' and test3 is still down [18:15:08] Dmehus: RightFunctions didn't have extension.json until 1.36. [18:15:08] Crepusculum, but why would it error now then, when we've had it installed since September? [18:15:13] 1.36 dmehus [18:15:16] Crepusculum, ah okay, interesting [18:15:17] Crepusculum: it will be hard to see any error with that RemovePII spam so let me know when you find the error [18:15:30] dmehus: it was a server/git pull error rather than anything else [18:15:34] Reception123: I see no test3 errors in graylog. [18:15:39] Reception123, ah [18:15:41] matomo upgraded now [18:15:41] Crepusculum: https://graylog.miraheze.org/search?q=%22test3wiki%22&rangetype=relative&relative=300 [18:15:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.78, 4.03, 3.21 [18:15:46] paladox: thanks :) [18:15:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::1c30/cpweb [18:16:00] PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hr.petrawiki.org could not be found [18:16:05] PROBLEM - www.sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sdiy.info could not be found [18:16:06] PROBLEM - history.estill.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.estill.org could not be found [18:16:11] oh whoops it's actually not on test3 itself, but the spam still is bad, even worse that it's on prod [18:16:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [18:16:34] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for antiguabarbudacalypso.com could not be found [18:16:34] PROBLEM - www.lukaibrisimovic.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lukaibrisimovic.org could not be found [18:16:35] PROBLEM - wiki.caker18.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.caker18.com could not be found [18:16:36] PROBLEM - planetnomads.speedbomb.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for planetnomads.speedbomb.de could not be found [18:16:39] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [18:16:40] PROBLEM - isv.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for isv.orain.org could not be found [18:16:40] PROBLEM - www.johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.johanloopmans.nl could not be found [18:16:42] PROBLEM - wiki.seredrau.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seredrau.xyz could not be found [18:16:43] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [18:16:43] PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found [18:16:44] PROBLEM - www.allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.allthetropes.org could not be found [18:16:55] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.025 second response time [18:17:22] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp12.miraheze.org [18:17:35] Reception123: see nothing worrisome when searching RemovePII on all streams. [18:17:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.71, 3.60, 3.15 [18:17:45] Reception123: where we stand now? [18:18:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JsxWI [18:18:21] [02miraheze/puppet] 07paladox 03f27625f - matomo: Deny access to /tmp and /lang [18:18:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:18:26] Crepusculum: did you try the link I sent you? [18:18:35] Yeah it found nothing. [18:18:38] RhinosF1: rebuild worked but there's still something up [18:18:52] https://usercontent.irccloud-cdn.com/file/jOTgw0M9/image.png [18:18:56] ^ Crepusculum this is what it looks like for me [18:19:12] Reception123: logs? [18:19:16] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [18:19:16] And just exclude them then [18:19:18] PROBLEM - spcodex.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for spcodex.wiki could not be found [18:19:19] PROBLEM - monarchists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for monarchists.wiki could not be found [18:19:19] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [18:19:21] PROBLEM - uncyclopediakr.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for uncyclopediakr.miraheze.org could not be found [18:19:22] RhinosF1: looking now [18:19:24] PROBLEM - en.phgalaxy.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.phgalaxy.xyz could not be found [18:19:27] PROBLEM - www.trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.trollpasta.com could not be found [18:19:28] PROBLEM - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.marinebiodiversitymatrix.org could not be found [18:19:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.86, 2.97, 2.97 [18:19:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:19:50] urgh it's like icinga intentionally knows we're busy and wants to annoy us with SSL warnings [18:20:04] hit the timeout i think [18:20:07] hmm mediawiki_wiki: test3wiki gives nothing at all [18:20:19] will try to run a php script instead [18:20:39] https://www.irccloud.com/pastebin/BGdA9bBX/ [18:20:40] [ Snippet | IRCCloud ] - www.irccloud.com [18:21:02] https://www.irccloud.com/pastebin/lQh4XWI7/ [18:21:03] [ Snippet | IRCCloud ] - www.irccloud.com [18:21:15] looks like the extensions are messed up? [18:21:18] Crepusculum: hmm, it is more widespread [18:21:28] but then how did extension message files and rebuild work [18:21:32] root@test3:/home/paladox# cd /srv/mediawiki/w/extensions/CategorySortHeader [18:21:41] paladox: yeah, though I don't get how/why [18:21:57] it seems like they weren't pulled correctly, as they don't have extension.json's even though 1.36 added them [18:22:14] Reception123: can you a git reset --hard origin then [18:22:33] Reception123: those logs for RemovePII shouldn't be worried about. There isn't even a job for it in production yet, must've been when testing job on test3 but it not being on production (so I used runJobs on test3) so it can't be run on production hence the logs, I'd assume. Shouldn't effect anything. [18:22:45] RECOVERY - hr.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - hr.petrawiki.org reverse DNS resolves to cp13.miraheze.org [18:22:52] I guess we need to sync extensions [18:22:54] RECOVERY - history.estill.org - reverse DNS on sslhost is OK: rDNS OK - history.estill.org reverse DNS resolves to cp13.miraheze.org [18:22:55] RECOVERY - www.sdiy.info - reverse DNS on sslhost is OK: rDNS OK - www.sdiy.info reverse DNS resolves to cp13.miraheze.org [18:22:58] Crepusculum: it's still spammy though [18:23:16] RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: rDNS OK - antiguabarbudacalypso.com reverse DNS resolves to cp13.miraheze.org [18:23:17] RECOVERY - www.lukaibrisimovic.org - reverse DNS on sslhost is OK: rDNS OK - www.lukaibrisimovic.org reverse DNS resolves to cp13.miraheze.org [18:23:19] RECOVERY - planetnomads.speedbomb.de - reverse DNS on sslhost is OK: rDNS OK - planetnomads.speedbomb.de reverse DNS resolves to cp13.miraheze.org [18:23:21] Reception123: yeah but I don't even see them in graylog. [18:23:23] RECOVERY - wiki.seredrau.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.seredrau.xyz reverse DNS resolves to cp13.miraheze.org [18:23:28] RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: rDNS OK - www.zenbuddhism.info reverse DNS resolves to cp13.miraheze.org [18:23:28] RECOVERY - isv.orain.org - reverse DNS on sslhost is OK: rDNS OK - isv.orain.org reverse DNS resolves to cp13.miraheze.org [18:23:29] RECOVERY - www.johanloopmans.nl - reverse DNS on sslhost is OK: rDNS OK - www.johanloopmans.nl reverse DNS resolves to cp13.miraheze.org [18:23:31] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org [18:23:33] RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - pt.graalmilitary.com reverse DNS resolves to cp13.miraheze.org [18:23:33] RECOVERY - wiki.caker18.com - reverse DNS on sslhost is OK: rDNS OK - wiki.caker18.com reverse DNS resolves to cp13.miraheze.org [18:23:34] RECOVERY - www.allthetropes.org - reverse DNS on sslhost is OK: rDNS OK - www.allthetropes.org reverse DNS resolves to cp13.miraheze.org [18:23:38] Weird [18:23:56] RhinosF1: I mean I could but it'd be nice to know the cause so we don't have any surprises when we do prod [18:24:44] Reception123: for prod tbh we can just delete w [18:24:47] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.85, 7.35, 7.88 [18:24:47] Reception123: did you clone it like git clone https://github.com/miraheze/mediwiki --branch REL1_36 w --recursive? [18:24:48] [ Page not found · GitHub · GitHub ] - github.com [18:24:49] That works [18:25:01] paladox: let puppet update the branch afaik [18:25:07] oh [18:25:17] I guess puppet not that smart [18:25:41] Reception123: just do a reclone [18:25:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.15, 6.84, 7.78 [18:25:50] extensions fixed i think [18:25:54] Oh right [18:25:56] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org [18:25:59] RECOVERY - spcodex.wiki - reverse DNS on sslhost is OK: rDNS OK - spcodex.wiki reverse DNS resolves to cp13.miraheze.org [18:26:00] But for prod we can do a reclone [18:26:00] RECOVERY - monarchists.wiki - reverse DNS on sslhost is OK: rDNS OK - monarchists.wiki reverse DNS resolves to cp13.miraheze.org [18:26:04] paladox: oh, what did you do? [18:26:11] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp13.miraheze.org [18:26:12] Still 500s [18:26:17] RECOVERY - www.trollpasta.com - reverse DNS on sslhost is OK: rDNS OK - www.trollpasta.com reverse DNS resolves to cp13.miraheze.org [18:26:18] i deleted extensions/** and reset and then git submodule updated [18:26:20] RECOVERY - uncyclopediakr.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - uncyclopediakr.miraheze.org reverse DNS resolves to cp13.miraheze.org [18:26:22] RECOVERY - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: rDNS OK - www.marinebiodiversitymatrix.org reverse DNS resolves to cp13.miraheze.org [18:26:23] RECOVERY - en.phgalaxy.xyz - reverse DNS on sslhost is OK: rDNS OK - en.phgalaxy.xyz reverse DNS resolves to cp13.miraheze.org [18:26:26] Fatal error: Uncaught Exception: Unable to open file /srv/mediawiki/w/extensions/RightFunctions/extension.json: filemtime(): stat failed for /srv/mediawiki/w/extensions/RightFunctions/extension.json in /srv/mediawiki/w/includes/registration/ExtensionRegistry.php:177 [18:26:30] but origin/REL1_36 doesn't exist? [18:26:30] ^ paladox now we're back to this [18:26:47] wrong branch checked out [18:26:48] let me fix [18:26:51] ah [18:28:48] oh hold on [18:28:54] i'm just gonna reclone [18:29:34] Yeah [18:30:53] PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.025 second response time [18:32:52] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.131 second response time [18:33:26] paladox: are you doing composer & i18n or do you want reception to take back? [18:33:35] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.22, 7.29, 7.44 [18:33:42] I can do that if you want, i don't want to step on any toes :) [18:34:28] paladox: if you're free go ahead [18:34:30] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:34:33] sure [18:34:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.54, 7.42, 7.52 [18:34:45] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:35:00] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:35:04] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:35:13] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:35:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 7.15, 7.37 [18:36:21] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:36:38] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.14, 6.63, 7.22 [18:38:51] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 1270 bytes in 0.017 second response time [18:39:03] Reception123, and RhinosF1, `exambotwiki` finally hit Special:DeletedWikis this morning :) [18:39:27] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [18:41:04] heh [18:41:28] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.85, 5.93, 6.70 [18:41:44] Fatal error: Uncaught ExtensionDependencyError: EventLogging requires EventBus to be installed. [18:41:52] paladox: it's the 404 error [18:41:59] 404 error? [18:42:00] so must be that CreateWiki cache one I think [18:42:08] yeah, try to access test3wiki [18:42:20] paladox: yup /cache is empty [18:42:25] https://www.irccloud.com/pastebin/MYp7P50Q/ [18:42:25] [ Snippet | IRCCloud ] - www.irccloud.com [18:42:32] i don't see it on my laptop either [18:42:52] PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2560 bytes in 0.011 second response time [18:43:02] paladox: do we need to install it then? [18:43:13] Reception123: first blocker https://phabricator.wikimedia.org/T283528 ! [18:43:14] [ ⚓ T283528 $phpVersionCheck->checkVendorExistence(); message does not include $wgScriptPath in URL ] - phabricator.wikimedia.org [18:43:15] i'm thinking yes [18:43:19] gonna check tho [18:43:46] heh [18:43:51] We're not upgrading to php 8.0 or whatever at the same time, are we? [18:43:56] * Reception123 forgot what the main file in cache is that gets it to work [18:44:01] dmehus: no, that would be too much work at once [18:44:02] https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/21f985fa98256e6a38c20ef33dced5b581d5c3e2 says yeh [18:44:02] [ Use EventBus to send events for EventLogging PHP client · wikimedia/mediawiki-extensions-EventLogging@21f985f · GitHub ] - github.com [18:44:04] we'll do that after [18:44:10] Reception123, yeah definitely [18:44:32] paladox: master doesn't require eventbus for EventLogging only REL1_36 does. Wonder why. [18:44:49] dmehus: php 8 will be a while off [18:45:07] * dmehus still remembers the switch from php 3 to php 4 and all the web file extensions changed from `.php3` to `.php` [18:45:07] copy/pasted test3wiki.json into the cache, but I don't remember if that's enough to fix [18:45:08] https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/2acaa7987b73dda1ee46e36f8fadfef8add6f0ae [18:45:08] [ Restore dependency-free nature of this extension · wikimedia/mediawiki-extensions-EventLogging@2acaa79 · GitHub ] - github.com [18:45:12] RhinosF1, ack [18:45:19] i'll backport [18:45:33] Thanks [18:45:41] dmehus: the morale of the story is, a MediaWiki update never works out well from the start! [18:45:48] there will always be something [18:45:56] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/693656 [18:46:04] Reception123, heh. Does Paladox have full merge access on WMF? [18:46:21] RhinosF1: do you have any contacts to get that merged ASAP? [18:46:28] paladox: yes [18:46:41] Lego may be around [18:46:42] great! Could you ask them to merge it please :) [18:46:52] dmehus: the message above answers that question [18:46:54] paladox: you should [18:47:03] Request cr+2 on mediawiki/* [18:47:08] yeah, things like phabricator and icingaweb aren't even fully compatible with php8 yet. [18:47:20] I asked Martin [18:47:26] thanks! [18:47:35] i don't think i would be approved cr+2 [18:47:40] MacFan4000: oh really? Are there any plans to do that soon? [18:47:43] Reception123, yep heh [18:47:57] dmehus: we will be updating to the new debian version soon though [18:48:04] Reception123: they will be [18:48:04] MacFan4000, ah [18:48:06] But definitely not before 1.36 is completely done [18:48:10] Reception123, yeah [18:48:13] https://github.com/Icinga/icingaweb2/commit/4bc5350ebaaeb7a60bbe20829a1d72def343a921 [18:48:13] Reception123: that'll be just 7. Whatever [18:48:13] [ Merge pull request #4289 from Icinga/bugfix/php8-4287 · Icinga/icingaweb2@4bc5350 · GitHub ] - github.com [18:48:24] paladox: why not? [18:48:29] icingaweb2 supports php8 now but there's no actual released version [18:48:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.40, 6.48, 7.80 [18:48:47] Zppix: because it's a very high barrier [18:48:54] Zppix: i'm not too active MW developing side. [18:49:06] icingaweb2 support will be in their next release, and phabricator there's https://secure.phabricator.com/T13588 but upstream are in no hurry [18:49:06] [ ⚓ T13588 PHP 8 Compatibility ] - secure.phabricator.com [18:52:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.19, 7.33, 7.08 [18:52:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [18:53:17] MacFan4000, ah [18:54:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.38, 7.06, 7.02 [18:56:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.19, 6.15, 6.70 [18:58:02] PROBLEM - wiki.umbrel.tech - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.umbrel.tech' expires in 7 day(s) (Tue 01 Jun 2021 18:51:21 GMT +0000). [18:59:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.84, 6.68, 7.88 [19:00:25] Shouldnt ssl be renewed at 15 days or something how’d it get to 7 [19:02:08] That's why it's critical [19:02:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.46, 5.90, 6.79 [19:03:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.18, 6.87, 7.65 [19:05:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.16, 6.68, 6.50 [19:05:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.82, 6.79, 7.52 [19:08:16] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [19:08:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.50, 6.26, 6.40 [19:11:37] paladox: can you see log spam here: https://graylog.miraheze.org/search?q=%22test3wiki%22&rangetype=relative&relative=900 ? [19:11:44] Crepusculum says he can't but I see RemovePII spam [19:11:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.41, 5.47, 6.71 [19:12:09] yeh i can, he won't because that's an "All stream" [19:12:19] mw team only have limited access to specific streams [19:12:27] [02mw-config] 07antonmiraheze opened pull request 03#3918: Patch 1 - 13https://git.io/Jsxzm [19:12:29] ahh I know why he can't, because it's syslog yeah [19:13:10] [02mw-config] 07antonmiraheze edited pull request 03#3918: Responsive Vector variables (1.36) - 13https://git.io/Jsxzm [19:15:57] paladox, I just realized we haven't setup miraheze-github alerts yet [19:16:16] alerts? [19:16:23] well like the notifications [19:16:45] on Freenode, we had Not-4c5b and miraheze-github that updated us if the build was passing or failing [19:17:28] Crepusculum: maybe you can help set that up again? :) [19:21:23] We have [19:21:43] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [19:21:57] They don't run on forks for first time users immediately [19:22:07] And it's his first config pr [19:22:13] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.75, 1.55, 1.23 [19:22:28] miraheze/mw-config - RhinosF1 the build has errored. [19:22:50] dmehus: there ^ [19:23:04] Not sure why says me not anton [19:23:10] But I approved him [19:24:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.27, 1.47, 1.24 [19:25:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.26, 6.78, 6.57 [19:26:52] dmehus: it is setup. [19:29:52] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.13, 6.65, 6.61 [19:30:22] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [19:33:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.60, 7.43, 6.91 [19:35:09] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.62, 7.48, 6.56 [19:35:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.58, 8.04, 6.82 [19:37:08] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.01, 6.82, 6.42 [19:37:25] Crepusculum, RhinosF1, paladox, oh right, puppet repo just doesn't have miraheze-github CI notifications [19:37:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.12, 7.27, 6.69 [19:39:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.37, 7.61, 6.87 [19:41:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.07, 7.24, 6.64 [19:41:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 7.22, 6.52 [19:41:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.89, 6.93, 6.71 [19:41:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.04, 7.69, 7.37 [19:43:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.28, 7.08, 6.64 [19:43:14] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.94, 6.69, 6.41 [19:43:47] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [19:43:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.18, 6.78, 6.69 [19:43:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.04, 8.16, 7.55 [19:45:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.55, 6.10, 6.34 [19:45:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.19, 7.43, 7.37 [19:49:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.71, 8.33, 7.66 [19:51:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±2] 13https://git.io/Jsxro [19:51:54] [02miraheze/mediawiki] 07Universal-Omega 03ad752cb - Switch to fork for AutoCreatePage on REL1_36 branch [19:51:55] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [19:52:32] [02mediawiki] 07Universal-Omega opened pull request 03#1961: Switch to fork for AutoCreatePage on REL1_36 branch - 13https://git.io/Jsxry [19:54:14] [02mediawiki] 07Universal-Omega closed pull request 03#1961: Switch to fork for AutoCreatePage on REL1_36 branch - 13https://git.io/Jsxry [19:54:16] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±2] 13https://git.io/Jsxrj [19:54:17] [02miraheze/mediawiki] 07Universal-Omega 03abc8804 - Switch to fork for AutoCreatePage on REL1_36 branch (#1961) [19:54:19] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [19:54:20] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:56:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.35, 6.95, 6.62 [19:57:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.89, 7.39, 7.09 [19:57:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 7.25, 7.03 [19:58:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.14, 6.33, 6.44 [20:01:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.00, 6.09, 6.61 [20:01:36] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.39, 7.55, 7.19 [20:01:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.45, 7.93, 7.95 [20:02:13] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 4.13, 2.26, 1.43 [20:03:41] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JsxKE [20:03:42] [02miraheze/mediawiki] 07paladox 03cd9ea19 - Update EventLogging [20:03:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.96, 8.18, 8.03 [20:04:44] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.532 second response time [20:04:51] RhinosF1: we can move forward, got merged ^ [20:04:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 7.35, 7.00 [20:06:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.81, 1.63, 1.38 [20:06:25] 2021-05-24 20:05:49 test3 test3wiki: [c6ea263f98da317d712089aa] /wiki/Main_Page Error: Call to undefined method MagicWord::get() [20:06:46] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20074 bytes in 1.538 second response time [20:06:49] ah https://github.com/staspika/mediawiki-numberedheadings/pull/3 [20:06:50] [ MagicWord -> MagicWordFactory by paladox · Pull Request #3 · staspika/mediawiki-numberedheadings · GitHub ] - github.com [20:06:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.42, 7.31, 7.03 [20:07:10] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:07:30] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JsxKN [20:07:32] [02miraheze/mediawiki] 07paladox 0319ca7e9 - Update NumberedHeadings to ada69e7fd80bc01bcfd518588559a4ff23f51834 [20:07:46] paladox: thanks for merging, and it seems like graylog still isn't working for test3 [20:08:25] 2021-05-24 20:08:04 test3 test3wiki: [0ae4325a7dbfb7961caf6341] /wiki/Main_Page Error: Call to undefined function MediaWiki\Extension\Interwiki\Hooks() [20:08:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 7.51, 6.91 [20:10:05] urgh, an upstream error with the Interwiki extension? [20:10:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.21, 6.32, 6.70 [20:11:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.60, 7.40, 7.60 [20:11:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.16, 7.26, 7.83 [20:13:06] Reception123: fun [20:13:33] https://github.com/wikimedia/mediawiki-extensions-Interwiki/commit/b21bafe3858bc97da050280aa5d26ff826ac5611 [20:13:33] [ Don't register InterwikiLoadPrefix hook unless it is needed · wikimedia/mediawiki-extensions-Interwiki@b21bafe · GitHub ] - github.com [20:13:38] could it be this RhinosF1 ? [20:14:17] What's using it paladox [20:14:39] https://www.irccloud.com/pastebin/i86MQyjL/ [20:14:39] [ Snippet | IRCCloud ] - www.irccloud.com [20:15:17] Yes but that commit says should be nothing [20:15:20] removing that at least got the site working [20:15:46] Can you upstream fix? [20:16:21] I still see Original exception: [9486f8c3beab96a4ed600df8] 2021-05-24 20:15:52: Fatal exception of type "Error" [20:16:48] i don't know what the fix is [20:22:12] Oh no, that's our MiraehzeMagic pseudo-interwiki prefix, isn't it? [20:22:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.34, 6.21, 6.64 [20:23:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.17, 6.02, 6.73 [20:29:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.74, 6.02, 6.63 [20:33:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.25, 7.15, 6.66 [20:35:18] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 7.25, 6.77 [20:35:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.01, 6.88, 6.76 [20:37:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.30, 6.26, 6.47 [20:37:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.64, 6.83, 6.76 [20:38:55] dmehus: not sure what yet but doubt it [20:39:31] paladox: full trace? [20:40:08] https://phabricator.wikimedia.org/P16180 [20:40:09] [ ✎ P16180 (An Untitled Masterwork) ] - phabricator.wikimedia.org [20:41:57] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.04, 6.76, 6.79 [20:44:33] RhinosF1, ah [20:48:02] dmehus: looks to be a core issue [20:52:08] dmehus: yeah it's utterly upstream [20:54:29] !log test3: 4 hours later, we're catching many bugs. [20:54:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:23] paladox: will schema changes via sql.php work and can I do them? [21:04:39] i think they will [21:05:54] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.22, 7.77, 6.65 [21:06:18] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [21:07:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 6.98, 6.50 [21:08:09] paladox: please shout if db gets unhappy [21:08:16] ok [21:09:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.83, 6.16, 6.25 [21:10:11] https://www.irccloud.com/pastebin/UzDC2IWH/ [21:10:12] [ Snippet | IRCCloud ] - www.irccloud.com [21:10:21] various warnings [21:11:30] RhinosF1: I already fixed CentralNotice DB errors, I made a local change to test3 to change the central database to test3wiki rather then metawiki also. [21:12:02] Crepusculum: I have 2 sql patches to run [21:12:18] !log test3wiki: run /srv/mediawiki/w/extensions/OAuth/schema/oauth2_allowed_grants.sql [21:12:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:12] !log test3wiki: run /srv/mediawiki/w/extensions/OAuth/schema/oauth2_is_confidential.sql [21:14:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:06] !log test3wiki: run /srv/mediawiki/w/extensions/OAuth/schema/oauth_version_accepted.sql [21:15:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:56] Crepusculum: find out what DB 2FA uses please? [21:16:18] Crepusculum: i don't think null is an accepted value hence those errors [21:17:42] !log test3wiki: run /srv/mediawiki/w/extensions/CentralNotice/patches/patch-notice-type-log.sql [21:17:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:52] RhinosF1: already did CN. [21:18:18] To fix all resource module stuff not working. [21:18:43] paladox: it has before I think, but maybe not. It is the extension default though. [21:19:01] Crepusculum: please log stuff [21:19:36] RhinosF1: why is logging necessary for test3? I didn't think it was. But will do. [21:19:47] Crepusculum: for schema changes yes [21:19:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.45, 7.06, 6.47 [21:19:56] i think that we should use the wmg configs then convert it from false to null [21:20:00] outside of MWS [21:21:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.73, 7.05, 6.56 [21:23:17] paladox: can you take a dump of mhglobal.oathauth_users ? [21:23:31] Uh, do not touch that table [21:23:34] it's production [21:23:46] paladox: Reedy said it would work [21:23:51] oh [21:23:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.00, 6.36, 6.37 [21:24:03] 22:15:10 RhinosF1: Should be... I'm pretty sure the code is in REL1_35 that would do that organically (ie when users enabled 2FA or caused another update) [21:24:17] but yes that's why I want a dump [21:24:53] !log [universalomega@test3] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/extensions/CentralNotice/patches/patch-notice-type-log.sql [21:24:53] !log [universalomega@test3] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/extensions/CentralNotice/patches/patch-notice_not_type.sql [21:24:53] !log change wgCentralDBname on test3 to test3wiki as to not touch metawiki database while testing schema changes for 1.36 [21:24:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:57] RhinosF1: dumped now [21:24:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:25:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:25:46] !log (affects all wikis) rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/OATHAuth/maintenance/updateTOTPScratchTokensToArray.php --wiki=test3wiki [21:25:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:26:20] Crepusculum: ext wise just wb_terms left [21:26:33] RhinosF1: Great! [21:26:46] wonder why my theme doesn't work on test3 [21:26:53] as in i picked timeless but it's using vector? [21:27:02] paladox: yeah same, my skin preference is not working [21:27:52] only works with `?useskin` the preference is being ignored. [21:28:17] RhinosF1: do we have a backup of test3 DB? [21:28:33] Crepusculum: yes [21:28:47] It's blue for me [21:29:07] RhinosF1: same, why is vector blue is another question. [21:29:24] vector blue? [21:29:25] and great will do the DROP then as we need to make sure it works. [21:29:34] paladox: yeah, vector is blue for me. [21:29:36] Crepusculum: ok, core schemas coming up [21:29:48] it's not blue for me [21:29:58] take a pic for me [21:30:10] oh [21:30:12] logged out [21:30:17] logged in, it's grey [21:30:17] wtf [21:30:41] paladox: I am logged in and it is blue. [21:30:52] it's not blue for me [21:31:04] hold on... could it be miss identifying users or something [21:31:06] this is weird [21:31:23] paladox: that even happens every once in awhile on 1.35, as in sometimes on Special:Preferences it was, then reloading and it is fixed. [21:31:32] but now it is constent [21:31:57] hmm [21:35:13] !log [universalomega@test3] DROP TABLE wb_terms; [21:35:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:31] where is it even loading blue vector [21:35:33] which skin/ [21:35:43] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.64, 7.89, 6.90 [21:35:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.13, 6.84, 6.36 [21:36:06] paladox: any ideas about these? https://www.irccloud.com/pastebin/cgMZNTe4/ [21:36:07] [ Snippet | IRCCloud ] - www.irccloud.com [21:36:14] no [21:36:27] ok, thanks anyways then. [21:36:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.66, 7.20, 6.61 [21:36:50] i cannot find the blue vector theme wtf [21:36:55] where is it getting it from [21:37:05] also, good question, no idea, though I do wonder if it is possible the fallback vector maybe? Only thing I can think of. [21:37:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.55, 7.35, 6.82 [21:37:47] !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-redirect-rd_title-varbinary.sql [21:37:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.93, 6.77, 6.41 [21:37:54] it isn't blue for me either [21:38:14] !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-pagelinks-pl_title-varbinary.sql [21:38:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:38] !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-templatelinks-tl_title-varbinary.sql [21:38:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:04] !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-imagelinks-il_to-varbinary.sql [21:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:29] !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-langlinks-ll_title-varbinary.sql [21:39:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.56, 6.75, 6.66 [21:40:59] MacFan4000: how it looks for me. https://usercontent.irccloud-cdn.com/file/93uPVh8L/image.png [21:41:14] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-iwlinks-iwl_title-varbinary.sql [21:41:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-category-cat_title-varbinary.sql [21:41:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-querycache-qc_title-varbinary.sql [21:41:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-querycachetwo-qcc_title-varbinary.sql [21:41:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-watchlist-wl_title-varbinary.sql [21:41:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-user_newtalk-user_last_timestamp-binary.sql [21:41:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-protected_titles-pt_title-varbinary.sql [21:41:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:43:02] !log all previous on test3 [21:43:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:43:37] found it [21:43:38] https://github.com/wikimedia/mediawiki-extensions-Theme [21:43:39] [ GitHub - wikimedia/mediawiki-extensions-Theme: Github mirror of MediaWiki extension Theme - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contri ] - github.com [21:43:42] RhinosF1: Crepusculum [21:44:00] Crepusculum: Go into your settings under appearance and check the theme setting [21:44:16] it's likely set to dark [21:44:38] MacFan4000: night time? [21:44:46] MacFan4000: "The skin you are currently using does not support themes. Please use another skin if you wish to use a non-default theme" [21:45:03] uninstalling it doesn't fix the theme you selected from showing [21:45:20] for me setting it to dark makes it look like it does in the screenshot [21:45:26] my skin is set to vector [21:45:54] MacFan4000: oh, my skin is not even set to vector, but vector is the skin it is using. [21:46:21] what skin are you set to? [21:46:41] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-ipblocks_restrictions-ir_type.sql [21:46:41] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-watchlist-namespace_title-rename-index.sql [21:46:41] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-job-job_title-varbinary.sql [21:46:41] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-job_job_timestamp.sql [21:46:41] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-job_job_token_timestamp.sql [21:46:42] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-watchlist-wl_notificationtimestamp.sql [21:46:42] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-slot_roles-role_id.sql [21:46:43] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-content_models-model_id.sql [21:46:43] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-categorylinks-cl_to-varbinary.sql [21:46:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:44] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-logging-log_title-varbinary.sql [21:46:44] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-uploadstash-us_timestamp.sql [21:46:45] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-user_properties-rename-index.sql [21:46:45] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-sites-rename-indexes.sql [21:46:46] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-logging-rename-indexes.sql [21:46:46] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-filearchive-fa_name.sql [21:46:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:49] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JsxbJ [21:46:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:51] [02miraheze/mediawiki] 07paladox 0358cd09f - Update Interwiki [21:46:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:27] MacFan4000: Cosmos, but just tried changing to apex, did not work, also tried Timeless as well, and MonoBook, nothing works, and always shows vector. [21:48:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.20, 6.56, 6.64 [21:48:35] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [21:49:05] paladox: does that fix interwiki issue then? [21:49:11] yeh [21:49:15] Great! [21:49:17] if I change my skin in settings I always see vector, but if I use the preview link (?useskin in the url) then it works fine [21:49:18] thanks! [21:49:27] MacFan4000: same. [21:49:36] Crepusculum: do you have 1.35 checked out locally? [21:49:44] if so could you deploy https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseQualityConstraints/+/693655 pls [21:49:53] paladox: https://phabricator.miraheze.org/T7357 [21:49:54] [ ⚓ T7357 Apply test3 drop default schema changes ] - phabricator.miraheze.org [21:50:10] I don't have files for hem [21:50:12] paladox: no (just 1.36 right now), but it will take me like 5 minutes to do, so will do. [21:50:12] them [21:50:18] RhinosF1: you can easily drop them by creating the files [21:50:26] you can at least make it normal vector by setting your skin to vector and then theme to default [21:50:30] paladox: I'm pants at sql [21:50:33] oh ok [21:51:04] currently busy looking at this theme issue. [21:51:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 6.90, 6.45 [21:53:36] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.42, 6.60, 6.39 [21:54:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-oldimage-oi_name-varbinary.sql [21:54:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-objectcache-exptime-notnull.sql [21:54:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-archive-rename-name_title_timestamp-index.sql [21:54:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-image-img_name-varbinary.sql [21:54:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-image-img_timestamp.sql [21:54:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-site_identifiers-rename-indexes.sql [21:54:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-recentchanges-rc_title-varbinary.sql [21:54:22] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-recentchanges-rc_timestamp.sql [21:54:22] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-recentchanges-rc_id.sql [21:54:23] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-recentchanges-rc_new_name_timestamp.sql [21:54:23] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-archive-ar_title-varbinary.sql [21:54:24] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-user_table-updates.sql [21:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:35] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jsxbx [21:54:36] [02miraheze/mediawiki] 07Universal-Omega 03c2a07b6 - Update WikibaseQualityConstraints [21:54:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:51] !log test3 schema changes with sql files done! \o/ See https://phabricator.miraheze.org/T7357 for remaining ones [21:54:52] [ ⚓ T7357 Apply test3 drop default schema changes ] - phabricator.miraheze.org [21:54:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:38] paladox: updated [21:56:03] thanks! [21:56:07] No problem. [21:56:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.08, 6.92, 6.77 [21:58:08] Crepusculum: 62e7657b3817c83b58620968 [21:58:16] looking [21:58:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.53, 6.47, 6.62 [21:58:53] checking out 1.35 branch for GP didn't fix the issue [21:59:40] RhinosF1 https://www.irccloud.com/pastebin/JZrerdvt/ [21:59:41] [ Snippet | IRCCloud ] - www.irccloud.com [22:00:12] 2021-05-24 21:59:45 test3 test3wiki: [ebbe6c285df4c57e91096fcc] /w/load.php?lang=en-gb&modules=startup&only=scripts&raw=1&skin=timeless ErrorException: Use of ParserSectionCreate hook (used in HideSectionHooks::onParserSectionCreate) was deprecated in MediaWiki 1.35. [Called from MediaWiki\HookContainer\HookContainer::run] [22:00:59] paladox: upstream? [22:01:10] Crepusculum: uh oh [22:01:11] yeh, that can be filed upstream [22:01:17] Crepusculum: that was me [22:01:23] i was testing some changes with GP [22:01:25] oh [22:01:41] 2021-05-24 21:59:45 test3 test3wiki: [ebbe6c285df4c57e91096fcc] /w/load.php?lang=en-gb&modules=startup&only=scripts&raw=1&skin=timeless ErrorException: Use of InternalParseBeforeSanitize hook (used in VariablesHooks::onInternalParseBeforeSanitize) was deprecated in MediaWiki 1.35. [Called from MediaWiki\HookContainer\HookContainer::run] [22:01:53] 2021-05-24 21:59:45 test3 test3wiki: [a10b0b26a0744ff61516467d] /w/index.php?title=Main_Page&useskin=timeless ErrorException: Use of SkinTemplateOutputPageBeforeExec hook (used in HideSectionHooks::onSkinTemplateOutputPageBeforeExec) was deprecated in MediaWiki 1.35. [Called from MediaWiki\HookContainer\HookContainer::run] [22:03:05] paladox: SkinTemplateOutputPageBeforeExec hook is going to be hard to fix on some extensions, such as LanguageSelector. I tried awhile ago when converting that to extensionregistry but could not find a replacement hook to do what it needs to do. [22:03:13] those warnings are not affecting the skins [22:03:20] so i'm not sure why this is happening [22:03:24] oh [22:03:45] 2021-05-24 21:59:42 test3 test3wiki: [deeff36ed2036061727d9758] /w/api.php?action=themelist&format=json ErrorException: Use of Skin::getSkinNames was deprecated in MediaWiki 1.36. [Called from ApiThemeList::appendThemes] [22:03:59] paladox: none of the timeless look new [22:04:54] 2021-05-24 21:59:36 test3 test3wiki: [5adaef72c8648e1f7f2d19e4] /wiki/Special:Preferences ErrorException: PHP Deprecated: A ParserGetVariableValueSwitch hook handler bypassed the cache, this is deprecated since MediaWiki 1.35 [22:05:27] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.64, 7.06, 6.49 [22:05:28] It does not say which extension paladox, or where from? [22:05:42] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 7.18, 6.54 [22:05:50] paladox: most of them say 1.35 [22:06:11] https://phabricator.miraheze.org/P435 [22:06:12] [ ✎ P435 (An Untitled Masterwork) ] - phabricator.miraheze.org [22:06:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.61, 7.16, 6.79 [22:07:24] yeh guess they changed the exception or something? Unless it's because we have the log level differently to what we have it now to log local [22:07:35] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JsxAc [22:07:36] [02miraheze/mediawiki] 07Universal-Omega 036b3df73 - Update FancyBoxThumbs [22:07:37] (using debug here for the local logging as that's the default) [22:07:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.65, 7.31, 6.67 [22:07:54] paladox: since 1.35 ain't an issue [22:07:57] I filed https://phabricator.wikimedia.org/T283547 [22:07:58] [ ⚓ T283547 ErrorException: Use of Skin::getSkinNames was deprecated in MediaWiki 1.36. [Called from ApiThemeList::appendThemes] ] - phabricator.wikimedia.org [22:08:01] ok [22:08:02] for the 1.36 one [22:08:44] RhinosF1: they are an issue on 1.36 though, they were soft deprecated on 1.35 and hard on 1.36, meaning constant error logs on 1.36 [22:08:51] paladox: my main concern is the theme issue though [22:08:54] Crepusculum: oh [22:08:57] yeh [22:09:00] file tasks then [22:09:02] that's what i'm trying to find out [22:09:14] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JsxA6 [22:09:16] [02miraheze/mediawiki] 07Universal-Omega 03006617d - Update FancyBoxThumbs [22:09:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.47, 7.44, 6.79 [22:09:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.12, 7.62, 6.85 [22:09:36] Crepusculum: Can you go on a task filing spree [22:09:53] add them to the list on https://phabricator.miraheze.org/T7117 [22:09:55] [ ⚓ T7117 Upgrade to 1.36 ] - phabricator.miraheze.org [22:10:12] RhinosF1: I will just submit patches upstream when possible. But for ones I can't I will try to file tasks yeah. [22:10:25] Crepusculum: as long as each blocker is noted [22:10:33] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:10:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.36, 6.64 [22:11:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 7.69, 6.97 [22:11:44] RhinosF1: not all extension deprecation notices will be blockers, as long as we cut down on some, we still have some warning from 1.34 deprecations also (maybe even some from before that) [22:12:00] Crepusculum: https://phabricator.miraheze.org/T7117 [22:12:01] [ ⚓ T7117 Upgrade to 1.36 ] - phabricator.miraheze.org [22:12:04] but I will note which ones warns yeah. [22:12:07] currently noted them all [22:12:17] but we can evaluate as time goes on [22:12:21] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:12:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.09, 7.70, 7.20 [22:12:36] let's try and get tasks upstream at least or patches [22:12:41] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:12:43] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 6.72, 6.50 [22:12:48] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:12:58] I fixed FancyBoxThumbs issue which caused JS notices on both 1.35 and 1.36 (and updated) by the way. [22:13:01] Crepusculum: the skin being blue is priority though [22:13:05] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:13:12] RhinosF1: I fixed my blue theme [22:13:17] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.28, 6.58, 6.61 [22:13:40] Crepusculum: I see it logged out [22:14:30] https://www.mediawiki.org/wiki/Manual:Hooks/ParserSectionCreate [22:14:30] [ Manual:Hooks/ParserSectionCreate - MediaWiki ] - www.mediawiki.org [22:14:34] says no replacement hook? [22:14:59] Fixed theme RhinosF1 [22:15:04] and paladox ^ [22:15:11] oh? [22:15:14] Crepusculum: how? [22:15:14] how did you fix it? [22:15:21] it was just a setting in Special:ManageWiki/settings [22:15:26] oh [22:16:51] Crepusculum: why did it only effect 1.36 though? [22:16:59] It didn't. [22:17:01] That doesn't solve the issue of not. being able to use a skin other then vector outside of previewing skins [22:17:05] Can we ensure it won't hit prod [22:17:06] paladox: the issue remains though, the skin preference is not being honored, always shows vector no matter what, that is for sure needing fixed. [22:17:17] ok, yeh [22:17:24] i'm not sure how we fix that [22:17:32] i've looked at a few things and cannot figure out why [22:17:52] any updates on what caused last nights outage yet? [22:17:55] RhinosF1: it was set to default to that on test3, I guess it did effect 1.35 also. Either that or theme was fixed to honor the setting. [22:18:37] Crepusculum: it's always legacy vector too [22:18:54] I can't even get new to work [22:19:09] RhinosF1: in that case, I have an idea. [22:19:23] thats not a bug its a feature (re new vector [22:19:44] Zppix: getting rid of new vector would be good [22:20:06] anyway sleep [22:20:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.39, 6.41, 6.69 [22:20:57] but ya know Vector is a commonly used skin [22:21:02] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.76, 6.61, 6.75 [22:21:40] Zppix: the issue is no skin preferences are working, always using vector, not even enabling the new vector works. Not that new vector is there. [22:22:02] Crepusculum: that would be a core issue [22:22:09] you can change the theme however [22:22:15] wonder if caused by https://github.com/wikimedia/mediawiki-extensions-GlobalPreferences/commit/dde39ef3d65a740f9c7d9ef90cdbaab021884e69#diff-b51349da1d5cc80bb73cbe51a79b96b2677397a1aa63854e8a5fa3fdf74fb224 [22:22:16] [ Remove GlobalPreferencesFactory::setUser() · wikimedia/mediawiki-extensions-GlobalPreferences@dde39ef · GitHub ] - github.com [22:23:23] but then doesn't explain why it works on wikimedia? [22:23:41] Could it be one of our config/code hacks [22:23:51] paladox: I don't think it is GlobalPreferences extension is it? I set a local exception and still does not work. [22:24:14] local exception would still be the extension tho [22:24:19] setting wgDefaultSkin works to change it also. [22:24:37] Crepusculum: could try disable global prefs on test3 and see if it does it [22:25:07] Zppix: ok, I guess we could try. paladox: sound ok to you? [22:25:33] Yeh [22:25:37] @zppix are you rejoining system admin? [22:25:48] No [22:26:03] Just bored [22:26:20] so you basically got bored of it? [22:26:40] doesn't work [22:26:44] paladox: disabled on test3 [22:27:18] yeah same issue [22:27:21] re-enabling [22:27:47] Cocopuff2017: no im just bored atm [22:28:00] Nice. [22:28:06] well i disabled the vector skin but somehow https://test3.miraheze.org/wiki/Main_Page is working??? [22:28:07] [ Internal error - Test3 ] - test3.miraheze.org [22:28:22] paladox: now it is fallback vector.' [22:28:33] oh [22:29:11] paladox: I set wgFallbackSkin awhile ago to that do to the issue where if skin is disabled it causes issues. so it now falls back to vector. [22:29:20] *due [22:29:21] oh [22:29:22] #2 /srv/mediawiki/w/skins/Tweeki/includes/Hooks.php(52): ContextSource->getSkin() [22:29:43] got it working [22:29:45] the cause is Tweeki [22:29:55] oh. [22:30:08] Tweeki needs a little tweeki [22:30:19] Zppix: too many times [22:30:29] It didn't work for a long while on mobile I think [22:31:04] RhinosF1: yeah it didn't [22:31:21] Crepusculum: candidate for bin? [22:32:10] yeah. [22:32:22] Just do it then [22:32:51] Only on 1.36 branches though [22:34:01] is there any fix we can do to the skin [22:34:45] paladox: I think it in particular just has had to many past issues, but it is popular so may be worth trying to fix upstream. No idea how though. [22:34:50] It's broke a few times though [22:34:57] ok [22:35:07] Unless we can guarntee maintenance or port it to something less breakable [22:35:19] Then it's probably it's time [22:35:22] We should probably open an issue on https://github.com/thaider/Tweeki/issues [22:35:23] [ Issues · thaider/Tweeki · GitHub ] - github.com [22:35:51] RhinosF1: agreed. [22:36:05] MacFan4000: there's issues dating back to Feb 2015 [22:36:28] Crepusculum: you have any desire to deal with it. If not, you know what to do. [22:36:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.79, 7.43, 6.80 [22:36:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.44, 6.85, 6.51 [22:36:38] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.53, 6.82, 6.32 [22:36:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.35, 7.19, 6.66 [22:38:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.92, 7.25, 6.81 [22:38:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.32, 6.82, 6.56 [22:38:44] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:38:53] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:39:01] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:40:52] RhinosF1: yeah. I do want to see what Reception123 thinks first though. [22:40:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.50, 7.55, 6.95 [22:41:12] Crepusculum: can you do a subtask of the upgrade one? [22:41:55] RhinosF1: in a minute. My WiFi just went down, so I'm mobile for now. [22:42:17] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [22:42:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.66, 6.80, 6.76 [22:42:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.32, 6.44, 6.51 [22:42:36] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 6.30, 6.30 [22:44:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.15, 6.69, 6.72 [22:46:22] Sleep for me [22:47:53] OK, night. [22:52:34] ParserFirstCallInit is the cause [22:52:38] the hook within that skin [22:55:23] Crepusculum: RhinosF1 ^ [22:57:53] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [23:05:04] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:06:18] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:11:19] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [23:19:15] * dmehus is surprised Crepusculum doesn't have a USB mobile hub for his mobile phone so he can create a mobile LTE hotspot for his laptop in such events [23:19:59] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [23:21:47] Crepusculum: https://github.com/thaider/Tweeki/pull/200 [23:21:48] [ Fix checking for skin within onParserFirstCallInit by paladox · Pull Request #200 · thaider/Tweeki · GitHub ] - github.com [23:34:20] paladox: oh. thanks! [23:34:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.31, 7.00, 6.07 [23:35:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.30, 6.90, 6.33 [23:35:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JspI6 [23:35:54] [02miraheze/mw-config] 07paladox 032248b20 - Disable cp10/cp11 in wgCdnServers [23:35:55] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [23:35:57] [02mw-config] 07paladox opened pull request 03#3919: Disable cp10/cp11 in wgCdnServers - 13https://git.io/JspIi [23:36:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.45, 6.10, 5.86 [23:36:54] miraheze/mw-config - paladox the build passed. [23:37:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.21, 6.24, 6.16 [23:38:38] [02mw-config] 07Universal-Omega closed pull request 03#3916: Remove two deprecated password policy checks - 13https://git.io/JsbPL [23:38:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JspI7 [23:38:41] [02miraheze/mw-config] 07Universal-Omega 03e0b6f28 - Remove two deprecated password policy checks (#3916) [23:38:43] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [23:38:44] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:39:21] [02mw-config] 07Universal-Omega opened pull request 03#3920: Merge branch 'master' into REL1_36 - 13https://git.io/JspIA [23:39:40] miraheze/mw-config - Universal-Omega the build passed. [23:39:53] [02mw-config] 07Universal-Omega closed pull request 03#3920: Merge branch 'master' into REL1_36 - 13https://git.io/JspIA [23:39:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JspLe [23:39:56] [02miraheze/mw-config] 07Universal-Omega 03e0914f3 - Merge branch 'master' into REL1_36 (#3920) [23:39:58] [02mw-config] 07Universal-Omega synchronize pull request 03#3824: Update config for 1.36 - 13https://git.io/JYjbk [23:40:24] miraheze/mw-config - Universal-Omega the build passed. [23:40:55] miraheze/mw-config - Universal-Omega the build passed.