Fork me on GitHub

Wikimedia IRC logs browser - #miraheze-sre

Filter:
Start date
End date

Displaying 2008 items:

2021-05-24 00:02:16 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 00:03:27 <icinga-miraheze> PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown)
2021-05-24 00:05:26 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.16, 9.08, 7.27
2021-05-24 00:05:40 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.78, 7.65, 6.61
2021-05-24 00:07:40 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.86, 7.27, 6.60
2021-05-24 00:09:17 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.98, 7.54, 7.06
2021-05-24 00:09:40 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.61, 6.30, 6.32
2021-05-24 00:10:48 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1925': Bump extensions/PageForms from `ba1db1a` to `81fa60c` - 'https://git.io/JsF9y'
2021-05-24 00:10:50 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/PageForms-81fa60c' [+0/-0/±1] 'https://git.io/JsF9S'
2021-05-24 00:10:51 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '3dbee8' - Bump extensions/PageForms from `ba1db1a` to `81fa60c`
2021-05-24 00:10:53 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1925': Bump extensions/PageForms from `ba1db1a` to `81fa60c` - 'https://git.io/JsF9y'
2021-05-24 00:10:54 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1925': Bump extensions/PageForms from `ba1db1a` to `81fa60c` - 'https://git.io/JsF9y'
2021-05-24 00:10:56 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/PageForms-81fa60c' - 'https://git.io/vbL5b'
2021-05-24 00:10:57 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/skins/Citizen-321f9a7' [+0/-0/±1] 'https://git.io/JsF99'
2021-05-24 00:10:59 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '0c44ef' - Bump skins/Citizen from `b698a20` to `321f9a7`
2021-05-24 00:11:00 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1926': Bump skins/Citizen from `b698a20` to `321f9a7` - 'https://git.io/JsF9H'
2021-05-24 00:11:02 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1874': Bump extensions/PageForms from `ba1db1a` to `7dcf4d4` - 'https://git.io/JsF9Q'
2021-05-24 00:11:03 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1926': Bump skins/Citizen from `b698a20` to `321f9a7` - 'https://git.io/JsF9H'
2021-05-24 00:11:05 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1926': Bump skins/Citizen from `b698a20` to `321f9a7` - 'https://git.io/JsF9H'
2021-05-24 00:11:06 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/skins/Citizen-321f9a7' - 'https://git.io/vbL5b'
2021-05-24 00:11:08 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1874': Bump extensions/PageForms from `ba1db1a` to `7dcf4d4` - 'https://git.io/Js4RK'
2021-05-24 00:11:09 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/PageForms-7dcf4d4' - 'https://git.io/vbL5b'
2021-05-24 00:11:11 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/PageForms-7dcf4d4'
2021-05-24 00:11:12 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/AJAXPoll-b36f7c1' [+0/-0/±1] 'https://git.io/JsF97'
2021-05-24 00:11:14 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '083a6d' - Bump extensions/AJAXPoll from `3830cb4` to `b36f7c1`
2021-05-24 00:11:15 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1927': Bump extensions/AJAXPoll from `3830cb4` to `b36f7c1` - 'https://git.io/JsF95'
2021-05-24 00:11:17 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1906': Bump skins/Citizen from `2653336` to `11ee87c` - 'https://git.io/JsF9d'
2021-05-24 00:11:18 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/AJAXPoll-b36f7c1' - 'https://git.io/vbL5b'
2021-05-24 00:11:20 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1927': Bump extensions/AJAXPoll from `3830cb4` to `b36f7c1` - 'https://git.io/JsF95'
2021-05-24 00:11:21 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1927': Bump extensions/AJAXPoll from `3830cb4` to `b36f7c1` - 'https://git.io/JsF95'
2021-05-24 00:11:23 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1906': Bump skins/Citizen from `2653336` to `11ee87c` - 'https://git.io/Js40Q'
2021-05-24 00:11:24 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/skins/Citizen-11ee87c' - 'https://git.io/vbL5b'
2021-05-24 00:11:26 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/skins/Citizen-11ee87c'
2021-05-24 00:11:27 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1865': Bump extensions/AJAXPoll from `3830cb4` to `137a43f` - 'https://git.io/JsF9F'
2021-05-24 00:11:29 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1865': Bump extensions/AJAXPoll from `3830cb4` to `137a43f` - 'https://git.io/Js4RG'
2021-05-24 00:11:30 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/MediaWikiChat-d80506a' [+0/-0/±1] 'https://git.io/JsF9b'
2021-05-24 00:11:32 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'ce89954' - Bump extensions/MediaWikiChat from `ca65d0d` to `d80506a`
2021-05-24 00:11:33 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/AJAXPoll-137a43f' - 'https://git.io/vbL5b'
2021-05-24 00:11:35 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/AJAXPoll-137a43f'
2021-05-24 00:11:36 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/MediaWikiChat-d80506a' - 'https://git.io/vbL5b'
2021-05-24 00:11:38 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1928': Bump extensions/MediaWikiChat from `ca65d0d` to `d80506a` - 'https://git.io/JsF9N'
2021-05-24 00:11:39 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1928': Bump extensions/MediaWikiChat from `ca65d0d` to `d80506a` - 'https://git.io/JsF9N'
2021-05-24 00:11:41 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1928': Bump extensions/MediaWikiChat from `ca65d0d` to `d80506a` - 'https://git.io/JsF9N'
2021-05-24 00:11:42 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/PollNY-f82cb41' [+0/-0/±1] 'https://git.io/JsF9A'
2021-05-24 00:11:44 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '5cba70' - Bump extensions/PollNY from `4d7b0e5` to `f82cb41`
2021-05-24 00:11:45 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1929': Bump extensions/PollNY from `4d7b0e5` to `f82cb41` - 'https://git.io/JsF9x'
2021-05-24 00:11:47 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/PollNY-f82cb41' - 'https://git.io/vbL5b'
2021-05-24 00:11:48 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1929': Bump extensions/PollNY from `4d7b0e5` to `f82cb41` - 'https://git.io/JsF9x'
2021-05-24 00:11:50 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1929': Bump extensions/PollNY from `4d7b0e5` to `f82cb41` - 'https://git.io/JsF9x'
2021-05-24 00:11:51 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1930': Bump extensions/GlobalUsage from `094acd3` to `19974c6` - 'https://git.io/JsF9h'
2021-05-24 00:11:53 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1930': Bump extensions/GlobalUsage from `094acd3` to `19974c6` - 'https://git.io/JsF9h'
2021-05-24 00:11:54 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1930': Bump extensions/GlobalUsage from `094acd3` to `19974c6` - 'https://git.io/JsF9h'
2021-05-24 00:11:56 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1852': Bump extensions/GlobalUsage from `094acd3` to `4c52edd` - 'https://git.io/JsF9j'
2021-05-24 00:11:57 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1852': Bump extensions/GlobalUsage from `094acd3` to `4c52edd` - 'https://git.io/Js4B9'
2021-05-24 00:11:59 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/GlobalUsage-4c52edd' - 'https://git.io/vbL5b'
2021-05-24 00:12:00 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/GlobalUsage-4c52edd'
2021-05-24 00:12:02 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-9c81e9b' [+0/-0/±1] 'https://git.io/JsFHe'
2021-05-24 00:12:03 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '743613' - Bump extensions/SnapProjectEmbed from `cf18475` to `9c81e9b`
2021-05-24 00:12:05 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1931': Bump extensions/SnapProjectEmbed from `cf18475` to `9c81e9b` - 'https://git.io/JsFHv'
2021-05-24 00:12:06 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-9c81e9b' - 'https://git.io/vbL5b'
2021-05-24 00:12:08 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1931': Bump extensions/SnapProjectEmbed from `cf18475` to `9c81e9b` - 'https://git.io/JsFHv'
2021-05-24 00:12:09 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1931': Bump extensions/SnapProjectEmbed from `cf18475` to `9c81e9b` - 'https://git.io/JsFHv'
2021-05-24 00:12:11 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/IncidentReporting-d03aa59' [+0/-0/±1] 'https://git.io/JsFHf'
2021-05-24 00:12:12 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'a2f7958' - Bump extensions/IncidentReporting from `85c5618` to `d03aa59`
2021-05-24 00:12:14 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1932': Bump extensions/IncidentReporting from `85c5618` to `d03aa59` - 'https://git.io/JsFHJ'
2021-05-24 00:12:15 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/IncidentReporting-d03aa59' - 'https://git.io/vbL5b'
2021-05-24 00:12:17 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1932': Bump extensions/IncidentReporting from `85c5618` to `d03aa59` - 'https://git.io/JsFHJ'
2021-05-24 00:12:18 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1932': Bump extensions/IncidentReporting from `85c5618` to `d03aa59` - 'https://git.io/JsFHJ'
2021-05-24 00:12:20 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/YouTube-80b7c65' [+0/-0/±1] 'https://git.io/JsFHU'
2021-05-24 00:12:21 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '624969' - Bump extensions/YouTube from `0941937` to `80b7c65`
2021-05-24 00:12:23 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/YouTube-80b7c65' - 'https://git.io/vbL5b'
2021-05-24 00:12:24 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1933': Bump extensions/YouTube from `0941937` to `80b7c65` - 'https://git.io/JsFHT'
2021-05-24 00:12:26 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1933': Bump extensions/YouTube from `0941937` to `80b7c65` - 'https://git.io/JsFHT'
2021-05-24 00:12:27 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1933': Bump extensions/YouTube from `0941937` to `80b7c65` - 'https://git.io/JsFHT'
2021-05-24 00:12:29 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/skins/Truglass-d48ad91' [+0/-0/±1] 'https://git.io/JsFHk'
2021-05-24 00:12:30 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '26d802' - Bump skins/Truglass from `cc8fe11` to `d48ad91`
2021-05-24 00:12:32 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1934': Bump skins/Truglass from `cc8fe11` to `d48ad91` - 'https://git.io/JsFHI'
2021-05-24 00:12:33 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/skins/Truglass-d48ad91' - 'https://git.io/vbL5b'
2021-05-24 00:12:35 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1934': Bump skins/Truglass from `cc8fe11` to `d48ad91` - 'https://git.io/JsFHI'
2021-05-24 00:12:36 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1934': Bump skins/Truglass from `cc8fe11` to `d48ad91` - 'https://git.io/JsFHI'
2021-05-24 00:12:38 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/Report-0a98649' [+0/-0/±1] 'https://git.io/JsFHt'
2021-05-24 00:12:39 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'c2d82f' - Bump extensions/Report from `f828dc6` to `0a98649`
2021-05-24 00:12:41 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1935': Bump extensions/Report from `f828dc6` to `0a98649` - 'https://git.io/JsFHq'
2021-05-24 00:12:42 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/Report-0a98649' - 'https://git.io/vbL5b'
2021-05-24 00:12:44 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1935': Bump extensions/Report from `f828dc6` to `0a98649` - 'https://git.io/JsFHq'
2021-05-24 00:12:45 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1935': Bump extensions/Report from `f828dc6` to `0a98649` - 'https://git.io/JsFHq'
2021-05-24 00:12:47 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/Video-7675eae' [+0/-0/±1] 'https://git.io/JsFHm'
2021-05-24 00:12:48 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'fa369d9' - Bump extensions/Video from `28a3af3` to `7675eae`
2021-05-24 00:12:50 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/Video-7675eae' - 'https://git.io/vbL5b'
2021-05-24 00:12:51 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1936': Bump extensions/Video from `28a3af3` to `7675eae` - 'https://git.io/JsFHY'
2021-05-24 00:12:53 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1936': Bump extensions/Video from `28a3af3` to `7675eae` - 'https://git.io/JsFHY'
2021-05-24 00:12:54 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1936': Bump extensions/Video from `28a3af3` to `7675eae` - 'https://git.io/JsFHY'
2021-05-24 00:12:56 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1881': Bump extensions/Video from `28a3af3` to `b41fdf1` - 'https://git.io/JsFHO'
2021-05-24 00:12:57 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1881': Bump extensions/Video from `28a3af3` to `b41fdf1` - 'https://git.io/Js4Rb'
2021-05-24 00:12:59 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/Video-b41fdf1'
2021-05-24 00:13:00 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/Video-b41fdf1' - 'https://git.io/vbL5b'
2021-05-24 00:13:02 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/Cargo-0ccd967' [+0/-0/±1] 'https://git.io/JsFH3'
2021-05-24 00:13:03 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'f24dd16' - Bump extensions/Cargo from `8fc3204` to `0ccd967`
2021-05-24 00:13:05 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1937': Bump extensions/Cargo from `8fc3204` to `0ccd967` - 'https://git.io/JsFHs'
2021-05-24 00:13:06 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/Cargo-0ccd967' - 'https://git.io/vbL5b'
2021-05-24 00:13:08 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1937': Bump extensions/Cargo from `8fc3204` to `0ccd967` - 'https://git.io/JsFHs'
2021-05-24 00:13:09 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1937': Bump extensions/Cargo from `8fc3204` to `0ccd967` - 'https://git.io/JsFHs'
2021-05-24 00:13:11 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/SpriteSheet-cdb73e7' [+0/-0/±1] 'https://git.io/JsFHG'
2021-05-24 00:13:12 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '07520d' - Bump extensions/SpriteSheet from `594d6da` to `cdb73e7`
2021-05-24 00:13:14 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/SpriteSheet-cdb73e7' - 'https://git.io/vbL5b'
2021-05-24 00:13:15 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1938': Bump extensions/SpriteSheet from `594d6da` to `cdb73e7` - 'https://git.io/JsFHZ'
2021-05-24 00:13:17 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1938': Bump extensions/SpriteSheet from `594d6da` to `cdb73e7` - 'https://git.io/JsFHZ'
2021-05-24 00:13:18 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1938': Bump extensions/SpriteSheet from `594d6da` to `cdb73e7` - 'https://git.io/JsFHZ'
2021-05-24 00:13:20 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1899': Bump extensions/SpriteSheet from `594d6da` to `c42e3a1` - 'https://git.io/JsFHn'
2021-05-24 00:13:21 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1899': Bump extensions/SpriteSheet from `594d6da` to `c42e3a1` - 'https://git.io/Js40V'
2021-05-24 00:13:23 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/SpriteSheet-c42e3a1'
2021-05-24 00:13:24 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/SpriteSheet-c42e3a1' - 'https://git.io/vbL5b'
2021-05-24 00:13:26 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/CheckUser-025d552' [+0/-0/±1] 'https://git.io/JsFHc'
2021-05-24 00:13:27 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '497f01' - Bump extensions/CheckUser from `528b959` to `025d552`
2021-05-24 00:13:29 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/CheckUser-025d552' - 'https://git.io/vbL5b'
2021-05-24 00:13:30 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1939': Bump extensions/CheckUser from `528b959` to `025d552` - 'https://git.io/JsFHC'
2021-05-24 00:13:32 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1939': Bump extensions/CheckUser from `528b959` to `025d552` - 'https://git.io/JsFHC'
2021-05-24 00:13:33 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1939': Bump extensions/CheckUser from `528b959` to `025d552` - 'https://git.io/JsFHC'
2021-05-24 00:13:35 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1859': Bump extensions/CheckUser from `528b959` to `12bc076` - 'https://git.io/JsFHW'
2021-05-24 00:13:36 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1859': Bump extensions/CheckUser from `528b959` to `12bc076` - 'https://git.io/Js4Rf'
2021-05-24 00:13:38 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/CheckUser-12bc076' - 'https://git.io/vbL5b'
2021-05-24 00:13:39 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/CheckUser-12bc076'
2021-05-24 00:13:41 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/QuizGame-fbc2ecb' [+0/-0/±1] 'https://git.io/JsFHl'
2021-05-24 00:13:42 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'f1f4b0' - Bump extensions/QuizGame from `1a9fbc3` to `fbc2ecb`
2021-05-24 00:13:44 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/QuizGame-fbc2ecb' - 'https://git.io/vbL5b'
2021-05-24 00:13:45 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1940': Bump extensions/QuizGame from `1a9fbc3` to `fbc2ecb` - 'https://git.io/JsFH8'
2021-05-24 00:13:47 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1940': Bump extensions/QuizGame from `1a9fbc3` to `fbc2ecb` - 'https://git.io/JsFH8'
2021-05-24 00:13:48 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1940': Bump extensions/QuizGame from `1a9fbc3` to `fbc2ecb` - 'https://git.io/JsFH8'
2021-05-24 00:13:50 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1913': Bump extensions/QuizGame from `1a9fbc3` to `5ca61d5` - 'https://git.io/JsFH4'
2021-05-24 00:13:51 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1913': Bump extensions/QuizGame from `1a9fbc3` to `5ca61d5` - 'https://git.io/Js4EI'
2021-05-24 00:13:53 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/skins/Cosmos-60d2e29' [+0/-0/±1] 'https://git.io/JsFHB'
2021-05-24 00:13:54 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'efcef45' - Bump skins/Cosmos from `e8a77e6` to `60d2e29`
2021-05-24 00:13:56 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1941': Bump skins/Cosmos from `e8a77e6` to `60d2e29` - 'https://git.io/JsFHR'
2021-05-24 00:13:57 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/QuizGame-5ca61d5' - 'https://git.io/vbL5b'
2021-05-24 00:13:59 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/QuizGame-5ca61d5'
2021-05-24 00:14:00 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/skins/Cosmos-60d2e29' - 'https://git.io/vbL5b'
2021-05-24 00:14:02 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1941': Bump skins/Cosmos from `e8a77e6` to `60d2e29` - 'https://git.io/JsFHR'
2021-05-24 00:14:03 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1941': Bump skins/Cosmos from `e8a77e6` to `60d2e29` - 'https://git.io/JsFHR'
2021-05-24 00:14:05 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/VoteNY-8e81fcb' [+0/-0/±1] 'https://git.io/JsFH0'
2021-05-24 00:14:06 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'bdfa1ec' - Bump extensions/VoteNY from `c7a829a` to `8e81fcb`
2021-05-24 00:14:08 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1942': Bump extensions/VoteNY from `c7a829a` to `8e81fcb` - 'https://git.io/JsFHE'
2021-05-24 00:14:09 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/VoteNY-8e81fcb' - 'https://git.io/vbL5b'
2021-05-24 00:14:11 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/GlobalUsage-19974c6' [+0/-0/±1] 'https://git.io/JsFHu'
2021-05-24 00:14:12 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'cf501f' - Bump extensions/GlobalUsage from `094acd3` to `19974c6`
2021-05-24 00:14:14 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/GlobalUsage-19974c6' - 'https://git.io/vbL5b'
2021-05-24 00:14:15 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1942': Bump extensions/VoteNY from `c7a829a` to `8e81fcb` - 'https://git.io/JsFHE'
2021-05-24 00:14:17 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1942': Bump extensions/VoteNY from `c7a829a` to `8e81fcb` - 'https://git.io/JsFHE'
2021-05-24 00:14:18 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/PDFEmbed-22ea240' [+0/-0/±1] 'https://git.io/JsFHz'
2021-05-24 00:14:20 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '83eff0' - Bump extensions/PDFEmbed from `5c1efbf` to `22ea240`
2021-05-24 00:14:21 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/PDFEmbed-22ea240' - 'https://git.io/vbL5b'
2021-05-24 00:14:23 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1943': Bump extensions/PDFEmbed from `5c1efbf` to `22ea240` - 'https://git.io/JsFHg'
2021-05-24 00:14:24 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1943': Bump extensions/PDFEmbed from `5c1efbf` to `22ea240` - 'https://git.io/JsFHg'
2021-05-24 00:14:26 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1943': Bump extensions/PDFEmbed from `5c1efbf` to `22ea240` - 'https://git.io/JsFHg'
2021-05-24 00:14:27 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/WikiDiscover-d88811a' [+0/-0/±1] 'https://git.io/JsFHa'
2021-05-24 00:14:29 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'bcb78d' - Bump extensions/WikiDiscover from `2be49d8` to `d88811a`
2021-05-24 00:14:30 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1944': Bump extensions/WikiDiscover from `2be49d8` to `d88811a` - 'https://git.io/JsFHV'
2021-05-24 00:14:32 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/WikiDiscover-d88811a' - 'https://git.io/vbL5b'
2021-05-24 00:14:33 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1944': Bump extensions/WikiDiscover from `2be49d8` to `d88811a` - 'https://git.io/JsFHV'
2021-05-24 00:14:35 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1944': Bump extensions/WikiDiscover from `2be49d8` to `d88811a` - 'https://git.io/JsFHV'
2021-05-24 00:14:36 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/RottenLinks-41e13dd' [+0/-0/±1] 'https://git.io/JsFHr'
2021-05-24 00:14:38 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'ace49d6' - Bump extensions/RottenLinks from `c5a8212` to `41e13dd`
2021-05-24 00:14:39 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/RottenLinks-41e13dd' - 'https://git.io/vbL5b'
2021-05-24 00:14:41 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1945': Bump extensions/RottenLinks from `c5a8212` to `41e13dd` - 'https://git.io/JsFHo'
2021-05-24 00:14:42 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1945': Bump extensions/RottenLinks from `c5a8212` to `41e13dd` - 'https://git.io/JsFHo'
2021-05-24 00:14:44 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1945': Bump extensions/RottenLinks from `c5a8212` to `41e13dd` - 'https://git.io/JsFHo'
2021-05-24 00:14:45 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/MatomoAnalytics-b59c519' [+0/-0/±1] 'https://git.io/JsFHK'
2021-05-24 00:14:47 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'e27c1ba' - Bump extensions/MatomoAnalytics from `1d6d1f2` to `b59c519`
2021-05-24 00:14:48 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1946': Bump extensions/MatomoAnalytics from `1d6d1f2` to `b59c519` - 'https://git.io/JsFH6'
2021-05-24 00:14:50 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/MatomoAnalytics-b59c519' - 'https://git.io/vbL5b'
2021-05-24 00:14:51 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1946': Bump extensions/MatomoAnalytics from `1d6d1f2` to `b59c519` - 'https://git.io/JsFH6'
2021-05-24 00:14:53 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1946': Bump extensions/MatomoAnalytics from `1d6d1f2` to `b59c519` - 'https://git.io/JsFH6'
2021-05-24 00:14:54 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/GlobalNewFiles-e5e58cc' [+0/-0/±1] 'https://git.io/JsFHi'
2021-05-24 00:14:56 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '76d6e7' - Bump extensions/GlobalNewFiles from `d831a1a` to `e5e58cc`
2021-05-24 00:14:57 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1947': Bump extensions/GlobalNewFiles from `d831a1a` to `e5e58cc` - 'https://git.io/JsFHP'
2021-05-24 00:14:59 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/GlobalNewFiles-e5e58cc' - 'https://git.io/vbL5b'
2021-05-24 00:15:00 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1947': Bump extensions/GlobalNewFiles from `d831a1a` to `e5e58cc` - 'https://git.io/JsFHP'
2021-05-24 00:15:02 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1947': Bump extensions/GlobalNewFiles from `d831a1a` to `e5e58cc` - 'https://git.io/JsFHP'
2021-05-24 00:15:02 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.23, 6.13, 6.67
2021-05-24 00:15:03 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/CodeMirror-84846ec' [+0/-0/±1] 'https://git.io/JsFHX'
2021-05-24 00:15:05 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '10d838' - Bump extensions/CodeMirror from `a4ebaae` to `84846ec`
2021-05-24 00:15:06 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/CodeMirror-84846ec' - 'https://git.io/vbL5b'
2021-05-24 00:15:08 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1948': Bump extensions/CodeMirror from `a4ebaae` to `84846ec` - 'https://git.io/JsFH1'
2021-05-24 00:15:09 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1948': Bump extensions/CodeMirror from `a4ebaae` to `84846ec` - 'https://git.io/JsFH1'
2021-05-24 00:15:11 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1948': Bump extensions/CodeMirror from `a4ebaae` to `84846ec` - 'https://git.io/JsFH1'
2021-05-24 00:15:12 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/ReplaceText-7c4fbd1' [+0/-0/±1] 'https://git.io/JsFHM'
2021-05-24 00:15:14 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'f8c6e8' - Bump extensions/ReplaceText from `feb8a9c` to `7c4fbd1`
2021-05-24 00:15:15 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1949': Bump extensions/ReplaceText from `feb8a9c` to `7c4fbd1` - 'https://git.io/JsFHD'
2021-05-24 00:15:17 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/ReplaceText-7c4fbd1' - 'https://git.io/vbL5b'
2021-05-24 00:15:18 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1949': Bump extensions/ReplaceText from `feb8a9c` to `7c4fbd1` - 'https://git.io/JsFHD'
2021-05-24 00:15:20 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1949': Bump extensions/ReplaceText from `feb8a9c` to `7c4fbd1` - 'https://git.io/JsFHD'
2021-05-24 00:15:21 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1864': Bump extensions/ReplaceText from `feb8a9c` to `f7f0ddb` - 'https://git.io/JsFHy'
2021-05-24 00:15:23 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1864': Bump extensions/ReplaceText from `feb8a9c` to `f7f0ddb` - 'https://git.io/Js4R3'
2021-05-24 00:15:24 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/ReplaceText-f7f0ddb'
2021-05-24 00:15:26 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/RemovePII-8eb89df' [+0/-0/±1] 'https://git.io/JsFHS'
2021-05-24 00:15:27 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'cc4e291' - Bump extensions/RemovePII from `738036f` to `8eb89df`
2021-05-24 00:15:29 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/ReplaceText-f7f0ddb' - 'https://git.io/vbL5b'
2021-05-24 00:15:30 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1950': Bump extensions/RemovePII from `738036f` to `8eb89df` - 'https://git.io/JsFH9'
2021-05-24 00:15:32 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/RemovePII-8eb89df' - 'https://git.io/vbL5b'
2021-05-24 00:15:33 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1950': Bump extensions/RemovePII from `738036f` to `8eb89df` - 'https://git.io/JsFH9'
2021-05-24 00:15:35 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1950': Bump extensions/RemovePII from `738036f` to `8eb89df` - 'https://git.io/JsFH9'
2021-05-24 00:15:36 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/AutoCreateCategoryPages-905e85a' [+0/-0/±1] 'https://git.io/JsFHH'
2021-05-24 00:15:38 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'c25e786' - Bump extensions/AutoCreateCategoryPages from `463effd` to `905e85a`
2021-05-24 00:15:39 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1951': Bump extensions/AutoCreateCategoryPages from `463effd` to `905e85a` - 'https://git.io/JsFH7'
2021-05-24 00:15:41 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/AutoCreateCategoryPages-905e85a' - 'https://git.io/vbL5b'
2021-05-24 00:15:42 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1951': Bump extensions/AutoCreateCategoryPages from `463effd` to `905e85a` - 'https://git.io/JsFH7'
2021-05-24 00:15:44 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1951': Bump extensions/AutoCreateCategoryPages from `463effd` to `905e85a` - 'https://git.io/JsFH7'
2021-05-24 00:15:45 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/cldr-3eadcc4' [+0/-0/±1] 'https://git.io/JsFHd'
2021-05-24 00:15:47 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '3bb4b6' - Bump extensions/cldr from `4b52abe` to `3eadcc4`
2021-05-24 00:15:48 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/cldr-3eadcc4' - 'https://git.io/vbL5b'
2021-05-24 00:15:49 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 00:15:50 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1952': Bump extensions/cldr from `4b52abe` to `3eadcc4` - 'https://git.io/JsFHF'
2021-05-24 00:15:51 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1952': Bump extensions/cldr from `4b52abe` to `3eadcc4` - 'https://git.io/JsFHF'
2021-05-24 00:15:53 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1952': Bump extensions/cldr from `4b52abe` to `3eadcc4` - 'https://git.io/JsFHF'
2021-05-24 00:15:54 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/ApprovedRevs-3e3edf0' [+0/-0/±1] 'https://git.io/JsFHb'
2021-05-24 00:15:56 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '73a565' - Bump extensions/ApprovedRevs from `69f6dc7` to `3e3edf0`
2021-05-24 00:15:57 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1953': Bump extensions/ApprovedRevs from `69f6dc7` to `3e3edf0` - 'https://git.io/JsFHN'
2021-05-24 00:15:59 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/ApprovedRevs-3e3edf0' - 'https://git.io/vbL5b'
2021-05-24 00:16:00 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1953': Bump extensions/ApprovedRevs from `69f6dc7` to `3e3edf0` - 'https://git.io/JsFHN'
2021-05-24 00:16:02 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1953': Bump extensions/ApprovedRevs from `69f6dc7` to `3e3edf0` - 'https://git.io/JsFHN'
2021-05-24 00:16:03 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/CommentStreams-4686df9' [+0/-0/±1] 'https://git.io/JsFHA'
2021-05-24 00:16:05 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'f5b68df' - Bump extensions/CommentStreams from `ed3fc73` to `4686df9`
2021-05-24 00:16:06 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/CommentStreams-4686df9' - 'https://git.io/vbL5b'
2021-05-24 00:16:08 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1954': Bump extensions/CommentStreams from `ed3fc73` to `4686df9` - 'https://git.io/JsFHx'
2021-05-24 00:16:09 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1905': Bump extensions/ApprovedRevs from `69f6dc7` to `c01d76a` - 'https://git.io/JsFHp'
2021-05-24 00:16:11 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1954': Bump extensions/CommentStreams from `ed3fc73` to `4686df9` - 'https://git.io/JsFHx'
2021-05-24 00:16:12 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1954': Bump extensions/CommentStreams from `ed3fc73` to `4686df9` - 'https://git.io/JsFHx'
2021-05-24 00:16:14 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1905': Bump extensions/ApprovedRevs from `69f6dc7` to `c01d76a` - 'https://git.io/Js409'
2021-05-24 00:16:15 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/ApprovedRevs-c01d76a' - 'https://git.io/vbL5b'
2021-05-24 00:16:17 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/ApprovedRevs-c01d76a'
2021-05-24 00:16:18 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/GettingStarted-4a65130' [+0/-0/±1] 'https://git.io/JsFHh'
2021-05-24 00:16:20 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'e742a25' - Bump extensions/GettingStarted from `5ec1c75` to `4a65130`
2021-05-24 00:16:21 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' commented on pull request '#1872': Bump extensions/CommentStreams from `ed3fc73` to `52be0c2` - 'https://git.io/JsFHj'
2021-05-24 00:16:23 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1955': Bump extensions/GettingStarted from `5ec1c75` to `4a65130` - 'https://git.io/JsFQe'
2021-05-24 00:16:24 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/GettingStarted-4a65130' - 'https://git.io/vbL5b'
2021-05-24 00:16:26 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' closed pull request '#1872': Bump extensions/CommentStreams from `ed3fc73` to `52be0c2` - 'https://git.io/Js4Ra'
2021-05-24 00:16:27 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1955': Bump extensions/GettingStarted from `5ec1c75` to `4a65130` - 'https://git.io/JsFQe'
2021-05-24 00:16:29 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1955': Bump extensions/GettingStarted from `5ec1c75` to `4a65130` - 'https://git.io/JsFQe'
2021-05-24 00:16:30 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/CommentStreams-52be0c2'
2021-05-24 00:16:32 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' deleted branch 'dependabot/submodules/REL1_35/extensions/CommentStreams-52be0c2' - 'https://git.io/vbL5b'
2021-05-24 00:16:33 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/BlogPage-33af4de' [+0/-0/±1] 'https://git.io/JsFQf'
2021-05-24 00:16:35 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'daac5b9' - Bump extensions/BlogPage from `1672d46` to `33af4de`
2021-05-24 00:16:36 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1956': Bump extensions/BlogPage from `1672d46` to `33af4de` - 'https://git.io/JsFQJ'
2021-05-24 00:16:38 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/BlogPage-33af4de' - 'https://git.io/vbL5b'
2021-05-24 00:16:39 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1956': Bump extensions/BlogPage from `1672d46` to `33af4de` - 'https://git.io/JsFQJ'
2021-05-24 00:16:41 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1956': Bump extensions/BlogPage from `1672d46` to `33af4de` - 'https://git.io/JsFQJ'
2021-05-24 00:16:42 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/SocialProfile-f70d79e' [+0/-0/±1] 'https://git.io/JsFQU'
2021-05-24 00:16:44 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'cf411e' - Bump extensions/SocialProfile from `b258833` to `f70d79e`
2021-05-24 00:16:45 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1957': Bump extensions/SocialProfile from `b258833` to `f70d79e` - 'https://git.io/JsFQT'
2021-05-24 00:16:47 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/SocialProfile-f70d79e' - 'https://git.io/vbL5b'
2021-05-24 00:16:48 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1957': Bump extensions/SocialProfile from `b258833` to `f70d79e` - 'https://git.io/JsFQT'
2021-05-24 00:16:50 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1957': Bump extensions/SocialProfile from `b258833` to `f70d79e` - 'https://git.io/JsFQT'
2021-05-24 00:16:51 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/ArticleToCategory2-ce3b839' [+0/-0/±1] 'https://git.io/JsFQk'
2021-05-24 00:16:53 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'f1cc2f8' - Bump extensions/ArticleToCategory2 from `9fa7338` to `ce3b839`
2021-05-24 00:16:54 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1958': Bump extensions/ArticleToCategory2 from `9fa7338` to `ce3b839` - 'https://git.io/JsFQI'
2021-05-24 00:16:56 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/ArticleToCategory2-ce3b839' - 'https://git.io/vbL5b'
2021-05-24 00:16:57 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1958': Bump extensions/ArticleToCategory2 from `9fa7338` to `ce3b839` - 'https://git.io/JsFQI'
2021-05-24 00:16:59 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1958': Bump extensions/ArticleToCategory2 from `9fa7338` to `ce3b839` - 'https://git.io/JsFQI'
2021-05-24 00:17:00 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-282a40e' [+0/-0/±1] 'https://git.io/JsFQq'
2021-05-24 00:17:02 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' '263dea' - Bump extensions/UniversalLanguageSelector from `b556ff5` to `282a40e`
2021-05-24 00:17:03 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1959': Bump extensions/UniversalLanguageSelector from `b556ff5` to `282a40e` - 'https://git.io/JsFQm'
2021-05-24 00:17:05 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-282a40e' - 'https://git.io/vbL5b'
2021-05-24 00:17:06 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1959': Bump extensions/UniversalLanguageSelector from `b556ff5` to `282a40e` - 'https://git.io/JsFQm'
2021-05-24 00:17:08 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1959': Bump extensions/UniversalLanguageSelector from `b556ff5` to `282a40e` - 'https://git.io/JsFQm'
2021-05-24 00:17:09 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' pushed '' commit to 'dependabot/submodules/REL1_35/extensions/Maps-eb95ae6' [+0/-0/±1] 'https://git.io/JsFQY'
2021-05-24 00:17:11 <Not-4c5b> ['miraheze/mediawiki'] 'dependabot[bot]' 'd2a0ed' - Bump extensions/Maps from `11d8882` to `eb95ae6`
2021-05-24 00:17:12 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' opened pull request '#1960': Bump extensions/Maps from `11d8882` to `eb95ae6` - 'https://git.io/JsFQO'
2021-05-24 00:17:14 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' created branch 'dependabot/submodules/REL1_35/extensions/Maps-eb95ae6' - 'https://git.io/vbL5b'
2021-05-24 00:17:15 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1960': Bump extensions/Maps from `11d8882` to `eb95ae6` - 'https://git.io/JsFQO'
2021-05-24 00:17:17 <Not-4c5b> ['mediawiki'] 'dependabot[bot]' labeled pull request '#1960': Bump extensions/Maps from `11d8882` to `eb95ae6` - 'https://git.io/JsFQO'
2021-05-24 00:24:35 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 00:31:25 <icinga-miraheze> RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 00:35:12 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.60, 6.59, 6.07
2021-05-24 00:37:10 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.53, 6.09, 5.94
2021-05-24 00:38:11 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 00:46:55 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 01:05:46 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.46, 7.03, 5.86
2021-05-24 01:07:46 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.40, 6.55, 5.84
2021-05-24 01:41:14 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 01:55:02 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.91, 6.23, 5.66
2021-05-24 01:57:01 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.18, 6.83, 5.92
2021-05-24 01:59:43 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 18.76, 10.95, 7.25
2021-05-24 02:00:02 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.01, 10.12, 7.37
2021-05-24 02:00:03 <icinga-miraheze> PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
2021-05-24 02:00:12 <icinga-miraheze> PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:00:18 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.02, 9.53, 6.75
2021-05-24 02:00:24 <icinga-miraheze> PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is CRITICAL: CRITICAL - NGINX Error Rate is 95%
2021-05-24 02:00:42 <icinga-miraheze> PROBLEM - cp13 HTTPS on cp13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
2021-05-24 02:00:47 <icinga-miraheze> PROBLEM - cp12 HTTPS on cp12 is CRITICAL: HTTP CRITICAL - No data received from host
2021-05-24 02:00:49 <icinga-miraheze> PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:00:54 <icinga-miraheze> PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.76, 1.60, 1.22
2021-05-24 02:01:30 <icinga-miraheze> PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
2021-05-24 02:01:30 <icinga-miraheze> PROBLEM - wiki.scrungecraft.gg - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:01:42 <icinga-miraheze> PROBLEM - planetnomads.speedbomb.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:01:43 <icinga-miraheze> PROBLEM - icinga.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:01:47 <icinga-miraheze> PROBLEM - rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:01:48 <icinga-miraheze> PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:01:48 <icinga-miraheze> PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:01:58 <icinga-miraheze> PROBLEM - wiki.globasa.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:00 <icinga-miraheze> PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:04 <icinga-miraheze> PROBLEM - wiki.candela.digital - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:10 <dmehus> !sre 502 Bad Gateway and 500 Internal Server Error on `metawiki`
2021-05-24 02:02:11 <icinga-miraheze> PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:11 <icinga-miraheze> PROBLEM - wiki.yumeka.icu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:12 <icinga-miraheze> PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:16 <icinga-miraheze> PROBLEM - egoish.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:29 <icinga-miraheze> PROBLEM - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is CRITICAL: CRITICAL - NGINX Error Rate is 96%
2021-05-24 02:02:31 <icinga-miraheze> PROBLEM - christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:31 <icinga-miraheze> PROBLEM - wiki.thedev.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:33 <icinga-miraheze> PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 84%
2021-05-24 02:02:34 <icinga-miraheze> PROBLEM - wiki.rpointer.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:34 <icinga-miraheze> PROBLEM - electowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:42 <icinga-miraheze> PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:45 <icinga-miraheze> PROBLEM - erikapedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:51 <icinga-miraheze> PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:02:55 <icinga-miraheze> PROBLEM - cp11 Stunnel Http for mw8 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:02:56 <icinga-miraheze> PROBLEM - kkutu.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:12 <icinga-miraheze> PROBLEM - wiki.jacksonheights.nyc - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:16 <icinga-miraheze> PROBLEM - cp13 Stunnel Http for mw8 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:03:17 <icinga-miraheze> PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:19 <icinga-miraheze> PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:19 <icinga-miraheze> PROBLEM - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:21 <icinga-miraheze> PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:24 <icinga-miraheze> PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 4.85, 2.59, 1.62
2021-05-24 02:03:24 <icinga-miraheze> PROBLEM - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:28 <icinga-miraheze> PROBLEM - programming.red - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:28 <icinga-miraheze> PROBLEM - www.zenbuddhism.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:29 <icinga-miraheze> PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:29 <icinga-miraheze> PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:34 <icinga-miraheze> PROBLEM - puardian.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:37 <icinga-miraheze> PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
2021-05-24 02:03:38 <icinga-miraheze> PROBLEM - wiki.thefactoryhka.com.pa - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:42 <icinga-miraheze> PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:46 <icinga-miraheze> PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:49 <icinga-miraheze> PROBLEM - es.nocyclo.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:51 <icinga-miraheze> PROBLEM - wiki.usagihime.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:51 <icinga-miraheze> PROBLEM - www.rangpurpedia.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:52 <icinga-miraheze> PROBLEM - cp10 Stunnel Http for mw8 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:03:52 <icinga-miraheze> PROBLEM - animatedfeet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:53 <icinga-miraheze> PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:57 <icinga-miraheze> PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:58 <icinga-miraheze> PROBLEM - wiki.jumpbound.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:58 <icinga-miraheze> PROBLEM - wiki.helioss.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:03:59 <icinga-miraheze> PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:02 <icinga-miraheze> PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:08 <icinga-miraheze> PROBLEM - wiki.xfbs.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:09 <icinga-miraheze> PROBLEM - cp3 Stunnel Http for mw8 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:04:37 <icinga-miraheze> PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:04:38 <icinga-miraheze> PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:41 <icinga-miraheze> PROBLEM - oecumene.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:41 <icinga-miraheze> PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:42 <icinga-miraheze> PROBLEM - www.wikilukas.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:44 <icinga-miraheze> PROBLEM - cp12 APT on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:04:44 <icinga-miraheze> PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:56 <icinga-miraheze> PROBLEM - www.mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:57 <icinga-miraheze> PROBLEM - wiki.ostans.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:04:59 <icinga-miraheze> PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:01 <icinga-miraheze> PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:02 <icinga-miraheze> PROBLEM - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:04 <icinga-miraheze> PROBLEM - opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:06 <icinga-miraheze> PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:06 <icinga-miraheze> PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:06 <icinga-miraheze> PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:07 <icinga-miraheze> PROBLEM - www.fossilfighterswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:08 <icinga-miraheze> PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:15 <icinga-miraheze> PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:15 <icinga-miraheze> PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:15 <icinga-miraheze> PROBLEM - bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:15 <icinga-miraheze> PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:15 <icinga-miraheze> PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:21 <icinga-miraheze> RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 9.192 second response time
2021-05-24 02:05:25 <icinga-miraheze> PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:26 <icinga-miraheze> PROBLEM - mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:05:30 <icinga-miraheze> RECOVERY - cp11 Stunnel Http for mw8 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 7.158 second response time
2021-05-24 02:05:47 <icinga-miraheze> RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 3.355 second response time
2021-05-24 02:06:02 <icinga-miraheze> PROBLEM - idleon.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:06:02 <icinga-miraheze> PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:06:05 <icinga-miraheze> RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20887 bytes in 3.716 second response time
2021-05-24 02:06:09 <icinga-miraheze> PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:06:18 <icinga-miraheze> PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:06:18 <icinga-miraheze> RECOVERY - cp10 Stunnel Http for mw8 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 3.898 second response time
2021-05-24 02:06:19 <icinga-miraheze> PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:06:26 <icinga-miraheze> PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:06:26 <icinga-miraheze> PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:06:29 <icinga-miraheze> PROBLEM - wikibeats.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:06:36 <icinga-miraheze> RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 8.684 second response time
2021-05-24 02:07:03 <icinga-miraheze> RECOVERY - cp12 APT on cp12 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
2021-05-24 02:07:11 <icinga-miraheze> PROBLEM - wiki.velaan.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:19 <icinga-miraheze> PROBLEM - petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:20 <icinga-miraheze> PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:22 <icinga-miraheze> PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.37, 1.69, 0.91
2021-05-24 02:07:23 <icinga-miraheze> PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:25 <icinga-miraheze> PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:29 <icinga-miraheze> PROBLEM - wiki.arnab.at - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:32 <icinga-miraheze> PROBLEM - wiki.ct777.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:32 <icinga-miraheze> PROBLEM - www.erikapedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:32 <icinga-miraheze> PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:36 <icinga-miraheze> PROBLEM - lama.madooa.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:07:44 <icinga-miraheze> PROBLEM - pa.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:08:28 <icinga-miraheze> PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:08:28 <icinga-miraheze> PROBLEM - miraheze.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:08:28 <icinga-miraheze> PROBLEM - www.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:08:33 <icinga-miraheze> PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:08:36 <icinga-miraheze> RECOVERY - wiki.scrungecraft.gg - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scrungecraft.gg' will expire on Tue 29 Jun 2021 18:43:12 GMT +0000.
2021-05-24 02:08:36 <icinga-miraheze> PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:08:43 <icinga-miraheze> PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:08:44 <dmehus> !sre
2021-05-24 02:08:56 <icinga-miraheze> PROBLEM - cp12 Stunnel Http for mw10 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:09:01 <icinga-miraheze> RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Sat 07 Aug 2021 03:45:48 GMT +0000.
2021-05-24 02:09:06 <icinga-miraheze> PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
2021-05-24 02:09:08 <icinga-miraheze> RECOVERY - wiki.globasa.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Sun 11 Jul 2021 16:41:59 GMT +0000.
2021-05-24 02:09:20 <icinga-miraheze> RECOVERY - egoish.org - LetsEncrypt on sslhost is OK: OK - Certificate 'egoish.org' will expire on Wed 07 Jul 2021 14:29:21 GMT +0000.
2021-05-24 02:09:21 <icinga-miraheze> PROBLEM - cp13 Stunnel Http for mw10 on cp13 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:09:33 <icinga-miraheze> PROBLEM - cp10 Stunnel Http for mw10 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:09:42 <icinga-miraheze> PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:09:43 <icinga-miraheze> RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.15, 1.54, 0.97
2021-05-24 02:09:47 <icinga-miraheze> PROBLEM - matttest.mtwiki.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:09:47 <icinga-miraheze> PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:09:48 <icinga-miraheze> PROBLEM - wiki.umbrel.tech - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:09:48 <icinga-miraheze> PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
2021-05-24 02:10:05 <icinga-miraheze> PROBLEM - cp11 Stunnel Http for mw10 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:10:10 <icinga-miraheze> RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Sat 07 Aug 2021 04:56:01 GMT +0000.
2021-05-24 02:10:18 <icinga-miraheze> RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Sat 07 Aug 2021 14:10:59 GMT +0000.
2021-05-24 02:10:18 <icinga-miraheze> RECOVERY - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.umbranet.mandelbrotgames.com' will expire on Tue 06 Jul 2021 03:47:45 GMT +0000.
2021-05-24 02:10:20 <icinga-miraheze> RECOVERY - wiki.jacksonheights.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jacksonheights.nyc' will expire on Sat 07 Aug 2021 01:23:05 GMT +0000.
2021-05-24 02:10:25 <icinga-miraheze> RECOVERY - dc-multiverse.dcwikis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dc-multiverse.dcwikis.com' will expire on Sat 07 Aug 2021 08:12:24 GMT +0000.
2021-05-24 02:10:34 <icinga-miraheze> PROBLEM - cp12 Stunnel Http for mw8 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
2021-05-24 02:10:34 <icinga-miraheze> RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sat 07 Aug 2021 05:30:44 GMT +0000.
2021-05-24 02:10:35 <icinga-miraheze> RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Sat 07 Aug 2021 01:11:15 GMT +0000.
2021-05-24 02:10:41 <icinga-miraheze> RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info'; will expire on Fri 20 Aug 2021 22:04:48 GMT +0000.
2021-05-24 02:10:45 <icinga-miraheze> RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sat 07 Aug 2021 05:56:13 GMT +0000.
2021-05-24 02:10:46 <icinga-miraheze> RECOVERY - wiki.astralprojections.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Sat 14 Aug 2021 18:06:42 GMT +0000.
2021-05-24 02:10:47 <icinga-miraheze> RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Sat 07 Aug 2021 09:16:22 GMT +0000.
2021-05-24 02:10:47 <Cocopuff2018> @dmehus all wikis are down
2021-05-24 02:10:49 <icinga-miraheze> RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sat 07 Aug 2021 11:02:49 GMT +0000.
2021-05-24 02:10:54 <icinga-miraheze> RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Sat 07 Aug 2021 10:42:34 GMT +0000.
2021-05-24 02:10:57 <icinga-miraheze> RECOVERY - puardian.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'puardian.xyz' will expire on Tue 27 Jul 2021 09:14:30 GMT +0000.
2021-05-24 02:10:58 <icinga-miraheze> RECOVERY - es.nocyclo.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'es.nocyclo.tk' will expire on Sat 07 Aug 2021 01:23:47 GMT +0000.
2021-05-24 02:10:58 <dmehus> Cocopuff2018, thanks :)
2021-05-24 02:10:59 <icinga-miraheze> RECOVERY - wiki.thefactoryhka.com.pa - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thefactoryhka.com.pa' will expire on Fri 09 Jul 2021 05:41:36 GMT +0000.
2021-05-24 02:11:00 <icinga-miraheze> RECOVERY - www.rangpurpedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'rangpurpedia.xyz' will expire on Fri 02 Jul 2021 06:08:10 GMT +0000.
2021-05-24 02:11:01 <icinga-miraheze> RECOVERY - animatedfeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000.
2021-05-24 02:11:01 <icinga-miraheze> RECOVERY - wiki.usagihime.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.usagihime.ml' will expire on Sat 07 Aug 2021 03:09:56 GMT +0000.
2021-05-24 02:11:01 <icinga-miraheze> RECOVERY - wiki.meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Sat 21 Aug 2021 14:09:26 GMT +0000.
2021-05-24 02:11:02 <icinga-miraheze> RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Sat 07 Aug 2021 03:34:59 GMT +0000.
2021-05-24 02:11:03 <dmehus> I'll update the task
2021-05-24 02:11:09 <icinga-miraheze> RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Sat 07 Aug 2021 05:50:29 GMT +0000.
2021-05-24 02:11:09 <icinga-miraheze> RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Sat 07 Aug 2021 01:39:47 GMT +0000.
2021-05-24 02:11:10 <icinga-miraheze> RECOVERY - wiki.jumpbound.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jumpbound.com' will expire on Wed 11 Aug 2021 04:43:11 GMT +0000.
2021-05-24 02:11:18 <icinga-miraheze> RECOVERY - wiki.helioss.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.helioss.co' will expire on Sat 07 Aug 2021 02:54:20 GMT +0000.
2021-05-24 02:11:18 <icinga-miraheze> RECOVERY - wiki.xfbs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.xfbs.net' will expire on Sat 07 Aug 2021 19:05:17 GMT +0000.
2021-05-24 02:11:29 <dmehus> ```<dmehus> Doug
2021-05-24 02:11:29 <dmehus> SENDOPS #miraheze-sre see #miraheze-sre - 502 Bad Gateways and 500 Internal Server Errors on all or most wikis going back 10 minutes or more. Phabricator task is https://phabricator.miraheze.org/T7352, and cause should be investigated
2021-05-24 02:11:29 <dmehus> 19:10 <MemoServ>
2021-05-24 02:11:29 <dmehus> The memo has been successfully sent to 6 operators on #miraheze-sre.```
2021-05-24 02:11:29 <dmehus> Cocopuff2018: ^
2021-05-24 02:11:30 <MirahezeBot> [ ⚓ T7352 502 Bad Gateway and 500 Internal Server Error on metawiki ] - phabricator.miraheze.org
2021-05-24 02:11:36 <icinga-miraheze> RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Tue 10 Aug 2021 19:08:35 GMT +0000.
2021-05-24 02:11:45 <icinga-miraheze> RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sun 08 Aug 2021 05:05:42 GMT +0000.
2021-05-24 02:11:45 <icinga-miraheze> RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate 'oecumene.org' will expire on Sat 07 Aug 2021 10:23:41 GMT +0000.
2021-05-24 02:11:49 <icinga-miraheze> RECOVERY - www.wikilukas.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wikilukas.tk' will expire on Sat 07 Aug 2021 19:12:21 GMT +0000.
2021-05-24 02:11:52 <icinga-miraheze> RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Fri 06 Aug 2021 23:54:15 GMT +0000.
2021-05-24 02:11:53 <icinga-miraheze> RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000.
2021-05-24 02:12:02 <icinga-miraheze> RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Fri 20 Aug 2021 22:59:52 GMT +0000.
2021-05-24 02:12:05 <icinga-miraheze> RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Thu 29 Jul 2021 12:53:14 GMT +0000.
2021-05-24 02:12:08 <icinga-miraheze> RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Sat 07 Aug 2021 02:55:10 GMT +0000.
2021-05-24 02:12:12 <icinga-miraheze> RECOVERY - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitaldesignhq.com' will expire on Mon 09 Aug 2021 20:08:17 GMT +0000.
2021-05-24 02:12:16 <icinga-miraheze> RECOVERY - wiki.ostans.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ostans.com' will expire on Sat 07 Aug 2021 08:01:07 GMT +0000.
2021-05-24 02:12:17 <icinga-miraheze> RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Sat 07 Aug 2021 10:09:02 GMT +0000.
2021-05-24 02:12:21 <icinga-miraheze> RECOVERY - www.fossilfighterswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com'; will expire on Sun 20 Jun 2021 19:36:06 GMT +0000.
2021-05-24 02:12:27 <icinga-miraheze> RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Sat 07 Aug 2021 08:19:02 GMT +0000.
2021-05-24 02:12:29 <icinga-miraheze> RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Sat 07 Aug 2021 10:38:37 GMT +0000.
2021-05-24 02:12:29 <icinga-miraheze> RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 21 Aug 2021 00:11:35 GMT +0000.
2021-05-24 02:12:29 <icinga-miraheze> RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Sat 07 Aug 2021 08:35:47 GMT +0000.
2021-05-24 02:12:35 <icinga-miraheze> RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Sat 07 Aug 2021 18:22:20 GMT +0000.
2021-05-24 02:12:35 <icinga-miraheze> RECOVERY - wiki.slimelab.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimelab.net' will expire on Tue 29 Jun 2021 13:29:07 GMT +0000.
2021-05-24 02:12:35 <icinga-miraheze> RECOVERY - bots.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Thu 24 Jun 2021 11:21:27 GMT +0000.
2021-05-24 02:12:35 <icinga-miraheze> RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Sat 07 Aug 2021 03:27:18 GMT +0000.
2021-05-24 02:12:35 <icinga-miraheze> RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com'; will expire on Fri 20 Aug 2021 22:13:22 GMT +0000.
2021-05-24 02:12:36 <icinga-miraheze> RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Sat 07 Aug 2021 08:58:20 GMT +0000.
2021-05-24 02:12:42 <icinga-miraheze> RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Sat 07 Aug 2021 10:09:02 GMT +0000.
2021-05-24 02:12:51 <icinga-miraheze> RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15284 bytes in 4.632 second response time
2021-05-24 02:12:55 <icinga-miraheze> PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.60, 1.74, 1.76
2021-05-24 02:12:56 <icinga-miraheze> RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2271 bytes in 5.781 second response time
2021-05-24 02:13:04 <icinga-miraheze> RECOVERY - idleon.info - LetsEncrypt on sslhost is OK: OK - Certificate 'idleon.info' will expire on Sat 07 Aug 2021 10:14:05 GMT +0000.
2021-05-24 02:13:05 <icinga-miraheze> RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Sat 07 Aug 2021 07:36:14 GMT +0000.
2021-05-24 02:13:16 <icinga-miraheze> RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Sat 07 Aug 2021 00:37:14 GMT +0000.
2021-05-24 02:13:19 <icinga-miraheze> RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Sat 07 Aug 2021 09:03:51 GMT +0000.
2021-05-24 02:13:20 <icinga-miraheze> RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Sat 07 Aug 2021 01:15:48 GMT +0000.
2021-05-24 02:13:21 <icinga-miraheze> RECOVERY - wikibeats.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikibeats.net' will expire on Sat 07 Aug 2021 18:43:31 GMT +0000.
2021-05-24 02:13:34 <icinga-miraheze> RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Sat 07 Aug 2021 08:35:47 GMT +0000.
2021-05-24 02:13:36 <icinga-miraheze> RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Sat 07 Aug 2021 06:47:52 GMT +0000.
2021-05-24 02:13:42 <icinga-miraheze> RECOVERY - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is OK: OK - NGINX Error Rate is 9%
2021-05-24 02:13:55 <icinga-miraheze> RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 24%
2021-05-24 02:14:01 <icinga-miraheze> RECOVERY - wiki.velaan.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.velaan.org' will expire on Sat 07 Aug 2021 08:06:51 GMT +0000.
2021-05-24 02:14:20 <icinga-miraheze> RECOVERY - wiki.arnab.at - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.arnab.at' will expire on Sat 07 Aug 2021 07:26:41 GMT +0000.
2021-05-24 02:14:21 <icinga-miraheze> RECOVERY - petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.petrawiki.org'; will expire on Fri 25 Jun 2021 05:54:36 GMT +0000.
2021-05-24 02:14:22 <icinga-miraheze> RECOVERY - www.erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com'; will expire on Fri 20 Aug 2021 22:28:16 GMT +0000.
2021-05-24 02:14:22 <icinga-miraheze> RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Sat 07 Aug 2021 00:58:24 GMT +0000.
2021-05-24 02:14:22 <icinga-miraheze> RECOVERY - wiki.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ct777.cf' will expire on Fri 02 Jul 2021 05:32:35 GMT +0000.
2021-05-24 02:14:26 <icinga-miraheze> PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is WARNING: WARNING - NGINX Error Rate is 49%
2021-05-24 02:14:27 <icinga-miraheze> RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Fri 06 Aug 2021 23:39:26 GMT +0000.
2021-05-24 02:14:27 <Cocopuff2018> @dmehus look like none available
2021-05-24 02:14:29 <icinga-miraheze> RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Sat 07 Aug 2021 10:48:18 GMT +0000.
2021-05-24 02:14:33 <icinga-miraheze> RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pa.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:45:02 GMT +0000.
2021-05-24 02:14:34 <icinga-miraheze> RECOVERY - lama.madooa.org - LetsEncrypt on sslhost is OK: OK - Certificate 'lama.madooa.org' will expire on Sat 07 Aug 2021 08:51:42 GMT +0000.
2021-05-24 02:14:47 <icinga-miraheze> RECOVERY - cp13 HTTPS on cp13 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2272 bytes in 0.568 second response time
2021-05-24 02:14:56 <icinga-miraheze> RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.41, 1.27, 1.58
2021-05-24 02:14:58 <dmehus> Cocopuff2018, okay, thank you. I've updated the task. No one with server access is available at the moment
2021-05-24 02:15:20 <icinga-miraheze> RECOVERY - history.sdtef.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Sat 07 Aug 2021 03:36:43 GMT +0000.
2021-05-24 02:15:20 <icinga-miraheze> RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Fri 30 Jul 2021 05:01:06 GMT +0000.
2021-05-24 02:15:22 <icinga-miraheze> RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Sat 07 Aug 2021 07:44:17 GMT +0000.
2021-05-24 02:15:22 <icinga-miraheze> RECOVERY - www.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.petrawiki.org' will expire on Fri 25 Jun 2021 05:43:40 GMT +0000.
2021-05-24 02:15:31 <icinga-miraheze> RECOVERY - rangpurpedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'rangpurpedia.xyz' will expire on Fri 02 Jul 2021 06:08:10 GMT +0000.
2021-05-24 02:15:34 <icinga-miraheze> RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk'; will expire on Wed 30 Jun 2021 11:38:37 GMT +0000.
2021-05-24 02:15:39 <icinga-miraheze> RECOVERY - planetnomads.speedbomb.de - LetsEncrypt on sslhost is OK: OK - Certificate 'planetnomads.speedbomb.de' will expire on Sat 07 Aug 2021 04:19:27 GMT +0000.
2021-05-24 02:15:41 <icinga-miraheze> RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Sat 07 Aug 2021 07:05:38 GMT +0000.
2021-05-24 02:15:44 <icinga-miraheze> RECOVERY - icinga.bots.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate '*.bots.miraheze.wiki' will expire on Tue 17 Aug 2021 04:56:02 GMT +0000.
2021-05-24 02:15:49 <icinga-miraheze> RECOVERY - cp12 Stunnel Http for mw10 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15299 bytes in 9.362 second response time
2021-05-24 02:15:54 <icinga-miraheze> RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Sat 07 Aug 2021 03:00:47 GMT +0000.
2021-05-24 02:15:54 <icinga-miraheze> RECOVERY - wiki.yumeka.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.icu' will expire on Sun 13 Jun 2021 13:30:44 GMT +0000.
2021-05-24 02:15:58 <icinga-miraheze> RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Sat 07 Aug 2021 07:54:14 GMT +0000.
2021-05-24 02:16:00 <icinga-miraheze> RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Sat 07 Aug 2021 07:38:27 GMT +0000.
2021-05-24 02:16:06 <icinga-miraheze> RECOVERY - mr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mr.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:44:14 GMT +0000.
2021-05-24 02:16:10 <icinga-miraheze> RECOVERY - cp13 Stunnel Http for mw10 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15299 bytes in 9.062 second response time
2021-05-24 02:16:12 <icinga-miraheze> RECOVERY - wiki.candela.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.candela.digital' will expire on Sat 07 Aug 2021 03:42:24 GMT +0000.
2021-05-24 02:16:23 <icinga-miraheze> RECOVERY - cp10 Stunnel Http for mw10 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 8.150 second response time
2021-05-24 02:16:23 <dmehus> Wikis still down
2021-05-24 02:16:23 <icinga-miraheze> RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 9.487 second response time
2021-05-24 02:16:26 <icinga-miraheze> RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl'; will expire on Fri 20 Aug 2021 22:34:45 GMT +0000.
2021-05-24 02:16:27 <icinga-miraheze> RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 22%
2021-05-24 02:16:28 <icinga-miraheze> RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Sat 07 Aug 2021 03:28:42 GMT +0000.
2021-05-24 02:16:28 <icinga-miraheze> RECOVERY - wiki.rpointer.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rpointer.ml' will expire on Sat 24 Jul 2021 12:18:00 GMT +0000.
2021-05-24 02:16:31 <icinga-miraheze> RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Sat 07 Aug 2021 10:12:24 GMT +0000.
2021-05-24 02:16:34 <icinga-miraheze> PROBLEM - wiki.umbrel.tech - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.umbrel.tech' expires in 8 day(s) (Tue 01 Jun 2021 18:51:21 GMT +0000).
2021-05-24 02:16:37 <icinga-miraheze> RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:45:41 GMT +0000.
2021-05-24 02:16:37 <icinga-miraheze> RECOVERY - matttest.mtwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'matttest.mtwiki.cf' will expire on Tue 06 Jul 2021 03:50:41 GMT +0000.
2021-05-24 02:16:43 <icinga-miraheze> RECOVERY - lg.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lg.gyaanipedia.com' will expire on Sat 07 Aug 2021 01:59:00 GMT +0000.
2021-05-24 02:16:44 <icinga-miraheze> RECOVERY - kkutu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kkutu.wiki' will expire on Sat 07 Aug 2021 04:15:13 GMT +0000.
2021-05-24 02:16:48 <icinga-miraheze> RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2274 bytes in 7.984 second response time
2021-05-24 02:16:52 <icinga-miraheze> RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Sat 07 Aug 2021 04:18:14 GMT +0000.
2021-05-24 02:16:54 <icinga-miraheze> RECOVERY - wiki.thedev.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thedev.gq' will expire on Tue 17 Aug 2021 05:08:15 GMT +0000.
2021-05-24 02:16:56 <icinga-miraheze> RECOVERY - erikapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.erikapedia.com'; will expire on Fri 20 Aug 2021 22:28:16 GMT +0000.
2021-05-24 02:17:50 <icinga-miraheze> PROBLEM - wiki.invinciblewellbeing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.invinciblewellbeing.com could not be found
2021-05-24 02:17:54 <icinga-miraheze> PROBLEM - www.winenjoy.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.winenjoy.net could not be found
2021-05-24 02:17:54 <icinga-miraheze> PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found
2021-05-24 02:17:59 <icinga-miraheze> PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found
2021-05-24 02:18:06 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 02:18:08 <Cocopuff2018> @dmehus look like we gonna have to wait till sre wakes up
2021-05-24 02:18:19 <icinga-miraheze> PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found
2021-05-24 02:18:24 <icinga-miraheze> PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gesamtschule-nordkirchen.de could not be found
2021-05-24 02:18:25 <icinga-miraheze> PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikien.wildterra2.com could not be found
2021-05-24 02:18:26 <icinga-miraheze> PROBLEM - biblestrength.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for biblestrength.net could not be found
2021-05-24 02:18:28 <icinga-miraheze> PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found
2021-05-24 02:18:29 <icinga-miraheze> PROBLEM - wiki.wilderyogi.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wilderyogi.eu could not be found
2021-05-24 02:18:29 <icinga-miraheze> PROBLEM - translate.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for translate.petrawiki.org could not be found
2021-05-24 02:18:30 <icinga-miraheze> PROBLEM - aman.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for aman.awiki.org could not be found
2021-05-24 02:18:40 <icinga-miraheze> PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fbpml.org could not be found
2021-05-24 02:18:44 <icinga-miraheze> PROBLEM - famedata.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for famedata.org could not be found
2021-05-24 02:18:47 <dmehus> Cocopuff2018, I can access `metawiki`, but load times are atrocious
2021-05-24 02:18:53 <dmehus> and it's hit or miss
2021-05-24 02:19:06 <dmehus> that'll be another 3 hours before Reception123 is up
2021-05-24 02:19:10 <icinga-miraheze> RECOVERY - cp11 Stunnel Http for mw10 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15285 bytes in 9.637 second response time
2021-05-24 02:19:40 <icinga-miraheze> RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
2021-05-24 02:19:58 <icinga-miraheze> PROBLEM - wiki.joust.ro - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.joust.ro could not be found
2021-05-24 02:19:59 <icinga-miraheze> PROBLEM - wikilukas.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikilukas.tk could not be found
2021-05-24 02:20:02 <icinga-miraheze> PROBLEM - wiki.s23.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.s23.org could not be found
2021-05-24 02:20:05 <icinga-miraheze> PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found
2021-05-24 02:20:06 <icinga-miraheze> RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20889 bytes in 2.046 second response time
2021-05-24 02:20:09 <icinga-miraheze> PROBLEM - wiki.xysspon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.xysspon.com could not be found
2021-05-24 02:20:09 <Cocopuff2018> @dmehus can you load up any other wikis besides meta?
2021-05-24 02:20:10 <icinga-miraheze> PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.triplescripts.org could not be found
2021-05-24 02:20:17 <Cocopuff2018> @dmehus can you load up any other wikis besides meta?
2021-05-24 02:20:41 <icinga-miraheze> PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.80, 20.15, 15.99
2021-05-24 02:21:00 <icinga-miraheze> RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
2021-05-24 02:22:19 <Cocopuff2018> @dhemus looks like its back for me
2021-05-24 02:22:25 <dmehus> Cocopuff2018, let me try
2021-05-24 02:22:37 <icinga-miraheze> PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.21, 22.59, 17.33
2021-05-24 02:22:49 <Cocopuff2018> k.
2021-05-24 02:23:07 <dmehus> yes, but still slow load times
2021-05-24 02:23:18 <dmehus> It seems to be improving, but not great overall
2021-05-24 02:24:32 <icinga-miraheze> PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.93, 20.44, 17.15
2021-05-24 02:24:35 <icinga-miraheze> RECOVERY - wiki.invinciblewellbeing.com - reverse DNS on sslhost is OK: rDNS OK - wiki.invinciblewellbeing.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:24:36 <Cocopuff2018> gonna have to wait for sre
2021-05-24 02:24:41 <icinga-miraheze> RECOVERY - www.winenjoy.net - reverse DNS on sslhost is OK: rDNS OK - www.winenjoy.net reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:24:42 <icinga-miraheze> RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: rDNS OK - storytime.jdstroy.cf reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:24:54 <icinga-miraheze> RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: rDNS OK - www.lab612.at reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:08 <icinga-miraheze> RECOVERY - wikien.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikien.wildterra2.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:15 <icinga-miraheze> RECOVERY - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is OK: rDNS OK - wiki.gesamtschule-nordkirchen.de reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:15 <icinga-miraheze> RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:18 <icinga-miraheze> RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:18 <icinga-miraheze> RECOVERY - biblestrength.net - reverse DNS on sslhost is OK: rDNS OK - biblestrength.net reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:21 <icinga-miraheze> RECOVERY - translate.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - translate.petrawiki.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:28 <icinga-miraheze> RECOVERY - wiki.wilderyogi.eu - reverse DNS on sslhost is OK: rDNS OK - wiki.wilderyogi.eu reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:30 <icinga-miraheze> RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fbpml.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:31 <icinga-miraheze> RECOVERY - aman.awiki.org - reverse DNS on sslhost is OK: rDNS OK - aman.awiki.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:25:38 <icinga-miraheze> RECOVERY - famedata.org - reverse DNS on sslhost is OK: rDNS OK - famedata.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:26:02 <dmehus> Cocopuff2018, yep. Task updated hopefully for the final time till SRE examines it
2021-05-24 02:26:31 <icinga-miraheze> RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.14, 18.09, 16.71
2021-05-24 02:26:41 <icinga-miraheze> RECOVERY - wikilukas.tk - reverse DNS on sslhost is OK: rDNS OK - wikilukas.tk reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:26:51 <icinga-miraheze> RECOVERY - wiki.joust.ro - reverse DNS on sslhost is OK: rDNS OK - wiki.joust.ro reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:26:55 <icinga-miraheze> RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: rDNS OK - wiki.triplescripts.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:26:58 <icinga-miraheze> RECOVERY - wiki.xysspon.com - reverse DNS on sslhost is OK: rDNS OK - wiki.xysspon.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:26:59 <icinga-miraheze> RECOVERY - wiki.s23.org - reverse DNS on sslhost is OK: rDNS OK - wiki.s23.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:27:02 <icinga-miraheze> RECOVERY - wiki.climatechange.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.climatechange.ai reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:29:26 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.43, 5.73, 7.94
2021-05-24 02:31:12 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.22, 6.00, 7.84
2021-05-24 02:31:44 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 02:33:32 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 6.44, 7.87
2021-05-24 02:35:30 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 7.23, 8.01
2021-05-24 02:37:27 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.87, 7.38, 7.96
2021-05-24 02:41:07 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.07, 5.37, 6.58
2021-05-24 02:43:04 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.61, 5.38, 6.65
2021-05-24 02:43:21 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.81, 6.73, 7.76
2021-05-24 02:45:27 <Crepusculum> Sorry I was away also. Is everything working now? Matomo and Login says those two are back up now, is everything working? dmehus, Cocopuff2018 ?
2021-05-24 02:51:12 <icinga-miraheze> PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.77, 1.36, 1.20
2021-05-24 02:51:22 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.65, 5.66, 6.58
2021-05-24 02:53:12 <icinga-miraheze> RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.83, 1.17, 1.15
2021-05-24 02:54:05 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 02:55:19 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.05, 5.76, 6.65
2021-05-24 02:57:12 <icinga-miraheze> PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.27, 1.72, 1.41
2021-05-24 02:59:12 <icinga-miraheze> RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.99, 1.46, 1.35
2021-05-24 02:59:21 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 6.78, 6.85
2021-05-24 03:00:39 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.03, 6.93, 6.67
2021-05-24 03:02:36 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.07, 6.66, 6.61
2021-05-24 03:03:19 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.97, 7.53, 7.12
2021-05-24 03:06:29 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.13, 6.83, 6.73
2021-05-24 03:07:39 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 03:08:26 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.70, 6.09, 6.47
2021-05-24 03:09:19 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.02, 7.65, 7.38
2021-05-24 03:21:20 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.79, 6.08, 6.70
2021-05-24 03:26:52 <dmehus> Crepusculum, seems to be, as far as I know. Should be investigated, though, when Paladox and Southparkfan are around
2021-05-24 03:28:33 <Crepusculum> dmehus: Yeah definitely should be investigated. Seems to be SSL or DNS related maybe.
2021-05-24 03:41:06 <dmehus> Crepusculum, yeah, maybe. I'm thinking the cache proxies OOM'ed
2021-05-24 03:43:33 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 03:45:11 <Not-4c5b> ['miraheze/services'] 'MirahezeSSLBot' pushed '' commit to 'master' [+0/-0/±1] 'https://git.io/Jsb4v'
2021-05-24 03:45:12 <Not-4c5b> ['miraheze/services'] 'MirahezeSSLBot' 'cae525' - BOT: Updating services config for wikis
2021-05-24 03:49:39 <Crepusculum> dmehus: even Matomo went down, I'm not positive how that one really works but would that cause Matomo to be down also?
2021-05-24 04:18:23 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 04:34:25 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 04:43:54 <dmehus> Crepusculum, yeah, I mean, Nginx is the web server, so if that OOM'ed, in theory, Matomo could stay running, assuming it's a separate process; it just wouldn't stay accessible. But maybe that suggests it's broader than Nginx/Varnish, and was a cache proxy issue or maybe something OOM'ed and brought down the whole cloud server?
2021-05-24 04:47:55 <Crepusculum> dmehus: Yeah I thought it might've been the whole cloud server tbh, but if that happened we would have had way more alerts I think. Something caused it, and that definitely needs looked into as it was a total outage. I'm just glad it self resolved otherwise we'd still be down as I have access to handle issues beyond the MediaWiki servers, so couldn't have fixed it anyways, and pal_adox and SPF seemed to be away, and
2021-05-24 04:47:55 <Crepusculum> Reception_123 was likely asleep. (Put underscores in usernames so I don't ping them)
2021-05-24 04:47:58 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 04:49:52 <dmehus> Crepusculum, yeah, though there were quite a few alerts, I think, but yeah, definitely needs SRE infrastructure team to investigate thoroughly
2021-05-24 04:50:45 <dmehus> lol I mean underscores work, or we could invent nicknames for them...like pdox, SPF, and Reception
2021-05-24 04:51:25 <Crepusculum> Yeah lol could've.
2021-05-24 04:52:45 <Crepusculum> dmehus: And yeah definitely needs investigating for what happened shouldn't have, no one was doing anything on any servers at the time. It lasted 14 minutes (Matomo outage lasted 4 minutes is all though)
2021-05-24 05:07:04 <dmehus> Crepusculum, hrm, I think it lasted a little longer than that. It started 3 minutes before 7 pm my time and went at least until 7:14 pm and wasn't normalized in terms of page load times until 7:20-7:25 pm
2021-05-24 05:09:08 <Not-4c5b> ['miraheze/mw-config'] 'Universal-Omega' pushed '' commit to 'Universal-Omega-patch-1' [+0/-0/±1] 'https://git.io/JsbPk'
2021-05-24 05:09:10 <Not-4c5b> ['miraheze/mw-config'] 'Universal-Omega' 'eb1e3e2' - Remove two deprecated password policy checks
2021-05-24 05:09:11 <Not-4c5b> ['mw-config'] 'Universal-Omega' created branch 'Universal-Omega-patch-1' - 'https://git.io/vbvb3'
2021-05-24 05:09:13 <Not-4c5b> ['mw-config'] 'Universal-Omega' opened pull request '#3916': Remove two deprecated password policy checks - 'https://git.io/JsbPL'
2021-05-24 05:10:15 <miraheze-github> miraheze/mw-config - Universal-Omega the build passed.
2021-05-24 05:10:22 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 05:10:26 <Crepusculum> dmehus: not according to status.miraheze.wiki .
2021-05-24 05:16:09 <dmehus> Crepusculum, hrm, I'm just going by when I and others couldn't access Miraheze wikis and we couldn't access them from 6:57 pm through 7:14 pm my time, though there may have been some intermittent access in there
2021-05-24 05:18:15 <Crepusculum> dmehus: According to status.miraheze.wiki it started at 7:00 and was back at 7:14 (your time) but maybe it is off by 3 minutes?
2021-05-24 05:32:51 <dmehus> Crepusculum, yeah, that is about right, though it was definitely started 1-3 minutes before 7 pm. And by back it wasn't fully back. 502 errors did intermittently persist for another several minutes after 7:14 pm
2021-05-24 05:35:45 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 6.22, 5.41
2021-05-24 05:37:44 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.75, 5.97, 5.40
2021-05-24 05:39:50 <dmehus> Morning, RhinosF1, and Reception123 :)
2021-05-24 05:40:17 <Reception123> hi. I've seen the error but so far have no idea how it happened
2021-05-24 05:40:18 <RhinosF1> dmehus: saw your task
2021-05-24 05:40:29 <RhinosF1> Reception123: logs?
2021-05-24 05:40:30 <Reception123> checked OVH and it doesn't seem to be a problem from there
2021-05-24 05:40:33 <dmehus> Reception123, RhinosF1: ack, yeah I don't know
2021-05-24 05:40:35 <RhinosF1> On cp12/13
2021-05-24 05:41:02 <Reception123> was it only cp12/cp13? If so why wouldn't the other ones just take over?
2021-05-24 05:41:04 <dmehus> My working theory is Nginx or Varnish OOM'ed badly, and brought down cache proxies and cloud server
2021-05-24 05:41:48 <Reception123> while good in theory, the kernel log shows no OOMs on any cp servers in the past 8 hours
2021-05-24 05:41:53 <RhinosF1> Reception123: they should have
2021-05-24 05:41:54 <dmehus> oh :(
2021-05-24 05:41:59 <dmehus> hrm
2021-05-24 05:42:00 <Reception123> unless I've done something wrong
2021-05-24 05:42:00 <RhinosF1> Check varnish logs
2021-05-24 05:42:11 <RhinosF1> It doesn't look like an oom
2021-05-24 05:42:27 <dmehus> hopes it was a DoS attack
2021-05-24 05:42:44 <dmehus> hopes he didn't just get RhinosF1 into panic mode with that thought bubble
2021-05-24 05:43:21 <RhinosF1> Reception123: cp3 shows faults
2021-05-24 05:43:25 <dmehus> yeah
2021-05-24 05:43:34 <RhinosF1> Just didn't see it in Doug's icinga alerts
2021-05-24 05:43:41 <dmehus> I do think cp3 was a major source of the problem, but it wasn't the only one
2021-05-24 05:43:50 <Reception123> https://www.irccloud.com/pastebin/6fz29FIy/
2021-05-24 05:43:51 <MirahezeBot> [ Snippet | IRCCloud ] - www.irccloud.com
2021-05-24 05:43:54 <dmehus> yeah that might've come later
2021-05-24 05:44:15 <Reception123> nothing that tells us anything new really
2021-05-24 05:44:21 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 05:44:22 <dmehus> oh
2021-05-24 05:44:43 <RhinosF1> Reception123: that means mw at fault
2021-05-24 05:45:06 <RhinosF1> Oh f
2021-05-24 05:45:11 <RhinosF1> That's some load
2021-05-24 05:45:18 <dmehus> yeah
2021-05-24 05:45:24 <dmehus> there were major load issues at the time
2021-05-24 05:45:48 <dmehus> and GDNSD servers were down according to the icinga logs
2021-05-24 05:46:06 <Reception123> there does seem to be a strange OOM error on mw8/mw9 then but the time doesn't match
2021-05-24 05:46:11 <dmehus> oh
2021-05-24 05:46:16 <dmehus> what time, Reception123?
2021-05-24 05:46:16 <Reception123> one is way before the error and one is after
2021-05-24 05:46:28 <Reception123> 23:52 UTC and 03:00 UTC
2021-05-24 05:46:41 <Reception123> and the error would've been at 2:00 UTC from what I've seen
2021-05-24 05:46:41 <dmehus> I got my first 502 or 500 (can't remember which) at 6:57 my time
2021-05-24 05:46:52 <RhinosF1> dmehus: gdnsd means all cps went down
2021-05-24 05:47:00 <dmehus> RhinosF1, ah
2021-05-24 05:47:12 <dmehus> Reception123, oh that's a good hour before
2021-05-24 05:47:24 <Reception123> yeah, so I'd say that's not really related
2021-05-24 05:47:52 <dmehus> possibly, but not necessarily. Hard to say, I guess
2021-05-24 05:48:03 <RhinosF1> Reception123: paste the log
2021-05-24 05:48:19 <Reception123> the varnish one you mean?
2021-05-24 05:48:38 <RhinosF1> No the strange OOM
2021-05-24 05:48:41 <Reception123> ah
2021-05-24 05:49:05 <Reception123> convert invoked oom-killer: gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
2021-05-24 05:49:10 <RhinosF1> dmehus: major load issues is an understatement
2021-05-24 05:49:19 <Reception123> 23:52 UTC on mw8 and 03:00 UTC on mw9
2021-05-24 05:49:21 <dmehus> RhinosF1, heh yeah
2021-05-24 05:49:57 <RhinosF1> Reception123: yeah that not helping
2021-05-24 05:50:15 <Reception123> yeah, though I'm unsure what else it could be
2021-05-24 05:50:31 <dmehus> heh I wish paladox or Southparkfan were around :D
2021-05-24 05:51:26 <RhinosF1> Reception123: check /etc/cron.d/
2021-05-24 05:51:37 <RhinosF1> Look if anything matches the timings
2021-05-24 05:52:50 <Reception123> I mean there's not much in there, nothing that matches
2021-05-24 05:53:05 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 05:53:23 <RhinosF1> Reception123: I was hoping a task in there would have started the right time
2021-05-24 05:53:41 <RhinosF1> Crepusculum, paladox, Southparkfan: I'm now out of ideas
2021-05-24 05:53:59 <Reception123> well there's only puppet and a php sessionclean
2021-05-24 05:54:00 <RhinosF1> Because I can't do top on something that's already happened
2021-05-24 05:56:25 <Crepusculum> Yeah I don't know what could've caused it either. I tried looking a bit earlier after it happened but didn't find anything that could cause it. I was away when it happened though so was about 45 minutes after.
2021-05-24 05:57:37 <Reception123> well I think even if people were there during that time there wouldn't have been much one could've done since even after we can't figure out what caused it
2021-05-24 05:58:26 <RhinosF1> I'm at a loss now
2021-05-24 06:06:39 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 06:18:57 <RhinosF1> Reception123: we had a spike of uncached requests
2021-05-24 06:19:07 <RhinosF1> So now we need to look at what was a cache miss then
2021-05-24 06:19:17 <Reception123> hmm
2021-05-24 06:20:09 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 6.29, 5.79
2021-05-24 06:20:56 <RhinosF1> Reception123: does graylog cache proxy logs show cache miss / hit
2021-05-24 06:21:02 <RhinosF1> On requests
2021-05-24 06:22:07 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.32, 6.05, 5.75
2021-05-24 06:22:21 <Reception123> varnish isn't on graylog
2021-05-24 06:22:39 <RhinosF1> Under nginx logs
2021-05-24 06:22:44 <RhinosF1> Probably
2021-05-24 06:22:55 <dmehus> Reception123, ah we haven't centralized the Varnish logs, I gather?
2021-05-24 06:23:12 <Reception123> yeah
2021-05-24 06:23:19 <dmehus> ah
2021-05-24 06:23:46 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.80, 6.24, 5.31
2021-05-24 06:24:01 <Reception123> searched for 'miss' and there doesn't seem to be anything
2021-05-24 06:25:20 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.13, 7.39, 6.10
2021-05-24 06:25:48 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.27, 7.43, 5.85
2021-05-24 06:30:34 <RhinosF1> Reception123: if it's there it'll be a filter
2021-05-24 06:30:41 <RhinosF1> cache_hit or something
2021-05-24 06:35:48 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 06:37:17 <RhinosF1> Reception123: varnishtop -i BereqURL
2021-05-24 06:38:17 <RhinosF1> But that might only have worked in the moment
2021-05-24 06:39:48 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.83, 7.17, 7.23
2021-05-24 06:43:48 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.98, 6.06, 6.77
2021-05-24 06:45:19 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.48, 7.43, 7.90
2021-05-24 06:47:56 <RhinosF1> Reception123: we're gonna have to manually read logs
2021-05-24 06:47:59 <RhinosF1> Well you are
2021-05-24 06:49:09 <dmehus> RhinosF1, like access logs, or system logs?
2021-05-24 06:49:22 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 06:49:35 <RhinosF1> dmehus: varnish ones
2021-05-24 06:51:14 <Reception123> I can look a bit now but I won't really be around soon until the afternoon
2021-05-24 06:51:19 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.53, 5.52, 6.80
2021-05-24 06:51:39 <RhinosF1> Reception123: okay
2021-05-24 06:51:51 <RhinosF1> I wish I could tell you there was an easy way
2021-05-24 06:51:57 <RhinosF1> But even wikimedia sre don't have one
2021-05-24 06:52:15 <dmehus> RhinosF1, ah
2021-05-24 07:04:59 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 07:11:37 <icinga-miraheze> PROBLEM - services4 APT on services4 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (3 critical updates).
2021-05-24 07:12:26 <icinga-miraheze> PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (4 critical updates).
2021-05-24 07:12:53 <icinga-miraheze> PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (3 critical updates).
2021-05-24 07:13:03 <icinga-miraheze> PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (2 critical updates).
2021-05-24 07:16:05 <icinga-miraheze> PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (3 critical updates).
2021-05-24 07:19:54 <icinga-miraheze> PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (2 critical updates).
2021-05-24 07:21:03 <icinga-miraheze> PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 108 packages available for upgrade (3 critical updates).
2021-05-24 07:21:04 <icinga-miraheze> PROBLEM - gluster3 APT on gluster3 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates).
2021-05-24 07:21:16 <icinga-miraheze> PROBLEM - mem2 APT on mem2 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates).
2021-05-24 07:21:20 <icinga-miraheze> PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 73 packages available for upgrade (2 critical updates).
2021-05-24 07:21:43 <icinga-miraheze> PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 73 packages available for upgrade (2 critical updates).
2021-05-24 07:22:14 <icinga-miraheze> PROBLEM - test4 APT on test4 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates).
2021-05-24 07:22:41 <icinga-miraheze> PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (2 critical updates).
2021-05-24 07:23:13 <icinga-miraheze> PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 51 packages available for upgrade (2 critical updates).
2021-05-24 07:23:41 <icinga-miraheze> PROBLEM - mon2 APT on mon2 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (2 critical updates).
2021-05-24 07:24:29 <icinga-miraheze> PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (2 critical updates).
2021-05-24 07:24:53 <icinga-miraheze> PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (4 critical updates).
2021-05-24 07:27:05 <icinga-miraheze> PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (3 critical updates).
2021-05-24 07:27:16 <icinga-miraheze> PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (2 critical updates).
2021-05-24 07:27:38 <icinga-miraheze> PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates).
2021-05-24 07:31:37 <icinga-miraheze> PROBLEM - cp13 APT on cp13 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (2 critical updates).
2021-05-24 07:32:27 <icinga-miraheze> PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (4 critical updates).
2021-05-24 07:32:38 <icinga-miraheze> PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates).
2021-05-24 07:34:29 <icinga-miraheze> PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates).
2021-05-24 07:34:54 <icinga-miraheze> PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates).
2021-05-24 07:35:01 <icinga-miraheze> PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 34 packages available for upgrade (4 critical updates).
2021-05-24 07:35:19 <icinga-miraheze> PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates).
2021-05-24 07:35:23 <icinga-miraheze> PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates).
2021-05-24 07:35:37 <icinga-miraheze> PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (4 critical updates).
2021-05-24 07:36:05 <icinga-miraheze> PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates).
2021-05-24 07:36:15 <icinga-miraheze> PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 32 packages available for upgrade (4 critical updates).
2021-05-24 07:37:36 <icinga-miraheze> PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (2 critical updates).
2021-05-24 07:38:49 <icinga-miraheze> PROBLEM - ldap2 APT on ldap2 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (2 critical updates).
2021-05-24 07:59:19 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 08:28:25 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 08:41:59 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 08:57:34 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 09:11:07 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 09:33:27 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 09:47:23 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 10:03:24 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 10:53:28 <icinga-miraheze> PROBLEM - ping6 on dbbackup1 is CRITICAL: CRITICAL - Destination Unreachable (fe80::f816:3eff:fe94:15f5)
2021-05-24 10:57:04 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 11:20:57 <icinga-miraheze> RECOVERY - ping6 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 91.18 ms
2021-05-24 11:26:08 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 12:00:39 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 12:10:12 <Not-4c5b> ['miraheze/MirahezeMagic'] 'translatewiki' pushed '' commit to 'master' [+0/-0/±18] 'https://git.io/JsAJ8'
2021-05-24 12:10:13 <Not-4c5b> ['miraheze/MirahezeMagic'] 'translatewiki' '0789f2' - Localisation updates from https://translatewiki.net.
2021-05-24 12:10:14 <MirahezeBot> [ Main page - translatewiki.net ] - translatewiki.net
2021-05-24 12:10:15 <Not-4c5b> ['miraheze/ManageWiki'] 'translatewiki' pushed '' commit to 'master' [+0/-0/±4] 'https://git.io/JsAJR'
2021-05-24 12:10:16 <Not-4c5b> ['miraheze/ManageWiki'] 'translatewiki' '84a6fe' - Localisation updates from https://translatewiki.net.
2021-05-24 12:10:17 <MirahezeBot> [ Main page - translatewiki.net ] - translatewiki.net
2021-05-24 12:10:18 <Not-4c5b> ['miraheze/CreateWiki'] 'translatewiki' pushed '' commit to 'master' [+0/-0/±2] 'https://git.io/JsAJ0'
2021-05-24 12:10:19 <Not-4c5b> ['miraheze/CreateWiki'] 'translatewiki' '3b1f7b' - Localisation updates from https://translatewiki.net.
2021-05-24 12:10:20 <MirahezeBot> [ Main page - translatewiki.net ] - translatewiki.net
2021-05-24 12:11:11 <miraheze-github> miraheze/ManageWiki - translatewiki the build passed.
2021-05-24 12:11:20 <miraheze-github_> miraheze/MirahezeMagic - translatewiki the build passed.
2021-05-24 12:11:22 <miraheze-github> miraheze/CreateWiki - translatewiki the build passed.
2021-05-24 12:16:25 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 12:17:03 <MacFan4000> here we go with banning azure IPs again
2021-05-24 12:30:12 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 12:34:34 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 6.13, 5.63
2021-05-24 12:36:29 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.68, 5.82, 5.57
2021-05-24 12:45:58 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 12:52:18 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 6.76, 6.06
2021-05-24 12:54:15 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.37, 7.05, 6.23
2021-05-24 12:56:10 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.38, 6.63, 6.17
2021-05-24 13:10:33 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.74, 7.04, 6.56
2021-05-24 13:14:22 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.69, 6.44, 6.41
2021-05-24 13:20:24 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 13:30:53 <icinga-miraheze> PROBLEM - wiki.twilightsignal.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.twilightsignal.com could not be found
2021-05-24 13:30:54 <icinga-miraheze> PROBLEM - test2.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for test2.miraheze.org could not be found
2021-05-24 13:32:27 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.01, 6.52, 6.08
2021-05-24 13:33:36 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.19, 7.91, 7.10
2021-05-24 13:34:27 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.18, 7.60, 6.53
2021-05-24 13:36:26 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.26, 6.84, 6.19
2021-05-24 13:36:27 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.87, 7.22, 6.53
2021-05-24 13:37:25 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.15, 7.43, 7.12
2021-05-24 13:37:35 <icinga-miraheze> RECOVERY - wiki.twilightsignal.com - reverse DNS on sslhost is OK: rDNS OK - wiki.twilightsignal.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 13:37:47 <icinga-miraheze> RECOVERY - test2.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - test2.miraheze.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 13:38:27 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.20, 6.24, 6.05
2021-05-24 13:38:28 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.12, 6.76, 6.44
2021-05-24 13:46:58 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.72, 6.60, 6.80
2021-05-24 13:50:49 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.02, 6.84, 6.86
2021-05-24 13:52:50 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.00, 6.65, 6.80
2021-05-24 13:55:34 <Not-4c5b> ['miraheze/puppet'] 'Reception123' pushed '' commit to 'Reception123-patch-2' [+0/-0/±1] 'https://git.io/JsARx'
2021-05-24 13:55:36 <Not-4c5b> ['miraheze/puppet'] 'Reception123' 'e8e6f69' - set test3 branch to REL1_36
2021-05-24 13:55:37 <Not-4c5b> ['puppet'] 'Reception123' created branch 'Reception123-patch-2' - 'https://git.io/vbiAS'
2021-05-24 13:55:39 <Not-4c5b> ['puppet'] 'Reception123' opened pull request '#1761': set test3 branch to REL1_36 - 'https://git.io/JsARh'
2021-05-24 13:56:51 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 14:04:40 <Not-4c5b> ['miraheze/dns'] 'Reception123' pushed '' commit to 'master' [+1/-0/±0] 'https://git.io/JsAuT'
2021-05-24 14:04:41 <Not-4c5b> ['miraheze/dns'] 'Reception123' '27c762' - add thestarsareright.org zone
2021-05-24 14:04:42 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.64, 7.64, 6.65
2021-05-24 14:05:04 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.45, 6.78, 6.20
2021-05-24 14:06:42 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.88, 6.87, 6.49
2021-05-24 14:07:00 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.19, 6.77, 6.29
2021-05-24 14:08:52 <Not-4c5b> ['miraheze/ssl'] 'Reception123' pushed '' commits to 'master' [+2/-0/±2] 'https://git.io/JsAzJ'
2021-05-24 14:08:53 <Not-4c5b> ['miraheze/ssl'] 'Reception123' 'd24087b' - add thestarsareright.org cert
2021-05-24 14:08:55 <Not-4c5b> ['miraheze/ssl'] 'Reception123' 'c51c5d' - add electrowiki.org cert
2021-05-24 14:10:37 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 14:12:09 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.64, 6.96, 6.74
2021-05-24 14:12:46 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.41, 6.37, 6.40
2021-05-24 14:14:03 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.17, 6.43, 6.58
2021-05-24 14:18:47 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.42, 7.07, 6.40
2021-05-24 14:20:53 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.10, 6.69, 6.35
2021-05-24 14:22:12 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.71, 8.12, 7.12
2021-05-24 14:23:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.58, 6.72, 6.17
2021-05-24 14:24:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.78, 7.30, 6.69
2021-05-24 14:25:44 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.21, 6.74
2021-05-24 14:26:10 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 7.94, 7.31
2021-05-24 14:26:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.32, 6.62, 6.52
2021-05-24 14:27:19 <Not-4c5b> ['miraheze/mw-config'] 'Reception123' created branch 'Reception123-patch-1' 'https://git.io/JsAVM'
2021-05-24 14:27:21 <Not-4c5b> ['miraheze/mw-config'] 'Reception123' created branch 'Reception123-patch-1-1' 'https://git.io/JsAVM'
2021-05-24 14:27:22 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.47, 6.63, 6.30
2021-05-24 14:27:22 <Not-4c5b> ['mw-config'] 'Reception123' created branch 'Reception123-patch-1' - 'https://git.io/vbvb3'
2021-05-24 14:27:24 <Not-4c5b> ['mw-config'] 'Reception123' created branch 'Reception123-patch-1-1' - 'https://git.io/vbvb3'
2021-05-24 14:27:26 <Not-4c5b> ['mw-config'] 'Reception123' opened pull request '#3917': extendedconfirmed protection level for lhmnwiki T7346 - 'https://git.io/JsAVS'
2021-05-24 14:28:28 <miraheze-github> miraheze/mw-config - Reception123 the build passed.
2021-05-24 14:28:40 <Not-4c5b> ['mw-config'] 'Reception123' closed pull request '#3917': extendedconfirmed protection level for lhmnwiki T7346 - 'https://git.io/JsAVS'
2021-05-24 14:28:42 <Not-4c5b> ['miraheze/mw-config'] 'Reception123' pushed '' commit to 'master' [+0/-0/±1] 'https://git.io/JsAwv'
2021-05-24 14:28:43 <Not-4c5b> ['miraheze/mw-config'] 'Reception123' 'e98289' - extendedconfirmed protection level for lhmnwiki T7346 (#3917)
2021-05-24 14:28:44 <Not-4c5b> ['mw-config'] 'Reception123' deleted branch 'Reception123-patch-1-1' - 'https://git.io/vbvb3'
2021-05-24 14:28:46 <Not-4c5b> ['miraheze/mw-config'] 'Reception123' deleted branch 'Reception123-patch-1-1'
2021-05-24 14:29:40 <miraheze-github> miraheze/mw-config - Reception123 the build passed.
2021-05-24 14:29:41 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.56, 6.74, 6.64
2021-05-24 14:30:33 <Not-4c5b> ['miraheze/mw-config'] 'Reception123' pushed '' commit to 'master' [+0/-0/±1] 'https://git.io/JsAw3'
2021-05-24 14:30:35 <Not-4c5b> ['miraheze/mw-config'] 'Reception123' '608a31' - fix mistake made in previous PR #3893
2021-05-24 14:31:32 <miraheze-github> miraheze/mw-config - Reception123 the build passed.
2021-05-24 14:32:09 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.45, 8.16, 7.54
2021-05-24 14:33:37 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.04, 8.01, 7.13
2021-05-24 14:34:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 6.92, 6.58
2021-05-24 14:36:53 <icinga-miraheze> PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
2021-05-24 14:37:03 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.34, 7.95, 6.96
2021-05-24 14:38:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.53, 8.60, 7.29
2021-05-24 14:38:58 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.76, 7.49, 6.92
2021-05-24 14:40:52 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.24, 7.76, 7.08
2021-05-24 14:42:47 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.57, 7.74, 7.15
2021-05-24 14:43:31 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.41, 7.97, 7.64
2021-05-24 14:44:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.93, 7.51, 7.30
2021-05-24 14:46:28 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 14:50:32 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.89, 7.73, 7.20
2021-05-24 14:51:24 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.11, 7.05, 7.18
2021-05-24 14:52:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.73, 7.68, 7.25
2021-05-24 14:53:23 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.54, 7.21, 7.22
2021-05-24 14:55:21 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 7.38, 7.27
2021-05-24 14:56:53 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.09, 5.69, 6.51
2021-05-24 14:57:20 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.76, 6.41, 6.93
2021-05-24 14:57:57 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 7.55, 7.92
2021-05-24 14:59:18 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.89, 6.02, 6.74
2021-05-24 14:59:54 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 15:00:32 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.06, 7.32, 7.21
2021-05-24 15:01:55 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.69, 7.85, 7.93
2021-05-24 15:02:53 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.83, 6.96, 6.76
2021-05-24 15:03:16 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.53, 8.75, 7.61
2021-05-24 15:04:54 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.43, 8.75, 7.48
2021-05-24 15:06:50 <RhinosF1> Reception123: hi
2021-05-24 15:06:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.57, 7.89, 7.32
2021-05-24 15:08:52 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 15:09:18 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.16, 7.67, 7.62
2021-05-24 15:09:58 <Not-4c5b> ['puppet'] 'RhinosF1' edited pull request '#1735': Switch test3 to 1.36 stuff - 'https://git.io/JOeT9'
2021-05-24 15:10:30 <Not-4c5b> ['puppet'] 'RhinosF1' commented on pull request '#1761': set test3 branch to REL1_36 - 'https://git.io/JsAXl'
2021-05-24 15:11:01 <RhinosF1> Reception123: use mine
2021-05-24 15:12:10 <Reception123> ah, good idea so we don't need to manually pull the branch
2021-05-24 15:12:30 <Cocopuff2018> hello
2021-05-24 15:12:34 <Not-4c5b> ['puppet'] 'Reception123' closed pull request '#1761': set test3 branch to REL1_36 - 'https://git.io/JsARh'
2021-05-24 15:12:46 <RhinosF1> Reception123: feel free to go whenever
2021-05-24 15:12:49 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.70, 7.46, 7.73
2021-05-24 15:12:56 <RhinosF1> I will do schema changes this evening
2021-05-24 15:13:02 <RhinosF1> Hi Cocopuff2018
2021-05-24 15:13:04 <Cocopuff2018> any updates on special Multi edit
2021-05-24 15:13:07 <RhinosF1> No
2021-05-24 15:13:39 <Cocopuff2018> well then might not be able to use it anymoe
2021-05-24 15:14:16 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.12, 7.40, 7.92
2021-05-24 15:14:37 <Reception123> will do later today then, when Crepusculum is also around
2021-05-24 15:15:13 <RhinosF1> Reception123: I wouldn't worry about breaking test3
2021-05-24 15:16:19 <Reception123> yeah, that's not a worry. we have the backup worst case even if it would be fine even without it
2021-05-24 15:16:28 <Reception123> probably will do it in an hour or so
2021-05-24 15:16:42 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.67, 7.89, 7.82
2021-05-24 15:16:43 <Cocopuff2018> big outage last night
2021-05-24 15:17:03 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.28, 6.49, 6.75
2021-05-24 15:17:24 <RhinosF1> !log test3 -> 1.36 in ~1 hour, please don't panic at exceptions until you see a message saying 1.36 schema changes done - See T7117
2021-05-24 15:17:27 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 15:17:34 <RhinosF1> Cocopuff2018: we know
2021-05-24 15:17:42 <RhinosF1> Not sure why yet
2021-05-24 15:18:44 <Reception123> RhinosF1: oh and if we could I'd like to fix the Flow issue before we upgrade so we can properly test it after
2021-05-24 15:18:56 <RhinosF1> Reception123: which flow issue?
2021-05-24 15:19:10 <Reception123> RhinosF1: exception and the thing is when I try to search the trace ID in graylog nothing shows up
2021-05-24 15:19:16 <Reception123> you can get the error by visiting any talk page
2021-05-24 15:19:20 <Reception123> like Talk:Test
2021-05-24 15:19:26 <RhinosF1> Reception123: link?
2021-05-24 15:19:54 <Reception123> https://test3.miraheze.org/wiki/Talk:Test
2021-05-24 15:19:56 <MirahezeBot> [ Unknown topic - Test3 ] - test3.miraheze.org
2021-05-24 15:20:03 <Cocopuff2018> the wikis seemed to come back up in up in a short period of time
2021-05-24 15:20:15 <Cocopuff2018> in a short period of time
2021-05-24 15:20:34 <RhinosF1> Reception123: nothing in graylog at all?
2021-05-24 15:20:53 <Reception123> no, that's the strange part, I tried to search IDs and nothing shows up
2021-05-24 15:21:01 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.74, 6.84, 6.84
2021-05-24 15:21:44 <Cocopuff2018> does miraheze have some kind of backup thing for if the system goes down?
2021-05-24 15:22:18 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 15:22:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.24, 7.86, 7.85
2021-05-24 15:22:39 <RhinosF1> Cocopuff2018: we don't have redundancy
2021-05-24 15:22:48 <RhinosF1> Backups yes to a degree
2021-05-24 15:23:06 <RhinosF1> But you couldn't run off them
2021-05-24 15:24:11 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.16, 7.51, 7.66
2021-05-24 15:24:56 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.83, 6.55, 6.75
2021-05-24 15:25:32 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.24, 6.23, 6.79
2021-05-24 15:26:10 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.47, 6.81, 7.39
2021-05-24 15:26:51 <MH-Discord> <Lake> test3 is going to 1.36 already?
2021-05-24 15:27:28 <Reception123> yeah, we're preparing for the 1.36 release which shouldn't be in way too long
2021-05-24 15:28:04 <Cocopuff2018> perhaps by July it should be ready?
2021-05-24 15:28:09 <RhinosF1> I hope so
2021-05-24 15:28:34 <RhinosF1> We'll have Debian upgrades shortly after too but hopefully you won't notice them directly
2021-05-24 15:28:56 <Cocopuff2018> how long does upgrading take? to finish
2021-05-24 15:29:13 <RhinosF1> 1.36 a while
2021-05-24 15:29:21 <RhinosF1> Normally we do two hours for production
2021-05-24 15:30:08 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.17, 7.73, 7.60
2021-05-24 15:30:48 <Cocopuff2018> after 1.36 i am sure it will be a few years before 1.37 is released
2021-05-24 15:30:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.59, 7.13, 6.89
2021-05-24 15:32:06 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.72, 7.41, 7.51
2021-05-24 15:32:54 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.62, 6.39, 6.66
2021-05-24 15:34:05 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 7.78, 7.63
2021-05-24 15:34:20 <Reception123> not a few years, usually it's two every year
2021-05-24 15:34:47 <Reception123> but because of COVID-19 there was only one last year
2021-05-24 15:35:14 <Cocopuff2018> The 1.36. 0 stable release is expected to come out in May 2021
2021-05-24 15:35:35 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.03, 6.87
2021-05-24 15:36:46 <Cocopuff2018> found a whole list of task for 1.36
2021-05-24 15:36:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.02, 7.21, 6.97
2021-05-24 15:37:33 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.54, 6.46, 6.68
2021-05-24 15:38:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.42, 7.35, 7.35
2021-05-24 15:38:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.65, 6.64, 6.79
2021-05-24 15:40:01 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.28, 7.84, 7.74
2021-05-24 15:40:32 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.54, 6.94, 7.19
2021-05-24 15:41:31 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 6.86, 6.83
2021-05-24 15:43:30 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.10, 7.46, 7.04
2021-05-24 15:43:58 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.02, 8.76, 8.09
2021-05-24 15:44:24 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 15:44:33 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.36, 7.57, 7.38
2021-05-24 15:44:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.76, 7.23, 6.98
2021-05-24 15:46:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.68, 7.22, 7.28
2021-05-24 15:47:28 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.58, 7.33, 7.07
2021-05-24 15:49:57 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.01, 7.60, 7.83
2021-05-24 15:51:26 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.97, 7.83, 7.27
2021-05-24 15:52:53 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.60, 6.55, 6.78
2021-05-24 15:53:25 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.68, 7.66, 7.28
2021-05-24 15:57:52 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 15:58:31 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.10, 6.12, 6.77
2021-05-24 16:03:18 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 6.91, 6.90
2021-05-24 16:03:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.20, 7.25, 7.26
2021-05-24 16:04:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.40, 6.75, 6.80
2021-05-24 16:04:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.99, 6.83, 6.63
2021-05-24 16:05:18 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.43, 6.81, 6.88
2021-05-24 16:05:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.40, 7.33, 7.29
2021-05-24 16:08:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.60, 6.06, 6.41
2021-05-24 16:09:16 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.36, 6.33, 6.65
2021-05-24 16:10:31 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.28, 6.26, 6.61
2021-05-24 16:12:48 <RhinosF1> Reception123: ping for puppet stuff
2021-05-24 16:13:15 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 16:13:51 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.45, 6.07, 6.76
2021-05-24 16:20:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.20, 6.13, 6.00
2021-05-24 16:24:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.37, 6.58, 6.24
2021-05-24 16:25:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.69, 6.89, 6.71
2021-05-24 16:27:54 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.65, 6.66
2021-05-24 16:34:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.14, 8.20, 6.93
2021-05-24 16:35:08 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.21, 7.27, 6.38
2021-05-24 16:36:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.27, 6.91, 6.35
2021-05-24 16:36:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 7.50, 6.84
2021-05-24 16:37:06 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.20, 6.82, 6.32
2021-05-24 16:37:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 6.99, 6.73
2021-05-24 16:40:31 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.30, 6.11, 6.20
2021-05-24 16:41:53 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 6.66, 6.68
2021-05-24 16:42:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.99, 6.37, 6.60
2021-05-24 16:43:02 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.93, 6.22, 6.30
2021-05-24 16:49:22 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.81, 7.72, 6.91
2021-05-24 16:50:56 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 6.93, 6.50
2021-05-24 16:51:52 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.17, 7.37, 6.93
2021-05-24 16:52:55 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.33, 6.63, 6.44
2021-05-24 16:53:52 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.11, 7.29, 6.96
2021-05-24 16:55:07 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.47, 8.04, 7.23
2021-05-24 16:55:52 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.83, 8.00, 7.26
2021-05-24 16:57:01 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.16, 7.71, 7.21
2021-05-24 16:58:53 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 6.88, 6.60
2021-05-24 16:58:57 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.79, 8.01, 7.38
2021-05-24 16:59:52 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 7.71, 7.32
2021-05-24 17:00:53 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.36, 7.39, 7.23
2021-05-24 17:02:52 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.11, 6.36, 6.44
2021-05-24 17:03:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 8.12, 7.59
2021-05-24 17:05:25 <Not-4c5b> ['mw-config'] 'dmehus' commented on commit '608a3109f05488074dd5928d61d2ac44fd36c5f' - 'https://git.io/JsxIl'
2021-05-24 17:05:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.93, 7.46, 7.40
2021-05-24 17:08:34 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.06, 6.10, 6.64
2021-05-24 17:13:52 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.16, 6.26, 6.78
2021-05-24 17:20:47 <Reception123> paladox: about the error last night do you have any idea which service might've actually caused the error?
2021-05-24 17:20:49 <Reception123> that seems a bit unclear
2021-05-24 17:21:13 <paladox> i gave some hints but it's unclear.
2021-05-24 17:21:18 <paladox> anything in the logs at the time?
2021-05-24 17:21:52 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 7.17, 6.95
2021-05-24 17:21:59 <Reception123> I mean I mostly just checked for OOMs and there wasn't anything related
2021-05-24 17:23:54 <paladox> https://grafana.miraheze.org/d/UzwQVt1mk/php-fpm-appservers?viewPanel=14&orgId=1&var-host=mw11.miraheze.org:9253&from=now-24h&to=now doesn't show it was a php-fpm running out of childs error
2021-05-24 17:23:55 <MirahezeBot> [ Grafana ] - grafana.miraheze.org
2021-05-24 17:24:21 <paladox> oh
2021-05-24 17:24:22 <paladox> wait
2021-05-24 17:24:24 <paladox> i misread it
2021-05-24 17:24:55 <paladox> that makes it look like all the php-fpm childs were in use for ~20 minutes
2021-05-24 17:26:09 <Reception123> so then you think it's rather the mw* servers that were the cause
2021-05-24 17:27:10 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 17:29:53 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.49, 6.63, 6.80
2021-05-24 17:30:18 <paladox> i mean looking at the graphs for the dbs i see extra activity.
2021-05-24 17:35:55 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.04, 6.83, 6.81
2021-05-24 17:39:55 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.71, 6.72, 6.75
2021-05-24 17:40:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.57, 7.55, 6.79
2021-05-24 17:40:43 <Not-4c5b> ['puppet'] 'Reception123' closed pull request '#1735': Switch test3 to 1.36 stuff - 'https://git.io/JOeT9'
2021-05-24 17:40:45 <Not-4c5b> ['miraheze/puppet'] 'Reception123' pushed '' commit to 'master' [+0/-0/±2] 'https://git.io/Jsx3R'
2021-05-24 17:40:46 <Not-4c5b> ['miraheze/puppet'] 'RhinosF1' 'dc30497' - Switch test3 to 1.36 stuff (#1735)
2021-05-24 17:44:11 <RhinosF1> Reception123: no it isnt
2021-05-24 17:44:16 <RhinosF1> dmehus: *
2021-05-24 17:45:56 <dmehus> RhinosF1, oh referring to my test3wiki edit?
2021-05-24 17:46:10 <dmehus> so we're just upgrading test3wiki to MW 1.35.2 then?
2021-05-24 17:46:25 <RhinosF1> dmehus: no we're updating to 1.36
2021-05-24 17:46:28 <dmehus> oh
2021-05-24 17:46:32 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.59, 6.74, 6.63
2021-05-24 17:46:35 <dmehus> well it shows as MW 1.35.2
2021-05-24 17:46:45 <RhinosF1> Because it's not done yet
2021-05-24 17:46:47 <dmehus> when Reception123 updated the test3wiki sitenotice
2021-05-24 17:46:48 <dmehus> ah
2021-05-24 17:47:02 <Reception123> dmehus: yeah heh it's because I used {{CURRENTVERSION}}
2021-05-24 17:47:04 <dmehus> Will he be disabling Puppet on test3 then?
2021-05-24 17:47:13 <Reception123> nope, running it!
2021-05-24 17:47:26 <Reception123> because the version is set in puppet this time
2021-05-24 17:47:33 <dmehus> Reception123, ah, but that should still go by test3wiki's version?
2021-05-24 17:47:47 <Reception123> well the version is set to REL1_36 in puppet
2021-05-24 17:47:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.35, 6.95, 6.86
2021-05-24 17:47:59 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.91, 6.93, 6.36
2021-05-24 17:48:00 <RhinosF1> dmehus: give it time to sync
2021-05-24 17:48:29 <dmehus> Reception123, ah, so it's just a matter of needing Puppet to run then
2021-05-24 17:48:35 <dmehus> in about 12 minutes or so
2021-05-24 17:48:38 <Reception123> heh, so far there's an error
2021-05-24 17:48:48 <Reception123> well I'm not going to wait, I ran it now manually :)
2021-05-24 17:49:02 <Reception123> but anyway don't expect it to work, RhinosF1 needs to do the SQL changes
2021-05-24 17:49:03 <dmehus> can't you make it so test3wiki is isolated from the mw servers as far as Puppet goes?
2021-05-24 17:49:12 <Reception123> dmehus: yes, that is exactly what we did!
2021-05-24 17:49:13 <dmehus> why is it causing errors on the mw servers?
2021-05-24 17:49:18 <Reception123> it's not
2021-05-24 17:49:32 <dmehus> oh that's just load errors
2021-05-24 17:49:37 <Reception123> yeah, unrelated
2021-05-24 17:49:39 <dmehus> but why'd you say there's a problem?
2021-05-24 17:50:21 <Reception123> because puppet did fail, but on test3
2021-05-24 17:50:25 <dmehus> hopes RhinosF1 is fully awake when he does the schema changes :D
2021-05-24 17:50:31 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.80, 7.00, 6.78
2021-05-24 17:50:38 <dmehus> Reception123, ah, okay, that makes sense then thanks :)
2021-05-24 17:50:46 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.17, 7.36, 6.41
2021-05-24 17:51:57 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.63, 7.49, 6.67
2021-05-24 17:52:50 <RhinosF1> !log rhinos@test3:/srv/mediawiki/config$ sudo -u www-data git checkout --track origin/REL1_36
2021-05-24 17:52:57 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 17:53:55 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.73, 7.47, 6.75
2021-05-24 17:54:35 <icinga-miraheze> PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.060 second response time
2021-05-24 17:54:43 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.16, 7.81, 6.89
2021-05-24 17:55:56 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.15, 6.62, 6.53
2021-05-24 17:56:34 <RhinosF1> !log test3: puppet is a snail, 500 expected
2021-05-24 17:56:38 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 17:57:50 <RhinosF1> Reception123: can you update composer
2021-05-24 17:57:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.24, 8.27, 7.52
2021-05-24 17:58:06 <Reception123> urgh, update it?
2021-05-24 17:58:36 <Reception123> hopes it's just php composer.phar update
2021-05-24 17:58:42 <icinga-miraheze> RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 17:58:45 <Reception123> */me
2021-05-24 17:58:58 <RhinosF1> Reception123: wget -O composer.phar https://getcomposer.org/composer-1.phar | php && php composer.phar install --no-dev
2021-05-24 17:59:03 <Not-4c5b> ['miraheze/puppet'] 'paladox' pushed '' commit to 'master' [+0/-0/±1] 'https://git.io/JsxZB'
2021-05-24 17:59:04 <RhinosF1> should be
2021-05-24 17:59:05 <Not-4c5b> ['miraheze/puppet'] 'paladox' 'ac1babe' - graylog: Upgrade elasticsearch, mongodb and graylog
2021-05-24 17:59:32 <Not-4c5b> ['puppet'] 'dmehus' opened pull request '#1762': Super Gnomey edit to remove extra space - 'https://git.io/JsxZz'
2021-05-24 17:59:52 <Reception123> paladox: if you're updating stuff mind also doing Matomo? :)
2021-05-24 17:59:56 <Reception123> and not sure if phab needs updating too
2021-05-24 18:00:01 <paladox> oh there's an update for matomo?
2021-05-24 18:00:05 <Reception123> yup
2021-05-24 18:00:06 <paladox> sure i can look at that in a min
2021-05-24 18:00:10 <Reception123> thanks!
2021-05-24 18:00:16 <Not-4c5b> ['puppet'] 'paladox' closed pull request '#1762': Super Gnomey edit to remove extra space - 'https://git.io/JsxZz'
2021-05-24 18:00:17 <Not-4c5b> ['miraheze/puppet'] 'paladox' pushed '' commit to 'master' [+0/-0/±1] 'https://git.io/JsxZK'
2021-05-24 18:00:19 <Not-4c5b> ['miraheze/puppet'] 'dmehus' '43ecd7' - Super Gnomey edit to remove extra space (#1762)
2021-05-24 18:00:23 <dmehus> thanks paladox
2021-05-24 18:00:33 <paladox> Reception123: https://github.com/Universal-Omega/Monaco/pull/3 is merged, probs want to deploy it
2021-05-24 18:00:33 <RhinosF1> !log test3: puppet finished a minute back, Reception123 is updating composer
2021-05-24 18:00:33 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.71, 7.26
2021-05-24 18:00:34 <MirahezeBot> [ Remove unused hook by paladox · Pull Request #3 · Universal-Omega/Monaco · GitHub ] - github.com
2021-05-24 18:00:36 <paladox> to 1.35 + 1.36
2021-05-24 18:00:37 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 18:00:50 <Reception123> paladox: will do
2021-05-24 18:00:56 <paladox> thanks!
2021-05-24 18:01:13 <icinga-miraheze> PROBLEM - wiki.mussa.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mussa.id could not be found
2021-05-24 18:01:23 <Not-4c5b> ['miraheze/mediawiki'] 'Reception123' pushed '' commit to 'REL1_36' [+0/-0/±1] 'https://git.io/JsxZQ'
2021-05-24 18:01:24 <Not-4c5b> ['miraheze/mediawiki'] 'Reception123' 'b7e782' - Update DataTransfer
2021-05-24 18:01:46 <icinga-miraheze> PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fc.songcontests.eu could not be found
2021-05-24 18:01:51 <icinga-miraheze> PROBLEM - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phabdigests.bots.miraheze.wiki could not be found
2021-05-24 18:01:52 <icinga-miraheze> PROBLEM - www.erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.erikapedia.com could not be found
2021-05-24 18:01:52 <icinga-miraheze> PROBLEM - miraheze.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ga could not be found
2021-05-24 18:01:53 <icinga-miraheze> PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found
2021-05-24 18:01:54 <icinga-miraheze> PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found
2021-05-24 18:02:00 <icinga-miraheze> PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found
2021-05-24 18:02:00 <icinga-miraheze> PROBLEM - wiki.ssij.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssij.se could not be found
2021-05-24 18:02:01 <icinga-miraheze> PROBLEM - www.tradwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.tradwiki.org could not be found
2021-05-24 18:02:05 <icinga-miraheze> PROBLEM - ao90.pinho.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ao90.pinho.org could not be found
2021-05-24 18:02:06 <icinga-miraheze> PROBLEM - es.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.gyaanipedia.com could not be found
2021-05-24 18:02:18 <RhinosF1> Reception123: let me know when composer has ran
2021-05-24 18:02:33 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.93, 7.51, 7.22
2021-05-24 18:02:38 <Not-4c5b> ['miraheze/mediawiki'] 'Reception123' pushed '' commit to 'REL1_36' [+0/-0/±1] 'https://git.io/JsxZj'
2021-05-24 18:02:39 <Not-4c5b> ['miraheze/mediawiki'] 'Reception123' 'e56774' - Update Monaco (skin)
2021-05-24 18:02:42 <Reception123> RhinosF1: it has just now
2021-05-24 18:03:12 <Not-4c5b> ['miraheze/mediawiki'] 'Reception123' pushed '' commit to 'REL1_35' [+0/-0/±1] 'https://git.io/JsxnI'
2021-05-24 18:03:13 <Not-4c5b> ['miraheze/mediawiki'] 'Reception123' '257d95' - Update Monaco (skin)
2021-05-24 18:03:20 <RhinosF1> Reception123: does merge message list & rebuild i18n run?
2021-05-24 18:03:44 <icinga-miraheze> PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 6.58, 3.95, 2.52
2021-05-24 18:03:54 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 18:03:55 <Reception123> RhinosF1: heh, you wish
2021-05-24 18:03:55 <Reception123> Fatal error: Uncaught Exception: Unable to open file /srv/mediawiki/w/extensions/RightFunctions/extension.json: filemtime(): stat failed for /srv/mediawiki/w/extensions/RightFunctions/extension.json in /srv/mediawiki/w/includes/registration/ExtensionRegistry.php:177
2021-05-24 18:03:56 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.33, 8.34, 7.16
2021-05-24 18:04:17 <Reception123> there is indeed no extension.json for that
2021-05-24 18:04:36 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.88, 8.60, 7.66
2021-05-24 18:04:46 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.64, 8.90, 7.63
2021-05-24 18:04:50 <Reception123> even though there is one here: https://github.com/wikimedia/mediawiki-extensions-RightFunctions/tree/REL1_36
2021-05-24 18:04:51 <MirahezeBot> [ GitHub - wikimedia/mediawiki-extensions-RightFunctions at REL1_36 ] - github.com
2021-05-24 18:05:08 <Reception123> so something is up
2021-05-24 18:05:40 <icinga-miraheze> PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.16, 3.85, 2.68
2021-05-24 18:07:39 <icinga-miraheze> RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.53, 3.31, 2.62
2021-05-24 18:07:40 <icinga-miraheze> PROBLEM - wiki.mnzp.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mnzp.xyz could not be found
2021-05-24 18:07:42 <icinga-miraheze> PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seamly.net could not be found
2021-05-24 18:07:44 <Crepusculum> Reception123: looking. I can't remember if that's one I did or not
2021-05-24 18:08:29 <icinga-miraheze> RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: rDNS OK - fc.songcontests.eu reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:35 <icinga-miraheze> RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: rDNS OK - wiki.zymonic.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:35 <Reception123> Crepusculum: well it wouldn't be your fault as there is an extension.json in the GH repo
2021-05-24 18:08:36 <icinga-miraheze> RECOVERY - miraheze.ga - reverse DNS on sslhost is OK: rDNS OK - miraheze.ga reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:39 <Reception123> just not on the server
2021-05-24 18:08:39 <icinga-miraheze> RECOVERY - www.erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - www.erikapedia.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:41 <icinga-miraheze> RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: rDNS OK - madgenderscience.wiki reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:46 <Reception123> so it means the server seems to have not pulled the extension correctly
2021-05-24 18:08:47 <icinga-miraheze> RECOVERY - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phabdigests.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:48 <icinga-miraheze> RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:54 <icinga-miraheze> RECOVERY - www.tradwiki.org - reverse DNS on sslhost is OK: rDNS OK - www.tradwiki.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:56 <icinga-miraheze> RECOVERY - ao90.pinho.org - reverse DNS on sslhost is OK: rDNS OK - ao90.pinho.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:08:58 <icinga-miraheze> RECOVERY - wiki.ssij.se - reverse DNS on sslhost is OK: rDNS OK - wiki.ssij.se reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:09:05 <icinga-miraheze> PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.73, 20.81, 17.76
2021-05-24 18:09:05 <icinga-miraheze> RECOVERY - es.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - es.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:09:18 <Reception123> now is not the time icinga-miraheze
2021-05-24 18:10:26 <Crepusculum> Reception123: try `cd /srv/mediawiki/w/extensions/RightFunctions && sudo -u www-data git fetch && sudo -u www-data git reset --hard origin/REL1_36`
2021-05-24 18:10:45 <icinga-miraheze> RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20928 bytes in 0.709 second response time
2021-05-24 18:10:50 <Not-4c5b> ['miraheze/puppet'] 'paladox' pushed '' commit to 'master' [+0/-0/±1] 'https://git.io/Jsxcy'
2021-05-24 18:10:52 <Not-4c5b> ['miraheze/puppet'] 'paladox' 'e63fd1' - matomo: Upgrade to 4.3.0
2021-05-24 18:10:55 <icinga-miraheze> PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 18:11:04 <Reception123> Crepusculum: will do, though I hope it's not a wider issue when no extensions worked
2021-05-24 18:11:12 <Reception123> as we wouldn't want to deal with that with prod when we do the real upgrade
2021-05-24 18:11:30 <Crepusculum> Yeah it's just to debug
2021-05-24 18:11:40 <icinga-miraheze> PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 2.32, 3.66, 3.02
2021-05-24 18:12:21 <icinga-miraheze> PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 18:12:24 <Reception123> Crepusculum: that did it yeah
2021-05-24 18:12:46 <icinga-miraheze> PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 18:12:55 <Reception123> RhinosF1: Crepusculum MessageFiles/rebuild running
2021-05-24 18:13:01 <icinga-miraheze> RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.33, 19.95, 18.15
2021-05-24 18:13:08 <icinga-miraheze> PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 18:13:47 <icinga-miraheze> RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.35, 3.12, 2.90
2021-05-24 18:13:53 <dmehus> lol, RightFunctions was the first extension Universal_Omega ever installed for me actually, iirc
2021-05-24 18:14:05 <dmehus> so he probably missed adding an extension.json file
2021-05-24 18:14:13 <Crepusculum> Great Reception123, maybe it was an issue with puppet pulling correct commit from git?
2021-05-24 18:14:15 <dmehus> Is extension.json now mandatory with MW 1.36?
2021-05-24 18:14:28 <icinga-miraheze> RECOVERY - wiki.mnzp.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.mnzp.xyz reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:14:30 <icinga-miraheze> RECOVERY - wiki.seamly.net - reverse DNS on sslhost is OK: rDNS OK - wiki.seamly.net reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:14:30 <icinga-miraheze> PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 18:14:31 <Crepusculum> No. But every extension on Miraheze should support it.
2021-05-24 18:14:34 <icinga-miraheze> PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 18:14:35 <Reception123> ^
2021-05-24 18:15:07 <Reception123> rebuild passed 'en' and test3 is still down
2021-05-24 18:15:08 <Crepusculum> Dmehus: RightFunctions didn't have extension.json until 1.36.
2021-05-24 18:15:08 <dmehus> Crepusculum, but why would it error now then, when we've had it installed since September?
2021-05-24 18:15:13 <Crepusculum> 1.36 dmehus
2021-05-24 18:15:16 <dmehus> Crepusculum, ah okay, interesting
2021-05-24 18:15:17 <Reception123> Crepusculum: it will be hard to see any error with that RemovePII spam so let me know when you find the error
2021-05-24 18:15:30 <Reception123> dmehus: it was a server/git pull error rather than anything else
2021-05-24 18:15:34 <Crepusculum> Reception123: I see no test3 errors in graylog.
2021-05-24 18:15:39 <dmehus> Reception123, ah
2021-05-24 18:15:41 <paladox> matomo upgraded now
2021-05-24 18:15:41 <Reception123> Crepusculum: https://graylog.miraheze.org/search?q=%22test3wiki%22&rangetype=relative&relative=300
2021-05-24 18:15:42 <icinga-miraheze> PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.78, 4.03, 3.21
2021-05-24 18:15:46 <Reception123> paladox: thanks :)
2021-05-24 18:15:52 <icinga-miraheze> PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::1c30/cpweb
2021-05-24 18:16:00 <icinga-miraheze> PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hr.petrawiki.org could not be found
2021-05-24 18:16:05 <icinga-miraheze> PROBLEM - www.sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sdiy.info could not be found
2021-05-24 18:16:06 <icinga-miraheze> PROBLEM - history.estill.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.estill.org could not be found
2021-05-24 18:16:11 <Reception123> oh whoops it's actually not on test3 itself, but the spam still is bad, even worse that it's on prod
2021-05-24 18:16:25 <icinga-miraheze> PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
2021-05-24 18:16:34 <icinga-miraheze> PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for antiguabarbudacalypso.com could not be found
2021-05-24 18:16:34 <icinga-miraheze> PROBLEM - www.lukaibrisimovic.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lukaibrisimovic.org could not be found
2021-05-24 18:16:35 <icinga-miraheze> PROBLEM - wiki.caker18.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.caker18.com could not be found
2021-05-24 18:16:36 <icinga-miraheze> PROBLEM - planetnomads.speedbomb.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for planetnomads.speedbomb.de could not be found
2021-05-24 18:16:39 <icinga-miraheze> PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found
2021-05-24 18:16:40 <icinga-miraheze> PROBLEM - isv.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for isv.orain.org could not be found
2021-05-24 18:16:40 <icinga-miraheze> PROBLEM - www.johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.johanloopmans.nl could not be found
2021-05-24 18:16:42 <icinga-miraheze> PROBLEM - wiki.seredrau.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seredrau.xyz could not be found
2021-05-24 18:16:43 <icinga-miraheze> PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found
2021-05-24 18:16:43 <icinga-miraheze> PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found
2021-05-24 18:16:44 <icinga-miraheze> PROBLEM - www.allthetropes.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.allthetropes.org could not be found
2021-05-24 18:16:55 <icinga-miraheze> PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.025 second response time
2021-05-24 18:17:22 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp12.miraheze.org
2021-05-24 18:17:35 <Crepusculum> Reception123: see nothing worrisome when searching RemovePII on all streams.
2021-05-24 18:17:41 <icinga-miraheze> PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.71, 3.60, 3.15
2021-05-24 18:17:45 <RhinosF1> Reception123: where we stand now?
2021-05-24 18:18:20 <Not-4c5b> ['miraheze/puppet'] 'paladox' pushed '' commit to 'master' [+0/-0/±1] 'https://git.io/JsxWI'
2021-05-24 18:18:21 <Not-4c5b> ['miraheze/puppet'] 'paladox' 'f27625f' - matomo: Deny access to /tmp and /lang
2021-05-24 18:18:25 <icinga-miraheze> RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
2021-05-24 18:18:26 <Reception123> Crepusculum: did you try the link I sent you?
2021-05-24 18:18:35 <Crepusculum> Yeah it found nothing.
2021-05-24 18:18:38 <Reception123> RhinosF1: rebuild worked but there's still something up
2021-05-24 18:18:52 <Reception123> https://usercontent.irccloud-cdn.com/file/jOTgw0M9/image.png
2021-05-24 18:18:56 <Reception123> ^ Crepusculum this is what it looks like for me
2021-05-24 18:19:12 <RhinosF1> Reception123: logs?
2021-05-24 18:19:16 <icinga-miraheze> PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found
2021-05-24 18:19:16 <RhinosF1> And just exclude them then
2021-05-24 18:19:18 <icinga-miraheze> PROBLEM - spcodex.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for spcodex.wiki could not be found
2021-05-24 18:19:19 <icinga-miraheze> PROBLEM - monarchists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for monarchists.wiki could not be found
2021-05-24 18:19:19 <icinga-miraheze> PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found
2021-05-24 18:19:21 <icinga-miraheze> PROBLEM - uncyclopediakr.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for uncyclopediakr.miraheze.org could not be found
2021-05-24 18:19:22 <Reception123> RhinosF1: looking now
2021-05-24 18:19:24 <icinga-miraheze> PROBLEM - en.phgalaxy.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.phgalaxy.xyz could not be found
2021-05-24 18:19:27 <icinga-miraheze> PROBLEM - www.trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.trollpasta.com could not be found
2021-05-24 18:19:28 <icinga-miraheze> PROBLEM - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.marinebiodiversitymatrix.org could not be found
2021-05-24 18:19:41 <icinga-miraheze> RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.86, 2.97, 2.97
2021-05-24 18:19:50 <icinga-miraheze> RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
2021-05-24 18:19:50 <Reception123> urgh it's like icinga intentionally knows we're busy and wants to annoy us with SSL warnings
2021-05-24 18:20:04 <paladox> hit the timeout i think
2021-05-24 18:20:07 <Reception123> hmm mediawiki_wiki: test3wiki gives nothing at all
2021-05-24 18:20:19 <Reception123> will try to run a php script instead
2021-05-24 18:20:39 <Reception123> https://www.irccloud.com/pastebin/BGdA9bBX/
2021-05-24 18:20:40 <MirahezeBot> [ Snippet | IRCCloud ] - www.irccloud.com
2021-05-24 18:21:02 <Reception123> https://www.irccloud.com/pastebin/lQh4XWI7/
2021-05-24 18:21:03 <MirahezeBot> [ Snippet | IRCCloud ] - www.irccloud.com
2021-05-24 18:21:15 <paladox> looks like the extensions are messed up?
2021-05-24 18:21:18 <Reception123> Crepusculum: hmm, it is more widespread
2021-05-24 18:21:28 <Reception123> but then how did extension message files and rebuild work
2021-05-24 18:21:32 <paladox> root@test3:/home/paladox# cd /srv/mediawiki/w/extensions/CategorySortHeader
2021-05-24 18:21:41 <Reception123> paladox: yeah, though I don't get how/why
2021-05-24 18:21:57 <Reception123> it seems like they weren't pulled correctly, as they don't have extension.json's even though 1.36 added them
2021-05-24 18:22:14 <RhinosF1> Reception123: can you a git reset --hard origin then
2021-05-24 18:22:33 <Crepusculum> Reception123: those logs for RemovePII shouldn't be worried about. There isn't even a job for it in production yet, must've been when testing job on test3 but it not being on production (so I used runJobs on test3) so it can't be run on production hence the logs, I'd assume. Shouldn't effect anything.
2021-05-24 18:22:45 <icinga-miraheze> RECOVERY - hr.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - hr.petrawiki.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:22:52 <RhinosF1> I guess we need to sync extensions
2021-05-24 18:22:54 <icinga-miraheze> RECOVERY - history.estill.org - reverse DNS on sslhost is OK: rDNS OK - history.estill.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:22:55 <icinga-miraheze> RECOVERY - www.sdiy.info - reverse DNS on sslhost is OK: rDNS OK - www.sdiy.info reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:22:58 <Reception123> Crepusculum: it's still spammy though
2021-05-24 18:23:16 <icinga-miraheze> RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: rDNS OK - antiguabarbudacalypso.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:17 <icinga-miraheze> RECOVERY - www.lukaibrisimovic.org - reverse DNS on sslhost is OK: rDNS OK - www.lukaibrisimovic.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:19 <icinga-miraheze> RECOVERY - planetnomads.speedbomb.de - reverse DNS on sslhost is OK: rDNS OK - planetnomads.speedbomb.de reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:21 <Crepusculum> Reception123: yeah but I don't even see them in graylog.
2021-05-24 18:23:23 <icinga-miraheze> RECOVERY - wiki.seredrau.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.seredrau.xyz reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:28 <icinga-miraheze> RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: rDNS OK - www.zenbuddhism.info reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:28 <icinga-miraheze> RECOVERY - isv.orain.org - reverse DNS on sslhost is OK: rDNS OK - isv.orain.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:29 <icinga-miraheze> RECOVERY - www.johanloopmans.nl - reverse DNS on sslhost is OK: rDNS OK - www.johanloopmans.nl reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:31 <icinga-miraheze> RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:33 <icinga-miraheze> RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - pt.graalmilitary.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:33 <icinga-miraheze> RECOVERY - wiki.caker18.com - reverse DNS on sslhost is OK: rDNS OK - wiki.caker18.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:34 <icinga-miraheze> RECOVERY - www.allthetropes.org - reverse DNS on sslhost is OK: rDNS OK - www.allthetropes.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:23:38 <Reception123> Weird
2021-05-24 18:23:56 <Reception123> RhinosF1: I mean I could but it'd be nice to know the cause so we don't have any surprises when we do prod
2021-05-24 18:24:44 <RhinosF1> Reception123: for prod tbh we can just delete w
2021-05-24 18:24:47 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.85, 7.35, 7.88
2021-05-24 18:24:47 <paladox> Reception123: did you clone it like git clone https://github.com/miraheze/mediwiki --branch REL1_36 w --recursive?
2021-05-24 18:24:48 <MirahezeBot> [ Page not found · GitHub · GitHub ] - github.com
2021-05-24 18:24:49 <RhinosF1> That works
2021-05-24 18:25:01 <RhinosF1> paladox: let puppet update the branch afaik
2021-05-24 18:25:07 <paladox> oh
2021-05-24 18:25:17 <RhinosF1> I guess puppet not that smart
2021-05-24 18:25:41 <RhinosF1> Reception123: just do a reclone
2021-05-24 18:25:44 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.15, 6.84, 7.78
2021-05-24 18:25:50 <paladox> extensions fixed i think
2021-05-24 18:25:54 <RhinosF1> Oh right
2021-05-24 18:25:56 <icinga-miraheze> RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:25:59 <icinga-miraheze> RECOVERY - spcodex.wiki - reverse DNS on sslhost is OK: rDNS OK - spcodex.wiki reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:26:00 <RhinosF1> But for prod we can do a reclone
2021-05-24 18:26:00 <icinga-miraheze> RECOVERY - monarchists.wiki - reverse DNS on sslhost is OK: rDNS OK - monarchists.wiki reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:26:04 <Reception123> paladox: oh, what did you do?
2021-05-24 18:26:11 <icinga-miraheze> RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:26:12 <RhinosF1> Still 500s
2021-05-24 18:26:17 <icinga-miraheze> RECOVERY - www.trollpasta.com - reverse DNS on sslhost is OK: rDNS OK - www.trollpasta.com reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:26:18 <paladox> i deleted extensions/** and reset and then git submodule updated
2021-05-24 18:26:20 <icinga-miraheze> RECOVERY - uncyclopediakr.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - uncyclopediakr.miraheze.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:26:22 <icinga-miraheze> RECOVERY - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: rDNS OK - www.marinebiodiversitymatrix.org reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:26:23 <icinga-miraheze> RECOVERY - en.phgalaxy.xyz - reverse DNS on sslhost is OK: rDNS OK - en.phgalaxy.xyz reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:26:26 <Reception123> Fatal error: Uncaught Exception: Unable to open file /srv/mediawiki/w/extensions/RightFunctions/extension.json: filemtime(): stat failed for /srv/mediawiki/w/extensions/RightFunctions/extension.json in /srv/mediawiki/w/includes/registration/ExtensionRegistry.php:177
2021-05-24 18:26:30 <paladox> but origin/REL1_36 doesn't exist?
2021-05-24 18:26:30 <Reception123> ^ paladox now we're back to this
2021-05-24 18:26:47 <paladox> wrong branch checked out
2021-05-24 18:26:48 <paladox> let me fix
2021-05-24 18:26:51 <Reception123> ah
2021-05-24 18:28:48 <paladox> oh hold on
2021-05-24 18:28:54 <paladox> i'm just gonna reclone
2021-05-24 18:29:34 <RhinosF1> Yeah
2021-05-24 18:30:53 <icinga-miraheze> PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.025 second response time
2021-05-24 18:32:52 <icinga-miraheze> PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.131 second response time
2021-05-24 18:33:26 <RhinosF1> paladox: are you doing composer & i18n or do you want reception to take back?
2021-05-24 18:33:35 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.22, 7.29, 7.44
2021-05-24 18:33:42 <paladox> I can do that if you want, i don't want to step on any toes :)
2021-05-24 18:34:28 <RhinosF1> paladox: if you're free go ahead
2021-05-24 18:34:30 <icinga-miraheze> RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
2021-05-24 18:34:33 <paladox> sure
2021-05-24 18:34:40 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.54, 7.42, 7.52
2021-05-24 18:34:45 <icinga-miraheze> RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
2021-05-24 18:35:00 <icinga-miraheze> RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 18:35:04 <icinga-miraheze> RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
2021-05-24 18:35:13 <icinga-miraheze> RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 18:35:34 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 7.15, 7.37
2021-05-24 18:36:21 <icinga-miraheze> RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 18:36:38 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.14, 6.63, 7.22
2021-05-24 18:38:51 <icinga-miraheze> RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 1270 bytes in 0.017 second response time
2021-05-24 18:39:03 <dmehus> Reception123, and RhinosF1, `exambotwiki` finally hit Special:DeletedWikis this morning :)
2021-05-24 18:39:27 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 18:41:04 <Reception123> heh
2021-05-24 18:41:28 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.85, 5.93, 6.70
2021-05-24 18:41:44 <paladox> Fatal error: Uncaught ExtensionDependencyError: EventLogging requires EventBus to be installed.
2021-05-24 18:41:52 <Reception123> paladox: it's the 404 error
2021-05-24 18:41:59 <paladox> 404 error?
2021-05-24 18:42:00 <Reception123> so must be that CreateWiki cache one I think
2021-05-24 18:42:08 <Reception123> yeah, try to access test3wiki
2021-05-24 18:42:20 <Reception123> paladox: yup /cache is empty
2021-05-24 18:42:25 <paladox> https://www.irccloud.com/pastebin/MYp7P50Q/
2021-05-24 18:42:25 <MirahezeBot> [ Snippet | IRCCloud ] - www.irccloud.com
2021-05-24 18:42:32 <paladox> i don't see it on my laptop either
2021-05-24 18:42:52 <icinga-miraheze> PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2560 bytes in 0.011 second response time
2021-05-24 18:43:02 <RhinosF1> paladox: do we need to install it then?
2021-05-24 18:43:13 <RhinosF1> Reception123: first blocker https://phabricator.wikimedia.org/T283528 !
2021-05-24 18:43:14 <MirahezeBot> [ ⚓ T283528 $phpVersionCheck->checkVendorExistence(); message does not include $wgScriptPath in URL ] - phabricator.wikimedia.org
2021-05-24 18:43:15 <paladox> i'm thinking yes
2021-05-24 18:43:19 <paladox> gonna check tho
2021-05-24 18:43:46 <Reception123> heh
2021-05-24 18:43:51 <dmehus> We're not upgrading to php 8.0 or whatever at the same time, are we?
2021-05-24 18:43:56 <Reception123> forgot what the main file in cache is that gets it to work
2021-05-24 18:44:01 <Reception123> dmehus: no, that would be too much work at once
2021-05-24 18:44:02 <paladox> https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/21f985fa98256e6a38c20ef33dced5b581d5c3e2 says yeh
2021-05-24 18:44:02 <MirahezeBot> [ Use EventBus to send events for EventLogging PHP client · wikimedia/mediawiki-extensions-EventLogging@21f985f · GitHub ] - github.com
2021-05-24 18:44:04 <Reception123> we'll do that after
2021-05-24 18:44:10 <dmehus> Reception123, yeah definitely
2021-05-24 18:44:32 <Crepusculum> paladox: master doesn't require eventbus for EventLogging only REL1_36 does. Wonder why.
2021-05-24 18:44:49 <RhinosF1> dmehus: php 8 will be a while off
2021-05-24 18:45:07 <dmehus> still remembers the switch from php 3 to php 4 and all the web file extensions changed from `.php3` to `.php`
2021-05-24 18:45:07 <Reception123> copy/pasted test3wiki.json into the cache, but I don't remember if that's enough to fix
2021-05-24 18:45:08 <paladox> https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/2acaa7987b73dda1ee46e36f8fadfef8add6f0ae
2021-05-24 18:45:08 <MirahezeBot> [ Restore dependency-free nature of this extension · wikimedia/mediawiki-extensions-EventLogging@2acaa79 · GitHub ] - github.com
2021-05-24 18:45:12 <dmehus> RhinosF1, ack
2021-05-24 18:45:19 <paladox> i'll backport
2021-05-24 18:45:33 <RhinosF1> Thanks
2021-05-24 18:45:41 <Reception123> dmehus: the morale of the story is, a MediaWiki update never works out well from the start!
2021-05-24 18:45:48 <Reception123> there will always be something
2021-05-24 18:45:56 <paladox> https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/693656
2021-05-24 18:46:04 <dmehus> Reception123, heh. Does Paladox have full merge access on WMF?
2021-05-24 18:46:21 <paladox> RhinosF1: do you have any contacts to get that merged ASAP?
2021-05-24 18:46:28 <RhinosF1> paladox: yes
2021-05-24 18:46:41 <Zppix> Lego may be around
2021-05-24 18:46:42 <paladox> great! Could you ask them to merge it please :)
2021-05-24 18:46:52 <Reception123> dmehus: the message above answers that question
2021-05-24 18:46:54 <Zppix> paladox: you should
2021-05-24 18:47:03 <Zppix> Request cr+2 on mediawiki/*
2021-05-24 18:47:08 <MacFan4000> yeah, things like phabricator and icingaweb aren't even fully compatible with php8 yet.
2021-05-24 18:47:20 <RhinosF1> I asked Martin
2021-05-24 18:47:26 <paladox> thanks!
2021-05-24 18:47:35 <paladox> i don't think i would be approved cr+2
2021-05-24 18:47:40 <Reception123> MacFan4000: oh really? Are there any plans to do that soon?
2021-05-24 18:47:43 <dmehus> Reception123, yep heh
2021-05-24 18:47:57 <Reception123> dmehus: we will be updating to the new debian version soon though
2021-05-24 18:48:04 <RhinosF1> Reception123: they will be
2021-05-24 18:48:04 <dmehus> MacFan4000, ah
2021-05-24 18:48:06 <Reception123> But definitely not before 1.36 is completely done
2021-05-24 18:48:10 <dmehus> Reception123, yeah
2021-05-24 18:48:13 <paladox> https://github.com/Icinga/icingaweb2/commit/4bc5350ebaaeb7a60bbe20829a1d72def343a921
2021-05-24 18:48:13 <RhinosF1> Reception123: that'll be just 7. Whatever
2021-05-24 18:48:13 <MirahezeBot> [ Merge pull request #4289 from Icinga/bugfix/php8-4287 · Icinga/icingaweb2@4bc5350 · GitHub ] - github.com
2021-05-24 18:48:24 <Zppix> paladox: why not?
2021-05-24 18:48:29 <paladox> icingaweb2 supports php8 now but there's no actual released version
2021-05-24 18:48:32 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.40, 6.48, 7.80
2021-05-24 18:48:47 <RhinosF1> Zppix: because it's a very high barrier
2021-05-24 18:48:54 <paladox> Zppix: i'm not too active MW developing side.
2021-05-24 18:49:06 <MacFan4000> icingaweb2 support will be in their next release, and phabricator there's https://secure.phabricator.com/T13588 but upstream are in no hurry
2021-05-24 18:49:06 <MirahezeBot> [ ⚓ T13588 PHP 8 Compatibility ] - secure.phabricator.com
2021-05-24 18:52:35 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.19, 7.33, 7.08
2021-05-24 18:52:54 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 18:53:17 <dmehus> MacFan4000, ah
2021-05-24 18:54:35 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.38, 7.06, 7.02
2021-05-24 18:56:35 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.19, 6.15, 6.70
2021-05-24 18:58:02 <icinga-miraheze> PROBLEM - wiki.umbrel.tech - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.umbrel.tech' expires in 7 day(s) (Tue 01 Jun 2021 18:51:21 GMT +0000).
2021-05-24 18:59:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.84, 6.68, 7.88
2021-05-24 19:00:25 <Zppix> Shouldnt ssl be renewed at 15 days or something how’d it get to 7
2021-05-24 19:02:08 <RhinosF1> That's why it's critical
2021-05-24 19:02:31 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.46, 5.90, 6.79
2021-05-24 19:03:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.18, 6.87, 7.65
2021-05-24 19:05:01 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.16, 6.68, 6.50
2021-05-24 19:05:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.82, 6.79, 7.52
2021-05-24 19:08:16 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 19:08:56 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.50, 6.26, 6.40
2021-05-24 19:11:37 <Reception123> paladox: can you see log spam here: https://graylog.miraheze.org/search?q=%22test3wiki%22&rangetype=relative&relative=900 ?
2021-05-24 19:11:44 <Reception123> Crepusculum says he can't but I see RemovePII spam
2021-05-24 19:11:54 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.41, 5.47, 6.71
2021-05-24 19:12:09 <paladox> yeh i can, he won't because that's an "All stream"
2021-05-24 19:12:19 <paladox> mw team only have limited access to specific streams
2021-05-24 19:12:27 <Not-4c5b> ['mw-config'] 'antonmiraheze' opened pull request '#3918': Patch 1 - 'https://git.io/Jsxzm'
2021-05-24 19:12:29 <Reception123> ahh I know why he can't, because it's syslog yeah
2021-05-24 19:13:10 <Not-4c5b> ['mw-config'] 'antonmiraheze' edited pull request '#3918': Responsive Vector variables (1.36) - 'https://git.io/Jsxzm'
2021-05-24 19:15:57 <dmehus> paladox, I just realized we haven't setup miraheze-github alerts yet
2021-05-24 19:16:16 <paladox> alerts?
2021-05-24 19:16:23 <dmehus> well like the notifications
2021-05-24 19:16:45 <dmehus> on Freenode, we had Not-4c5b and miraheze-github that updated us if the build was passing or failing
2021-05-24 19:17:28 <dmehus> Crepusculum: maybe you can help set that up again? :)
2021-05-24 19:21:23 <RhinosF1> We have
2021-05-24 19:21:43 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 19:21:57 <RhinosF1> They don't run on forks for first time users immediately
2021-05-24 19:22:07 <RhinosF1> And it's his first config pr
2021-05-24 19:22:13 <icinga-miraheze> PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.75, 1.55, 1.23
2021-05-24 19:22:28 <miraheze-github> miraheze/mw-config - RhinosF1 the build has errored.
2021-05-24 19:22:50 <RhinosF1> dmehus: there ^
2021-05-24 19:23:04 <RhinosF1> Not sure why says me not anton
2021-05-24 19:23:10 <RhinosF1> But I approved him
2021-05-24 19:24:13 <icinga-miraheze> RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.27, 1.47, 1.24
2021-05-24 19:25:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.26, 6.78, 6.57
2021-05-24 19:26:52 <Crepusculum> dmehus: it is setup.
2021-05-24 19:29:52 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.13, 6.65, 6.61
2021-05-24 19:30:22 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 19:33:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.60, 7.43, 6.91
2021-05-24 19:35:09 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.62, 7.48, 6.56
2021-05-24 19:35:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.58, 8.04, 6.82
2021-05-24 19:37:08 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.01, 6.82, 6.42
2021-05-24 19:37:25 <dmehus> Crepusculum, RhinosF1, paladox, oh right, puppet repo just doesn't have miraheze-github CI notifications
2021-05-24 19:37:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.12, 7.27, 6.69
2021-05-24 19:39:53 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.37, 7.61, 6.87
2021-05-24 19:41:05 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.07, 7.24, 6.64
2021-05-24 19:41:19 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 7.22, 6.52
2021-05-24 19:41:53 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.89, 6.93, 6.71
2021-05-24 19:41:55 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.04, 7.69, 7.37
2021-05-24 19:43:04 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.28, 7.08, 6.64
2021-05-24 19:43:14 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.94, 6.69, 6.41
2021-05-24 19:43:47 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 19:43:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.18, 6.78, 6.69
2021-05-24 19:43:55 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.04, 8.16, 7.55
2021-05-24 19:45:04 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.55, 6.10, 6.34
2021-05-24 19:45:56 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.19, 7.43, 7.37
2021-05-24 19:49:55 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.71, 8.33, 7.66
2021-05-24 19:51:52 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' pushed '' commit to 'Universal-Omega-patch-1' [+0/-0/±2] 'https://git.io/Jsxro'
2021-05-24 19:51:54 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' 'ad752cb' - Switch to fork for AutoCreatePage on REL1_36 branch
2021-05-24 19:51:55 <Not-4c5b> ['mediawiki'] 'Universal-Omega' created branch 'Universal-Omega-patch-1' - 'https://git.io/vbL5b'
2021-05-24 19:52:32 <Not-4c5b> ['mediawiki'] 'Universal-Omega' opened pull request '#1961': Switch to fork for AutoCreatePage on REL1_36 branch - 'https://git.io/Jsxry'
2021-05-24 19:54:14 <Not-4c5b> ['mediawiki'] 'Universal-Omega' closed pull request '#1961': Switch to fork for AutoCreatePage on REL1_36 branch - 'https://git.io/Jsxry'
2021-05-24 19:54:16 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' pushed '' commit to 'REL1_36' [+0/-0/±2] 'https://git.io/Jsxrj'
2021-05-24 19:54:17 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' 'abc8804' - Switch to fork for AutoCreatePage on REL1_36 branch (#1961)
2021-05-24 19:54:19 <Not-4c5b> ['mediawiki'] 'Universal-Omega' deleted branch 'Universal-Omega-patch-1' - 'https://git.io/vbL5b'
2021-05-24 19:54:20 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' deleted branch 'Universal-Omega-patch-1'
2021-05-24 19:56:44 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.35, 6.95, 6.62
2021-05-24 19:57:02 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.89, 7.39, 7.09
2021-05-24 19:57:40 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 7.25, 7.03
2021-05-24 19:58:39 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.14, 6.33, 6.44
2021-05-24 20:01:02 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.00, 6.09, 6.61
2021-05-24 20:01:36 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.39, 7.55, 7.19
2021-05-24 20:01:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.45, 7.93, 7.95
2021-05-24 20:02:13 <icinga-miraheze> PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 4.13, 2.26, 1.43
2021-05-24 20:03:41 <Not-4c5b> ['miraheze/mediawiki'] 'paladox' pushed '' commit to 'REL1_36' [+0/-0/±1] 'https://git.io/JsxKE'
2021-05-24 20:03:42 <Not-4c5b> ['miraheze/mediawiki'] 'paladox' 'cd9ea19' - Update EventLogging
2021-05-24 20:03:53 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.96, 8.18, 8.03
2021-05-24 20:04:44 <icinga-miraheze> PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.532 second response time
2021-05-24 20:04:51 <paladox> RhinosF1: we can move forward, got merged ^
2021-05-24 20:04:58 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 7.35, 7.00
2021-05-24 20:06:13 <icinga-miraheze> RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.81, 1.63, 1.38
2021-05-24 20:06:25 <paladox> 2021-05-24 20:05:49 test3 test3wiki: [c6ea263f98da317d712089aa] /wiki/Main_Page Error: Call to undefined method MagicWord::get()
2021-05-24 20:06:46 <icinga-miraheze> RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20074 bytes in 1.538 second response time
2021-05-24 20:06:49 <paladox> ah https://github.com/staspika/mediawiki-numberedheadings/pull/3
2021-05-24 20:06:50 <MirahezeBot> [ MagicWord -> MagicWordFactory by paladox · Pull Request #3 · staspika/mediawiki-numberedheadings · GitHub ] - github.com
2021-05-24 20:06:57 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.42, 7.31, 7.03
2021-05-24 20:07:10 <icinga-miraheze> PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 20:07:30 <Not-4c5b> ['miraheze/mediawiki'] 'paladox' pushed '' commit to 'REL1_36' [+0/-0/±1] 'https://git.io/JsxKN'
2021-05-24 20:07:32 <Not-4c5b> ['miraheze/mediawiki'] 'paladox' '9ca7e9' - Update NumberedHeadings to ada69e7fd80bc01bcfd518588559a4ff23f51834
2021-05-24 20:07:46 <Reception123> paladox: thanks for merging, and it seems like graylog still isn't working for test3
2021-05-24 20:08:25 <paladox> 2021-05-24 20:08:04 test3 test3wiki: [0ae4325a7dbfb7961caf6341] /wiki/Main_Page Error: Call to undefined function MediaWiki\Extension\Interwiki\Hooks()
2021-05-24 20:08:32 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 7.51, 6.91
2021-05-24 20:10:05 <Reception123> urgh, an upstream error with the Interwiki extension?
2021-05-24 20:10:54 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.21, 6.32, 6.70
2021-05-24 20:11:29 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.60, 7.40, 7.60
2021-05-24 20:11:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.16, 7.26, 7.83
2021-05-24 20:13:06 <RhinosF1> Reception123: fun
2021-05-24 20:13:33 <paladox> https://github.com/wikimedia/mediawiki-extensions-Interwiki/commit/b21bafe3858bc97da050280aa5d26ff826ac5611
2021-05-24 20:13:33 <MirahezeBot> [ Don't register InterwikiLoadPrefix hook unless it is needed · wikimedia/mediawiki-extensions-Interwiki@b21bafe · GitHub ] - github.com
2021-05-24 20:13:38 <paladox> could it be this RhinosF1 ?
2021-05-24 20:14:17 <RhinosF1> What's using it paladox
2021-05-24 20:14:39 <paladox> https://www.irccloud.com/pastebin/i86MQyjL/
2021-05-24 20:14:39 <MirahezeBot> [ Snippet | IRCCloud ] - www.irccloud.com
2021-05-24 20:15:17 <RhinosF1> Yes but that commit says should be nothing
2021-05-24 20:15:20 <paladox> removing that at least got the site working
2021-05-24 20:15:46 <RhinosF1> Can you upstream fix?
2021-05-24 20:16:21 <RhinosF1> I still see Original exception: [9486f8c3beab96a4ed600df8] 2021-05-24 20:15:52: Fatal exception of type "Error"
2021-05-24 20:16:48 <paladox> i don't know what the fix is
2021-05-24 20:22:12 <dmehus> Oh no, that's our MiraehzeMagic pseudo-interwiki prefix, isn't it?
2021-05-24 20:22:31 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.34, 6.21, 6.64
2021-05-24 20:23:16 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.17, 6.02, 6.73
2021-05-24 20:29:56 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.74, 6.02, 6.63
2021-05-24 20:33:52 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.25, 7.15, 6.66
2021-05-24 20:35:18 <icinga-miraheze> RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 20:35:53 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 7.25, 6.77
2021-05-24 20:35:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.01, 6.88, 6.76
2021-05-24 20:37:53 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.30, 6.26, 6.47
2021-05-24 20:37:54 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.64, 6.83, 6.76
2021-05-24 20:38:55 <RhinosF1> dmehus: not sure what yet but doubt it
2021-05-24 20:39:31 <RhinosF1> paladox: full trace?
2021-05-24 20:40:08 <paladox> https://phabricator.wikimedia.org/P16180
2021-05-24 20:40:09 <MirahezeBot> [ ✎ P16180 (An Untitled Masterwork) ] - phabricator.wikimedia.org
2021-05-24 20:41:57 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.04, 6.76, 6.79
2021-05-24 20:44:33 <dmehus> RhinosF1, ah
2021-05-24 20:48:02 <RhinosF1> dmehus: looks to be a core issue
2021-05-24 20:52:08 <RhinosF1> dmehus: yeah it's utterly upstream
2021-05-24 20:54:29 <RhinosF1> !log test3: 4 hours later, we're catching many bugs.
2021-05-24 20:54:35 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:04:23 <RhinosF1> paladox: will schema changes via sql.php work and can I do them?
2021-05-24 21:04:39 <paladox> i think they will
2021-05-24 21:05:54 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.22, 7.77, 6.65
2021-05-24 21:06:18 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 21:07:53 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 6.98, 6.50
2021-05-24 21:08:09 <RhinosF1> paladox: please shout if db gets unhappy
2021-05-24 21:08:16 <paladox> ok
2021-05-24 21:09:54 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.83, 6.16, 6.25
2021-05-24 21:10:11 <RhinosF1> https://www.irccloud.com/pastebin/UzDC2IWH/
2021-05-24 21:10:12 <MirahezeBot> [ Snippet | IRCCloud ] - www.irccloud.com
2021-05-24 21:10:21 <RhinosF1> various warnings
2021-05-24 21:11:30 <Crepusculum> RhinosF1: I already fixed CentralNotice DB errors, I made a local change to test3 to change the central database to test3wiki rather then metawiki also.
2021-05-24 21:12:02 <RhinosF1> Crepusculum: I have 2 sql patches to run
2021-05-24 21:12:18 <RhinosF1> !log test3wiki: run /srv/mediawiki/w/extensions/OAuth/schema/oauth2_allowed_grants.sql
2021-05-24 21:12:25 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:14:12 <RhinosF1> !log test3wiki: run /srv/mediawiki/w/extensions/OAuth/schema/oauth2_is_confidential.sql
2021-05-24 21:14:16 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:15:06 <RhinosF1> !log test3wiki: run /srv/mediawiki/w/extensions/OAuth/schema/oauth_version_accepted.sql
2021-05-24 21:15:10 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:15:56 <RhinosF1> Crepusculum: find out what DB 2FA uses please?
2021-05-24 21:16:18 <paladox> Crepusculum: i don't think null is an accepted value hence those errors
2021-05-24 21:17:42 <RhinosF1> !log test3wiki: run /srv/mediawiki/w/extensions/CentralNotice/patches/patch-notice-type-log.sql
2021-05-24 21:17:45 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:17:52 <Crepusculum> RhinosF1: already did CN.
2021-05-24 21:18:18 <Crepusculum> To fix all resource module stuff not working.
2021-05-24 21:18:43 <Crepusculum> paladox: it has before I think, but maybe not. It is the extension default though.
2021-05-24 21:19:01 <RhinosF1> Crepusculum: please log stuff
2021-05-24 21:19:36 <Crepusculum> RhinosF1: why is logging necessary for test3? I didn't think it was. But will do.
2021-05-24 21:19:47 <RhinosF1> Crepusculum: for schema changes yes
2021-05-24 21:19:53 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.45, 7.06, 6.47
2021-05-24 21:19:56 <paladox> i think that we should use the wmg configs then convert it from false to null
2021-05-24 21:20:00 <paladox> outside of MWS
2021-05-24 21:21:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.73, 7.05, 6.56
2021-05-24 21:23:17 <RhinosF1> paladox: can you take a dump of mhglobal.oathauth_users ?
2021-05-24 21:23:31 <paladox> Uh, do not touch that table
2021-05-24 21:23:34 <paladox> it's production
2021-05-24 21:23:46 <RhinosF1> paladox: Reedy said it would work
2021-05-24 21:23:51 <paladox> oh
2021-05-24 21:23:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.00, 6.36, 6.37
2021-05-24 21:24:03 <RhinosF1> 22:15:10 <Reedy> RhinosF1: Should be... I'm pretty sure the code is in REL1_35 that would do that organically (ie when users enabled 2FA or caused another update)
2021-05-24 21:24:17 <RhinosF1> but yes that's why I want a dump
2021-05-24 21:24:53 <Crepusculum> !log [universalomega@test3] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/extensions/CentralNotice/patches/patch-notice-type-log.sql
2021-05-24 21:24:53 <Crepusculum> !log [universalomega@test3] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/extensions/CentralNotice/patches/patch-notice_not_type.sql
2021-05-24 21:24:53 <Crepusculum> !log change wgCentralDBname on test3 to test3wiki as to not touch metawiki database while testing schema changes for 1.36
2021-05-24 21:24:56 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:24:57 <paladox> RhinosF1: dumped now
2021-05-24 21:24:59 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:25:02 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:25:46 <RhinosF1> !log (affects all wikis) rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/extensions/OATHAuth/maintenance/updateTOTPScratchTokensToArray.php --wiki=test3wiki
2021-05-24 21:25:49 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:26:20 <RhinosF1> Crepusculum: ext wise just wb_terms left
2021-05-24 21:26:33 <Crepusculum> RhinosF1: Great!
2021-05-24 21:26:46 <paladox> wonder why my theme doesn't work on test3
2021-05-24 21:26:53 <paladox> as in i picked timeless but it's using vector?
2021-05-24 21:27:02 <Crepusculum> paladox: yeah same, my skin preference is not working
2021-05-24 21:27:52 <Crepusculum> only works with `?useskin` the preference is being ignored.
2021-05-24 21:28:17 <Crepusculum> RhinosF1: do we have a backup of test3 DB?
2021-05-24 21:28:33 <RhinosF1> Crepusculum: yes
2021-05-24 21:28:47 <RhinosF1> It's blue for me
2021-05-24 21:29:07 <Crepusculum> RhinosF1: same, why is vector blue is another question.
2021-05-24 21:29:24 <paladox> vector blue?
2021-05-24 21:29:25 <Crepusculum> and great will do the DROP then as we need to make sure it works.
2021-05-24 21:29:34 <Crepusculum> paladox: yeah, vector is blue for me.
2021-05-24 21:29:36 <RhinosF1> Crepusculum: ok, core schemas coming up
2021-05-24 21:29:48 <paladox> it's not blue for me
2021-05-24 21:29:58 <paladox> take a pic for me
2021-05-24 21:30:10 <paladox> oh
2021-05-24 21:30:12 <paladox> logged out
2021-05-24 21:30:17 <paladox> logged in, it's grey
2021-05-24 21:30:17 <paladox> wtf
2021-05-24 21:30:41 <Crepusculum> paladox: I am logged in and it is blue.
2021-05-24 21:30:52 <paladox> it's not blue for me
2021-05-24 21:31:04 <paladox> hold on... could it be miss identifying users or something
2021-05-24 21:31:06 <paladox> this is weird
2021-05-24 21:31:23 <Crepusculum> paladox: that even happens every once in awhile on 1.35, as in sometimes on Special:Preferences it was, then reloading and it is fixed.
2021-05-24 21:31:32 <Crepusculum> but now it is constent
2021-05-24 21:31:57 <paladox> hmm
2021-05-24 21:35:13 <Crepusculum> !log [universalomega@test3] DROP TABLE wb_terms;
2021-05-24 21:35:16 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:35:31 <paladox> where is it even loading blue vector
2021-05-24 21:35:33 <paladox> which skin/
2021-05-24 21:35:43 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.64, 7.89, 6.90
2021-05-24 21:35:52 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.13, 6.84, 6.36
2021-05-24 21:36:06 <Crepusculum> paladox: any ideas about these? https://www.irccloud.com/pastebin/cgMZNTe4/
2021-05-24 21:36:07 <MirahezeBot> [ Snippet | IRCCloud ] - www.irccloud.com
2021-05-24 21:36:14 <paladox> no
2021-05-24 21:36:27 <Crepusculum> ok, thanks anyways then.
2021-05-24 21:36:40 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.66, 7.20, 6.61
2021-05-24 21:36:50 <paladox> i cannot find the blue vector theme wtf
2021-05-24 21:36:55 <paladox> where is it getting it from
2021-05-24 21:37:05 <Crepusculum> also, good question, no idea, though I do wonder if it is possible the fallback vector maybe? Only thing I can think of.
2021-05-24 21:37:39 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.55, 7.35, 6.82
2021-05-24 21:37:47 <RhinosF1> !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-redirect-rd_title-varbinary.sql
2021-05-24 21:37:51 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:37:52 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.93, 6.77, 6.41
2021-05-24 21:37:54 <MacFan4000> it isn't blue for me either
2021-05-24 21:38:14 <RhinosF1> !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-pagelinks-pl_title-varbinary.sql
2021-05-24 21:38:17 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:38:38 <RhinosF1> !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-templatelinks-tl_title-varbinary.sql
2021-05-24 21:38:42 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:39:04 <RhinosF1> !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-imagelinks-il_to-varbinary.sql
2021-05-24 21:39:06 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:39:29 <RhinosF1> !log rhinos@test3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-langlinks-ll_title-varbinary.sql
2021-05-24 21:39:32 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:39:35 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.56, 6.75, 6.66
2021-05-24 21:40:59 <Crepusculum> MacFan4000: how it looks for me. https://usercontent.irccloud-cdn.com/file/93uPVh8L/image.png
2021-05-24 21:41:14 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-iwlinks-iwl_title-varbinary.sql
2021-05-24 21:41:15 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-category-cat_title-varbinary.sql
2021-05-24 21:41:15 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-querycache-qc_title-varbinary.sql
2021-05-24 21:41:15 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-querycachetwo-qcc_title-varbinary.sql
2021-05-24 21:41:15 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-watchlist-wl_title-varbinary.sql
2021-05-24 21:41:15 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-user_newtalk-user_last_timestamp-binary.sql
2021-05-24 21:41:15 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-protected_titles-pt_title-varbinary.sql
2021-05-24 21:41:17 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:41:21 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:41:33 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:41:39 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:41:44 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:41:48 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:41:52 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:43:02 <RhinosF1> !log all previous on test3
2021-05-24 21:43:05 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:43:37 <paladox> found it
2021-05-24 21:43:38 <paladox> https://github.com/wikimedia/mediawiki-extensions-Theme
2021-05-24 21:43:39 <MirahezeBot> [ GitHub - wikimedia/mediawiki-extensions-Theme: Github mirror of MediaWiki extension Theme - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contri ] - github.com
2021-05-24 21:43:42 <paladox> RhinosF1: Crepusculum
2021-05-24 21:44:00 <MacFan4000> Crepusculum: Go into your settings under appearance and check the theme setting
2021-05-24 21:44:16 <MacFan4000> it's likely set to dark
2021-05-24 21:44:38 <RhinosF1> MacFan4000: night time?
2021-05-24 21:44:46 <Crepusculum> MacFan4000: "The skin you are currently using does not support themes. Please use another skin if you wish to use a non-default theme"
2021-05-24 21:45:03 <paladox> uninstalling it doesn't fix the theme you selected from showing
2021-05-24 21:45:20 <MacFan4000> for me setting it to dark makes it look like it does in the screenshot
2021-05-24 21:45:26 <MacFan4000> my skin is set to vector
2021-05-24 21:45:54 <Crepusculum> MacFan4000: oh, my skin is not even set to vector, but vector is the skin it is using.
2021-05-24 21:46:21 <MacFan4000> what skin are you set to?
2021-05-24 21:46:41 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-ipblocks_restrictions-ir_type.sql
2021-05-24 21:46:41 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-watchlist-namespace_title-rename-index.sql
2021-05-24 21:46:41 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-job-job_title-varbinary.sql
2021-05-24 21:46:41 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-job_job_timestamp.sql
2021-05-24 21:46:41 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-job_job_token_timestamp.sql
2021-05-24 21:46:42 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-watchlist-wl_notificationtimestamp.sql
2021-05-24 21:46:42 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-slot_roles-role_id.sql
2021-05-24 21:46:43 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-content_models-model_id.sql
2021-05-24 21:46:43 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-categorylinks-cl_to-varbinary.sql
2021-05-24 21:46:44 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:46:44 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-logging-log_title-varbinary.sql
2021-05-24 21:46:44 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-uploadstash-us_timestamp.sql
2021-05-24 21:46:45 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-user_properties-rename-index.sql
2021-05-24 21:46:45 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-sites-rename-indexes.sql
2021-05-24 21:46:46 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-logging-rename-indexes.sql
2021-05-24 21:46:46 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-filearchive-fa_name.sql
2021-05-24 21:46:46 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:46:49 <Not-4c5b> ['miraheze/mediawiki'] 'paladox' pushed '' commit to 'REL1_36' [+0/-0/±1] 'https://git.io/JsxbJ'
2021-05-24 21:46:51 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:46:51 <Not-4c5b> ['miraheze/mediawiki'] 'paladox' '8cd09f' - Update Interwiki
2021-05-24 21:46:54 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:46:56 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:46:59 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:02 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:05 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:08 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:10 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:13 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:17 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:20 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:23 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:26 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:47:27 <Crepusculum> MacFan4000: Cosmos, but just tried changing to apex, did not work, also tried Timeless as well, and MonoBook, nothing works, and always shows vector.
2021-05-24 21:48:32 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.20, 6.56, 6.64
2021-05-24 21:48:35 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 21:49:05 <Crepusculum> paladox: does that fix interwiki issue then?
2021-05-24 21:49:11 <paladox> yeh
2021-05-24 21:49:15 <Crepusculum> Great!
2021-05-24 21:49:17 <MacFan4000> if I change my skin in settings I always see vector, but if I use the preview link (?useskin in the url) then it works fine
2021-05-24 21:49:18 <Crepusculum> thanks!
2021-05-24 21:49:27 <Crepusculum> MacFan4000: same.
2021-05-24 21:49:36 <paladox> Crepusculum: do you have 1.35 checked out locally?
2021-05-24 21:49:44 <paladox> if so could you deploy https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseQualityConstraints/+/693655 pls
2021-05-24 21:49:53 <RhinosF1> paladox: https://phabricator.miraheze.org/T7357
2021-05-24 21:49:54 <MirahezeBot> [ ⚓ T7357 Apply test3 drop default schema changes ] - phabricator.miraheze.org
2021-05-24 21:50:10 <RhinosF1> I don't have files for hem
2021-05-24 21:50:12 <Crepusculum> paladox: no (just 1.36 right now), but it will take me like 5 minutes to do, so will do.
2021-05-24 21:50:12 <RhinosF1> them
2021-05-24 21:50:18 <paladox> RhinosF1: you can easily drop them by creating the files
2021-05-24 21:50:26 <MacFan4000> you can at least make it normal vector by setting your skin to vector and then theme to default
2021-05-24 21:50:30 <RhinosF1> paladox: I'm pants at sql
2021-05-24 21:50:33 <paladox> oh ok
2021-05-24 21:51:04 <paladox> currently busy looking at this theme issue.
2021-05-24 21:51:35 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 6.90, 6.45
2021-05-24 21:53:36 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.42, 6.60, 6.39
2021-05-24 21:54:21 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-oldimage-oi_name-varbinary.sql
2021-05-24 21:54:21 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-objectcache-exptime-notnull.sql
2021-05-24 21:54:21 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-archive-rename-name_title_timestamp-index.sql
2021-05-24 21:54:21 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-image-img_name-varbinary.sql
2021-05-24 21:54:21 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-image-img_timestamp.sql
2021-05-24 21:54:21 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-site_identifiers-rename-indexes.sql
2021-05-24 21:54:21 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-recentchanges-rc_title-varbinary.sql
2021-05-24 21:54:22 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-recentchanges-rc_timestamp.sql
2021-05-24 21:54:22 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-recentchanges-rc_id.sql
2021-05-24 21:54:23 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-recentchanges-rc_new_name_timestamp.sql
2021-05-24 21:54:23 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-archive-ar_title-varbinary.sql
2021-05-24 21:54:24 <RhinosF1> !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=test3wiki /srv/mediawiki/w/maintenance/archives/patch-user_table-updates.sql
2021-05-24 21:54:34 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:54:35 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' pushed '' commit to 'REL1_35' [+0/-0/±1] 'https://git.io/Jsxbx'
2021-05-24 21:54:36 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' 'c2a07b6' - Update WikibaseQualityConstraints
2021-05-24 21:54:39 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:54:49 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:54:51 <RhinosF1> !log test3 schema changes with sql files done! \o/ See https://phabricator.miraheze.org/T7357 for remaining ones
2021-05-24 21:54:52 <MirahezeBot> [ ⚓ T7357 Apply test3 drop default schema changes ] - phabricator.miraheze.org
2021-05-24 21:54:54 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:54:57 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:02 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:05 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:08 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:12 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:15 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:18 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:22 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:27 <MirahezeLogbot> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
2021-05-24 21:55:38 <Crepusculum> paladox: updated
2021-05-24 21:56:03 <paladox> thanks!
2021-05-24 21:56:07 <Crepusculum> No problem.
2021-05-24 21:56:31 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.08, 6.92, 6.77
2021-05-24 21:58:08 <RhinosF1> Crepusculum: 62e7657b3817c83b58620968
2021-05-24 21:58:16 <Crepusculum> looking
2021-05-24 21:58:31 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.53, 6.47, 6.62
2021-05-24 21:58:53 <paladox> checking out 1.35 branch for GP didn't fix the issue
2021-05-24 21:59:40 <Crepusculum> RhinosF1 https://www.irccloud.com/pastebin/JZrerdvt/
2021-05-24 21:59:41 <MirahezeBot> [ Snippet | IRCCloud ] - www.irccloud.com
2021-05-24 22:00:12 <paladox> 2021-05-24 21:59:45 test3 test3wiki: [ebbe6c285df4c57e91096fcc] /w/load.php?lang=en-gb&modules=startup&only=scripts&raw=1&skin=timeless ErrorException: Use of ParserSectionCreate hook (used in HideSectionHooks::onParserSectionCreate) was deprecated in MediaWiki 1.35. [Called from MediaWiki\HookContainer\HookContainer::run]
2021-05-24 22:00:59 <RhinosF1> paladox: upstream?
2021-05-24 22:01:10 <RhinosF1> Crepusculum: uh oh
2021-05-24 22:01:11 <paladox> yeh, that can be filed upstream
2021-05-24 22:01:17 <paladox> Crepusculum: that was me
2021-05-24 22:01:23 <paladox> i was testing some changes with GP
2021-05-24 22:01:25 <RhinosF1> oh
2021-05-24 22:01:41 <paladox> 2021-05-24 21:59:45 test3 test3wiki: [ebbe6c285df4c57e91096fcc] /w/load.php?lang=en-gb&modules=startup&only=scripts&raw=1&skin=timeless ErrorException: Use of InternalParseBeforeSanitize hook (used in VariablesHooks::onInternalParseBeforeSanitize) was deprecated in MediaWiki 1.35. [Called from MediaWiki\HookContainer\HookContainer::run]
2021-05-24 22:01:53 <paladox> 2021-05-24 21:59:45 test3 test3wiki: [a10b0b26a0744ff61516467d] /w/index.php?title=Main_Page&useskin=timeless ErrorException: Use of SkinTemplateOutputPageBeforeExec hook (used in HideSectionHooks::onSkinTemplateOutputPageBeforeExec) was deprecated in MediaWiki 1.35. [Called from MediaWiki\HookContainer\HookContainer::run]
2021-05-24 22:03:05 <Crepusculum> paladox: SkinTemplateOutputPageBeforeExec hook is going to be hard to fix on some extensions, such as LanguageSelector. I tried awhile ago when converting that to extensionregistry but could not find a replacement hook to do what it needs to do.
2021-05-24 22:03:13 <paladox> those warnings are not affecting the skins
2021-05-24 22:03:20 <paladox> so i'm not sure why this is happening
2021-05-24 22:03:24 <paladox> oh
2021-05-24 22:03:45 <paladox> 2021-05-24 21:59:42 test3 test3wiki: [deeff36ed2036061727d9758] /w/api.php?action=themelist&format=json ErrorException: Use of Skin::getSkinNames was deprecated in MediaWiki 1.36. [Called from ApiThemeList::appendThemes]
2021-05-24 22:03:59 <RhinosF1> paladox: none of the timeless look new
2021-05-24 22:04:54 <paladox> 2021-05-24 21:59:36 test3 test3wiki: [5adaef72c8648e1f7f2d19e4] /wiki/Special:Preferences ErrorException: PHP Deprecated: A ParserGetVariableValueSwitch hook handler bypassed the cache, this is deprecated since MediaWiki 1.35
2021-05-24 22:05:27 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.64, 7.06, 6.49
2021-05-24 22:05:28 <Crepusculum> It does not say which extension paladox, or where from?
2021-05-24 22:05:42 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.43, 7.18, 6.54
2021-05-24 22:05:50 <RhinosF1> paladox: most of them say 1.35
2021-05-24 22:06:11 <paladox> https://phabricator.miraheze.org/P435
2021-05-24 22:06:12 <MirahezeBot> [ ✎ P435 (An Untitled Masterwork) ] - phabricator.miraheze.org
2021-05-24 22:06:29 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.61, 7.16, 6.79
2021-05-24 22:07:24 <paladox> yeh guess they changed the exception or something? Unless it's because we have the log level differently to what we have it now to log local
2021-05-24 22:07:35 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' pushed '' commit to 'REL1_35' [+0/-0/±1] 'https://git.io/JsxAc'
2021-05-24 22:07:36 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' 'b3df73' - Update FancyBoxThumbs
2021-05-24 22:07:37 <paladox> (using debug here for the local logging as that's the default)
2021-05-24 22:07:37 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.65, 7.31, 6.67
2021-05-24 22:07:54 <RhinosF1> paladox: since 1.35 ain't an issue
2021-05-24 22:07:57 <RhinosF1> I filed https://phabricator.wikimedia.org/T283547
2021-05-24 22:07:58 <MirahezeBot> [ ⚓ T283547 ErrorException: Use of Skin::getSkinNames was deprecated in MediaWiki 1.36. [Called from ApiThemeList::appendThemes] ] - phabricator.wikimedia.org
2021-05-24 22:08:01 <paladox> ok
2021-05-24 22:08:02 <RhinosF1> for the 1.36 one
2021-05-24 22:08:44 <Crepusculum> RhinosF1: they are an issue on 1.36 though, they were soft deprecated on 1.35 and hard on 1.36, meaning constant error logs on 1.36
2021-05-24 22:08:51 <RhinosF1> paladox: my main concern is the theme issue though
2021-05-24 22:08:54 <RhinosF1> Crepusculum: oh
2021-05-24 22:08:57 <paladox> yeh
2021-05-24 22:09:00 <RhinosF1> file tasks then
2021-05-24 22:09:02 <paladox> that's what i'm trying to find out
2021-05-24 22:09:14 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' pushed '' commit to 'REL1_36' [+0/-0/±1] 'https://git.io/JsxA6'
2021-05-24 22:09:16 <Not-4c5b> ['miraheze/mediawiki'] 'Universal-Omega' '06617d' - Update FancyBoxThumbs
2021-05-24 22:09:22 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.47, 7.44, 6.79
2021-05-24 22:09:33 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.12, 7.62, 6.85
2021-05-24 22:09:36 <RhinosF1> Crepusculum: Can you go on a task filing spree
2021-05-24 22:09:53 <RhinosF1> add them to the list on https://phabricator.miraheze.org/T7117
2021-05-24 22:09:55 <MirahezeBot> [ ⚓ T7117 Upgrade to 1.36 ] - phabricator.miraheze.org
2021-05-24 22:10:12 <Crepusculum> RhinosF1: I will just submit patches upstream when possible. But for ones I can't I will try to file tasks yeah.
2021-05-24 22:10:25 <RhinosF1> Crepusculum: as long as each blocker is noted
2021-05-24 22:10:33 <icinga-miraheze> PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 22:10:44 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.36, 6.64
2021-05-24 22:11:28 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 7.69, 6.97
2021-05-24 22:11:44 <Crepusculum> RhinosF1: not all extension deprecation notices will be blockers, as long as we cut down on some, we still have some warning from 1.34 deprecations also (maybe even some from before that)
2021-05-24 22:12:00 <RhinosF1> Crepusculum: https://phabricator.miraheze.org/T7117
2021-05-24 22:12:01 <MirahezeBot> [ ⚓ T7117 Upgrade to 1.36 ] - phabricator.miraheze.org
2021-05-24 22:12:04 <Crepusculum> but I will note which ones warns yeah.
2021-05-24 22:12:07 <RhinosF1> currently noted them all
2021-05-24 22:12:17 <RhinosF1> but we can evaluate as time goes on
2021-05-24 22:12:21 <icinga-miraheze> PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 22:12:30 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.09, 7.70, 7.20
2021-05-24 22:12:36 <RhinosF1> let's try and get tasks upstream at least or patches
2021-05-24 22:12:41 <icinga-miraheze> PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 22:12:43 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 6.72, 6.50
2021-05-24 22:12:48 <icinga-miraheze> PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 22:12:58 <Crepusculum> I fixed FancyBoxThumbs issue which caused JS notices on both 1.35 and 1.36 (and updated) by the way.
2021-05-24 22:13:01 <RhinosF1> Crepusculum: the skin being blue is priority though
2021-05-24 22:13:05 <icinga-miraheze> PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
2021-05-24 22:13:12 <Crepusculum> RhinosF1: I fixed my blue theme
2021-05-24 22:13:17 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.28, 6.58, 6.61
2021-05-24 22:13:40 <RhinosF1> Crepusculum: I see it logged out
2021-05-24 22:14:30 <paladox> https://www.mediawiki.org/wiki/Manual:Hooks/ParserSectionCreate
2021-05-24 22:14:30 <MirahezeBot> [ Manual:Hooks/ParserSectionCreate - MediaWiki ] - www.mediawiki.org
2021-05-24 22:14:34 <paladox> says no replacement hook?
2021-05-24 22:14:59 <Crepusculum> Fixed theme RhinosF1
2021-05-24 22:15:04 <Crepusculum> and paladox ^
2021-05-24 22:15:11 <paladox> oh?
2021-05-24 22:15:14 <RhinosF1> Crepusculum: how?
2021-05-24 22:15:14 <paladox> how did you fix it?
2021-05-24 22:15:21 <Crepusculum> it was just a setting in Special:ManageWiki/settings
2021-05-24 22:15:26 <paladox> oh
2021-05-24 22:16:51 <RhinosF1> Crepusculum: why did it only effect 1.36 though?
2021-05-24 22:16:59 <Crepusculum> It didn't.
2021-05-24 22:17:01 <MacFan4000> That doesn't solve the issue of not. being able to use a skin other then vector outside of previewing skins
2021-05-24 22:17:05 <RhinosF1> Can we ensure it won't hit prod
2021-05-24 22:17:06 <Crepusculum> paladox: the issue remains though, the skin preference is not being honored, always shows vector no matter what, that is for sure needing fixed.
2021-05-24 22:17:17 <paladox> ok, yeh
2021-05-24 22:17:24 <paladox> i'm not sure how we fix that
2021-05-24 22:17:32 <paladox> i've looked at a few things and cannot figure out why
2021-05-24 22:17:52 <Cocopuff2017> any updates on what caused last nights outage yet?
2021-05-24 22:17:55 <Crepusculum> RhinosF1: it was set to default to that on test3, I guess it did effect 1.35 also. Either that or theme was fixed to honor the setting.
2021-05-24 22:18:37 <RhinosF1> Crepusculum: it's always legacy vector too
2021-05-24 22:18:54 <RhinosF1> I can't even get new to work
2021-05-24 22:19:09 <Crepusculum> RhinosF1: in that case, I have an idea.
2021-05-24 22:19:23 <Zppix> <s> thats not a bug its a feature (re new vector </s>
2021-05-24 22:19:44 <RhinosF1> Zppix: getting rid of new vector would be good
2021-05-24 22:20:06 <RhinosF1> anyway sleep
2021-05-24 22:20:32 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.39, 6.41, 6.69
2021-05-24 22:20:57 <Cocopuff2017> but ya know Vector is a commonly used skin
2021-05-24 22:21:02 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.76, 6.61, 6.75
2021-05-24 22:21:40 <Crepusculum> Zppix: the issue is no skin preferences are working, always using vector, not even enabling the new vector works. Not that new vector is there.
2021-05-24 22:22:02 <Zppix> Crepusculum: that would be a core issue
2021-05-24 22:22:09 <MacFan4000> you can change the theme however
2021-05-24 22:22:15 <paladox> wonder if caused by https://github.com/wikimedia/mediawiki-extensions-GlobalPreferences/commit/dde39ef3d65a740f9c7d9ef90cdbaab021884e69#diff-b51349da1d5cc80bb73cbe51a79b96b2677397a1aa63854e8a5fa3fdf74fb224
2021-05-24 22:22:16 <MirahezeBot> [ Remove GlobalPreferencesFactory::setUser() · wikimedia/mediawiki-extensions-GlobalPreferences@dde39ef · GitHub ] - github.com
2021-05-24 22:23:23 <paladox> but then doesn't explain why it works on wikimedia?
2021-05-24 22:23:41 <Zppix> Could it be one of our config/code hacks
2021-05-24 22:23:51 <Crepusculum> paladox: I don't think it is GlobalPreferences extension is it? I set a local exception and still does not work.
2021-05-24 22:24:14 <paladox> local exception would still be the extension tho
2021-05-24 22:24:19 <Crepusculum> setting wgDefaultSkin works to change it also.
2021-05-24 22:24:37 <Zppix> Crepusculum: could try disable global prefs on test3 and see if it does it
2021-05-24 22:25:07 <Crepusculum> Zppix: ok, I guess we could try. paladox: sound ok to you?
2021-05-24 22:25:33 <paladox> Yeh
2021-05-24 22:25:37 <Cocopuff2017> @zppix are you rejoining system admin?
2021-05-24 22:25:48 <Zppix> No
2021-05-24 22:26:03 <Zppix> Just bored
2021-05-24 22:26:20 <Cocopuff2017> so you basically got bored of it?
2021-05-24 22:26:40 <paladox> doesn't work
2021-05-24 22:26:44 <Crepusculum> paladox: disabled on test3
2021-05-24 22:27:18 <Crepusculum> yeah same issue
2021-05-24 22:27:21 <Crepusculum> re-enabling
2021-05-24 22:27:47 <Zppix> Cocopuff2017: no im just bored atm
2021-05-24 22:28:00 <Cocopuff2017> Nice.
2021-05-24 22:28:06 <paladox> well i disabled the vector skin but somehow https://test3.miraheze.org/wiki/Main_Page is working???
2021-05-24 22:28:07 <MirahezeBot> [ Internal error - Test3 ] - test3.miraheze.org
2021-05-24 22:28:22 <Crepusculum> paladox: now it is fallback vector.'
2021-05-24 22:28:33 <paladox> oh
2021-05-24 22:29:11 <Crepusculum> paladox: I set wgFallbackSkin awhile ago to that do to the issue where if skin is disabled it causes issues. so it now falls back to vector.
2021-05-24 22:29:20 <Crepusculum> *due
2021-05-24 22:29:21 <paladox> oh
2021-05-24 22:29:22 <paladox> #2 /srv/mediawiki/w/skins/Tweeki/includes/Hooks.php(52): ContextSource->getSkin()
2021-05-24 22:29:43 <paladox> got it working
2021-05-24 22:29:45 <paladox> the cause is Tweeki
2021-05-24 22:29:55 <Crepusculum> oh.
2021-05-24 22:30:08 <Zppix> Tweeki needs a little tweeki
2021-05-24 22:30:19 <RhinosF1> Zppix: too many times
2021-05-24 22:30:29 <RhinosF1> It didn't work for a long while on mobile I think
2021-05-24 22:31:04 <Crepusculum> RhinosF1: yeah it didn't
2021-05-24 22:31:21 <RhinosF1> Crepusculum: candidate for bin?
2021-05-24 22:32:10 <Crepusculum> yeah.
2021-05-24 22:32:22 <RhinosF1> Just do it then
2021-05-24 22:32:51 <RhinosF1> Only on 1.36 branches though
2021-05-24 22:34:01 <paladox> is there any fix we can do to the skin
2021-05-24 22:34:45 <Crepusculum> paladox: I think it in particular just has had to many past issues, but it is popular so may be worth trying to fix upstream. No idea how though.
2021-05-24 22:34:50 <RhinosF1> It's broke a few times though
2021-05-24 22:34:57 <paladox> ok
2021-05-24 22:35:07 <RhinosF1> Unless we can guarntee maintenance or port it to something less breakable
2021-05-24 22:35:19 <RhinosF1> Then it's probably it's time
2021-05-24 22:35:22 <MacFan4000> We should probably open an issue on https://github.com/thaider/Tweeki/issues
2021-05-24 22:35:23 <MirahezeBot> [ Issues · thaider/Tweeki · GitHub ] - github.com
2021-05-24 22:35:51 <Crepusculum> RhinosF1: agreed.
2021-05-24 22:36:05 <RhinosF1> MacFan4000: there's issues dating back to Feb 2015
2021-05-24 22:36:28 <RhinosF1> Crepusculum: you have any desire to deal with it. If not, you know what to do.
2021-05-24 22:36:30 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.79, 7.43, 6.80
2021-05-24 22:36:32 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.44, 6.85, 6.51
2021-05-24 22:36:38 <icinga-miraheze> PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.53, 6.82, 6.32
2021-05-24 22:36:52 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.35, 7.19, 6.66
2021-05-24 22:38:29 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.92, 7.25, 6.81
2021-05-24 22:38:32 <icinga-miraheze> PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.32, 6.82, 6.56
2021-05-24 22:38:44 <icinga-miraheze> RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
2021-05-24 22:38:53 <icinga-miraheze> RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 22:39:01 <icinga-miraheze> RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 22:40:52 <Crepusculum> RhinosF1: yeah. I do want to see what Reception123 thinks first though.
2021-05-24 22:40:53 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.50, 7.55, 6.95
2021-05-24 22:41:12 <RhinosF1> Crepusculum: can you do a subtask of the upgrade one?
2021-05-24 22:41:55 <Crepusculum> RhinosF1: in a minute. My WiFi just went down, so I'm mobile for now.
2021-05-24 22:42:17 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 22:42:27 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.66, 6.80, 6.76
2021-05-24 22:42:31 <icinga-miraheze> RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.32, 6.44, 6.51
2021-05-24 22:42:36 <icinga-miraheze> RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 6.30, 6.30
2021-05-24 22:44:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.15, 6.69, 6.72
2021-05-24 22:46:22 <RhinosF1> Sleep for me
2021-05-24 22:47:53 <Crepusculum> OK, night.
2021-05-24 22:52:34 <paladox> ParserFirstCallInit is the cause
2021-05-24 22:52:38 <paladox> the hook within that skin
2021-05-24 22:55:23 <paladox> Crepusculum: RhinosF1 ^
2021-05-24 22:57:53 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 23:05:04 <icinga-miraheze> RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
2021-05-24 23:06:18 <icinga-miraheze> RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
2021-05-24 23:11:19 <icinga-miraheze> RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org
2021-05-24 23:19:15 <dmehus> is surprised Crepusculum doesn't have a USB mobile hub for his mobile phone so he can create a mobile LTE hotspot for his laptop in such events
2021-05-24 23:19:59 <icinga-miraheze> PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found
2021-05-24 23:21:47 <paladox> Crepusculum: https://github.com/thaider/Tweeki/pull/200
2021-05-24 23:21:48 <MirahezeBot> [ Fix checking for skin within onParserFirstCallInit by paladox · Pull Request #200 · thaider/Tweeki · GitHub ] - github.com
2021-05-24 23:34:20 <Crepusculum> paladox: oh. thanks!
2021-05-24 23:34:53 <icinga-miraheze> PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.31, 7.00, 6.07
2021-05-24 23:35:13 <icinga-miraheze> PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.30, 6.90, 6.33
2021-05-24 23:35:52 <Not-4c5b> ['miraheze/mw-config'] 'paladox' pushed '' commit to 'paladox-patch-1' [+0/-0/±1] 'https://git.io/JspI6'
2021-05-24 23:35:54 <Not-4c5b> ['miraheze/mw-config'] 'paladox' '248b20' - Disable cp10/cp11 in wgCdnServers
2021-05-24 23:35:55 <Not-4c5b> ['mw-config'] 'paladox' created branch 'paladox-patch-1' - 'https://git.io/vbvb3'
2021-05-24 23:35:57 <Not-4c5b> ['mw-config'] 'paladox' opened pull request '#3919': Disable cp10/cp11 in wgCdnServers - 'https://git.io/JspIi'
2021-05-24 23:36:52 <icinga-miraheze> RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.45, 6.10, 5.86
2021-05-24 23:36:54 <miraheze-github> miraheze/mw-config - paladox the build passed.
2021-05-24 23:37:12 <icinga-miraheze> RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.21, 6.24, 6.16
2021-05-24 23:38:38 <Not-4c5b> ['mw-config'] 'Universal-Omega' closed pull request '#3916': Remove two deprecated password policy checks - 'https://git.io/JsbPL'
2021-05-24 23:38:40 <Not-4c5b> ['miraheze/mw-config'] 'Universal-Omega' pushed '' commit to 'master' [+0/-0/±1] 'https://git.io/JspI7'
2021-05-24 23:38:41 <Not-4c5b> ['miraheze/mw-config'] 'Universal-Omega' 'e0b6f28' - Remove two deprecated password policy checks (#3916)
2021-05-24 23:38:43 <Not-4c5b> ['mw-config'] 'Universal-Omega' deleted branch 'Universal-Omega-patch-1' - 'https://git.io/vbvb3'
2021-05-24 23:38:44 <Not-4c5b> ['miraheze/mw-config'] 'Universal-Omega' deleted branch 'Universal-Omega-patch-1'
2021-05-24 23:39:21 <Not-4c5b> ['mw-config'] 'Universal-Omega' opened pull request '#3920': Merge branch 'master' into REL1_36 - 'https://git.io/JspIA'
2021-05-24 23:39:40 <miraheze-github> miraheze/mw-config - Universal-Omega the build passed.
2021-05-24 23:39:53 <Not-4c5b> ['mw-config'] 'Universal-Omega' closed pull request '#3920': Merge branch 'master' into REL1_36 - 'https://git.io/JspIA'
2021-05-24 23:39:55 <Not-4c5b> ['miraheze/mw-config'] 'Universal-Omega' pushed '' commit to 'REL1_36' [+0/-0/±1] 'https://git.io/JspLe'
2021-05-24 23:39:56 <Not-4c5b> ['miraheze/mw-config'] 'Universal-Omega' 'e0914f3' - Merge branch 'master' into REL1_36 (#3920)
2021-05-24 23:39:58 <Not-4c5b> ['mw-config'] 'Universal-Omega' synchronize pull request '#3824': Update config for 1.36 - 'https://git.io/JYjbk'
2021-05-24 23:40:24 <miraheze-github> miraheze/mw-config - Universal-Omega the build passed.
2021-05-24 23:40:55 <miraheze-github> miraheze/mw-config - Universal-Omega the build passed.

This page is generated from SQL logs, you can also download static txt files from here