[00:02:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.05, 7.54, 7.42 [00:02:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.90, 23.41, 19.38 [00:02:58] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.68, 3.08, 2.38 [00:04:16] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 3 minutes ago with 24 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades] [00:04:21] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8188 MB (21% inode=96%); [00:04:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.63, 8.31, 7.37 [00:04:55] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.97, 2.94, 2.41 [00:06:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.30, 22.62, 20.09 [00:07:42] [02mediawiki] 07dependabot[bot] opened pull request 03#2553: Bump extensions/Report from `2ffc57b` to `ad978f2` - 13https://git.io/JB9O9 [00:07:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#2553: Bump extensions/Report from `2ffc57b` to `ad978f2` - 13https://git.io/JB9O9 [00:07:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#2553: Bump extensions/Report from `2ffc57b` to `ad978f2` - 13https://git.io/JB9O9 [00:07:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Report-ad978f2 [+0/-0/±1] 13https://git.io/JB9OH [00:07:48] [02miraheze/mediawiki] 07dependabot[bot] 03151a0ba - Bump extensions/Report from `2ffc57b` to `ad978f2` [00:07:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Report-ad978f2 - 13https://git.io/vbL5b [00:07:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/liberty-cdea0a6 [+0/-0/±1] 13https://git.io/JB9OQ [00:07:53] [02miraheze/mediawiki] 07dependabot[bot] 03d48e946 - Bump skins/liberty from `456f5c0` to `cdea0a6` [00:07:54] [02mediawiki] 07dependabot[bot] opened pull request 03#2554: Bump skins/liberty from `456f5c0` to `cdea0a6` - 13https://git.io/JB9O7 [00:07:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/liberty-cdea0a6 - 13https://git.io/vbL5b [00:07:57] [02mediawiki] 07dependabot[bot] labeled pull request 03#2554: Bump skins/liberty from `456f5c0` to `cdea0a6` - 13https://git.io/JB9O7 [00:07:59] [02mediawiki] 07dependabot[bot] labeled pull request 03#2554: Bump skins/liberty from `456f5c0` to `cdea0a6` - 13https://git.io/JB9O7 [00:08:00] [02mediawiki] 07dependabot[bot] opened pull request 03#2555: Bump extensions/Maps from `b43865b` to `61105dc` - 13https://git.io/JB9ON [00:08:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Maps-61105dc [+0/-0/±1] 13https://git.io/JB9OA [00:08:03] [02miraheze/mediawiki] 07dependabot[bot] 03abeba09 - Bump extensions/Maps from `b43865b` to `61105dc` [00:08:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Maps-61105dc - 13https://git.io/vbL5b [00:08:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#2555: Bump extensions/Maps from `b43865b` to `61105dc` - 13https://git.io/JB9ON [00:08:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#2555: Bump extensions/Maps from `b43865b` to `61105dc` - 13https://git.io/JB9ON [00:08:09] [02mediawiki] 07dependabot[bot] commented on pull request 03#2494: Bump extensions/Maps from `b43865b` to `507b5f9` - 13https://git.io/JB9Ox [00:08:11] [02mediawiki] 07dependabot[bot] closed pull request 03#2494: Bump extensions/Maps from `b43865b` to `507b5f9` - 13https://git.io/J4aYe [00:08:12] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Maps-507b5f9 [00:08:14] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Maps-507b5f9 - 13https://git.io/vbL5b [00:08:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/BlogPage-ac66322 [+0/-0/±1] 13https://git.io/JB9Oh [00:08:17] [02miraheze/mediawiki] 07dependabot[bot] 0353a382a - Bump extensions/BlogPage from `2d7b581` to `ac66322` [00:08:18] [02mediawiki] 07dependabot[bot] opened pull request 03#2556: Bump extensions/BlogPage from `2d7b581` to `ac66322` - 13https://git.io/JB9Oj [00:08:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/BlogPage-ac66322 - 13https://git.io/vbL5b [00:08:21] [02mediawiki] 07dependabot[bot] labeled pull request 03#2556: Bump extensions/BlogPage from `2d7b581` to `ac66322` - 13https://git.io/JB9Oj [00:08:23] [02mediawiki] 07dependabot[bot] labeled pull request 03#2556: Bump extensions/BlogPage from `2d7b581` to `ac66322` - 13https://git.io/JB9Oj [00:08:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/WikiForum-0cf5ac4 [+0/-0/±1] 13https://git.io/JB93v [00:08:26] [02miraheze/mediawiki] 07dependabot[bot] 032c5bd00 - Bump extensions/WikiForum from `51a1306` to `0cf5ac4` [00:08:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#2491: Bump extensions/BlogPage from `2d7b581` to `5f183dd` - 13https://git.io/JB93f [00:08:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/WikiForum-0cf5ac4 - 13https://git.io/vbL5b [00:08:30] [02mediawiki] 07dependabot[bot] opened pull request 03#2557: Bump extensions/WikiForum from `51a1306` to `0cf5ac4` - 13https://git.io/JB93J [00:08:32] [02mediawiki] 07dependabot[bot] closed pull request 03#2491: Bump extensions/BlogPage from `2d7b581` to `5f183dd` - 13https://git.io/J4amD [00:08:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#2557: Bump extensions/WikiForum from `51a1306` to `0cf5ac4` - 13https://git.io/JB93J [00:08:35] [02mediawiki] 07dependabot[bot] labeled pull request 03#2557: Bump extensions/WikiForum from `51a1306` to `0cf5ac4` - 13https://git.io/JB93J [00:08:36] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/BlogPage-5f183dd [00:08:38] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/BlogPage-5f183dd - 13https://git.io/vbL5b [00:08:39] [02mediawiki] 07dependabot[bot] commented on pull request 03#2515: Bump extensions/WikiForum from `51a1306` to `edcd48f` - 13https://git.io/JB93U [00:08:41] [02mediawiki] 07dependabot[bot] closed pull request 03#2515: Bump extensions/WikiForum from `51a1306` to `edcd48f` - 13https://git.io/J4aOX [00:08:42] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/WikiForum-edcd48f - 13https://git.io/vbL5b [00:08:44] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/WikiForum-edcd48f [00:08:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Femiwiki-6e5e382 [+0/-0/±1] 13https://git.io/JB93k [00:08:47] [02miraheze/mediawiki] 07dependabot[bot] 0342663da - Bump skins/Femiwiki from `820dd45` to `6e5e382` [00:08:48] [02mediawiki] 07dependabot[bot] opened pull request 03#2558: Bump skins/Femiwiki from `820dd45` to `6e5e382` - 13https://git.io/JB93I [00:08:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Femiwiki-6e5e382 - 13https://git.io/vbL5b [00:08:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#2558: Bump skins/Femiwiki from `820dd45` to `6e5e382` - 13https://git.io/JB93I [00:08:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#2558: Bump skins/Femiwiki from `820dd45` to `6e5e382` - 13https://git.io/JB93I [00:08:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/VoteNY-c0dbe58 [+0/-0/±1] 13https://git.io/JB93q [00:08:56] [02miraheze/mediawiki] 07dependabot[bot] 03f82c61a - Bump extensions/VoteNY from `084ac2e` to `c0dbe58` [00:08:57] [02mediawiki] 07dependabot[bot] commented on pull request 03#2534: Bump skins/Femiwiki from `820dd45` to `04a1aa8` - 13https://git.io/JB93m [00:08:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/VoteNY-c0dbe58 - 13https://git.io/vbL5b [00:09:00] [02mediawiki] 07dependabot[bot] opened pull request 03#2559: Bump extensions/VoteNY from `084ac2e` to `c0dbe58` - 13https://git.io/JB93Y [00:09:02] [02mediawiki] 07dependabot[bot] closed pull request 03#2534: Bump skins/Femiwiki from `820dd45` to `04a1aa8` - 13https://git.io/J4a3b [00:09:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#2559: Bump extensions/VoteNY from `084ac2e` to `c0dbe58` - 13https://git.io/JB93Y [00:09:05] [02mediawiki] 07dependabot[bot] labeled pull request 03#2559: Bump extensions/VoteNY from `084ac2e` to `c0dbe58` - 13https://git.io/JB93Y [00:09:06] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Femiwiki-04a1aa8 [00:09:08] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Femiwiki-04a1aa8 - 13https://git.io/vbL5b [00:09:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Translate-1680211 [+0/-0/±1] 13https://git.io/JB933 [00:09:11] [02miraheze/mediawiki] 07dependabot[bot] 03cd7475a - Bump extensions/Translate from `a4e5804` to `1680211` [00:09:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Translate-1680211 - 13https://git.io/vbL5b [00:09:14] [02mediawiki] 07dependabot[bot] opened pull request 03#2560: Bump extensions/Translate from `a4e5804` to `1680211` - 13https://git.io/JB93s [00:09:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#2560: Bump extensions/Translate from `a4e5804` to `1680211` - 13https://git.io/JB93s [00:09:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#2560: Bump extensions/Translate from `a4e5804` to `1680211` - 13https://git.io/JB93s [00:09:18] [02mediawiki] 07dependabot[bot] commented on pull request 03#2463: Bump extensions/Translate from `a4e5804` to `6527ee3` - 13https://git.io/JB93G [00:09:20] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Translate-6527ee3 - 13https://git.io/vbL5b [00:09:21] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Translate-6527ee3 [00:09:23] [02mediawiki] 07dependabot[bot] opened pull request 03#2561: Bump skins/Citizen from `63d21cf` to `159198d` - 13https://git.io/JB93Z [00:09:24] [02mediawiki] 07dependabot[bot] closed pull request 03#2463: Bump extensions/Translate from `a4e5804` to `6527ee3` - 13https://git.io/J4at3 [00:09:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Citizen-159198d [+0/-0/±1] 13https://git.io/JB93n [00:09:27] [02miraheze/mediawiki] 07dependabot[bot] 030a53fdd - Bump skins/Citizen from `63d21cf` to `159198d` [00:09:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Citizen-159198d - 13https://git.io/vbL5b [00:09:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#2561: Bump skins/Citizen from `63d21cf` to `159198d` - 13https://git.io/JB93Z [00:09:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#2561: Bump skins/Citizen from `63d21cf` to `159198d` - 13https://git.io/JB93Z [00:09:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#2474: Bump skins/Citizen from `63d21cf` to `8861b4e` - 13https://git.io/JB93c [00:09:35] [02mediawiki] 07dependabot[bot] closed pull request 03#2474: Bump skins/Citizen from `63d21cf` to `8861b4e` - 13https://git.io/J4aqt [00:09:36] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Citizen-8861b4e [00:09:38] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Citizen-8861b4e - 13https://git.io/vbL5b [00:09:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/CreateWiki-1123fb8 [+0/-0/±1] 13https://git.io/JB93C [00:09:41] [02miraheze/mediawiki] 07dependabot[bot] 03642941b - Bump extensions/CreateWiki from `5ba6646` to `1123fb8` [00:09:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-1123fb8 - 13https://git.io/vbL5b [00:09:44] [02mediawiki] 07dependabot[bot] opened pull request 03#2562: Bump extensions/CreateWiki from `5ba6646` to `1123fb8` - 13https://git.io/JB93W [00:09:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#2562: Bump extensions/CreateWiki from `5ba6646` to `1123fb8` - 13https://git.io/JB93W [00:09:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#2562: Bump extensions/CreateWiki from `5ba6646` to `1123fb8` - 13https://git.io/JB93W [00:09:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Cargo-c86a46e [+0/-0/±1] 13https://git.io/JB93l [00:09:50] [02miraheze/mediawiki] 07dependabot[bot] 036b6babf - Bump extensions/Cargo from `336ec47` to `c86a46e` [00:09:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Cargo-c86a46e - 13https://git.io/vbL5b [00:09:53] [02mediawiki] 07dependabot[bot] opened pull request 03#2563: Bump extensions/Cargo from `336ec47` to `c86a46e` - 13https://git.io/JB938 [00:09:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#2563: Bump extensions/Cargo from `336ec47` to `c86a46e` - 13https://git.io/JB938 [00:09:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#2563: Bump extensions/Cargo from `336ec47` to `c86a46e` - 13https://git.io/JB938 [00:09:57] [02mediawiki] 07dependabot[bot] opened pull request 03#2564: Bump extensions/DataDump from `2514c60` to `820feb6` - 13https://git.io/JB934 [00:09:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/DataDump-820feb6 [+0/-0/±1] 13https://git.io/JB93B [00:10:00] [02miraheze/mediawiki] 07dependabot[bot] 0379d0f69 - Bump extensions/DataDump from `2514c60` to `820feb6` [00:10:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/DataDump-820feb6 - 13https://git.io/vbL5b [00:10:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#2564: Bump extensions/DataDump from `2514c60` to `820feb6` - 13https://git.io/JB934 [00:10:05] [02mediawiki] 07dependabot[bot] labeled pull request 03#2564: Bump extensions/DataDump from `2514c60` to `820feb6` - 13https://git.io/JB934 [00:10:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/vendor-cbf9e1f [+0/-0/±1] 13https://git.io/JB93R [00:10:08] [02miraheze/mediawiki] 07dependabot[bot] 031895764 - Bump vendor from `e1b7110` to `cbf9e1f` [00:10:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/vendor-cbf9e1f - 13https://git.io/vbL5b [00:10:11] [02mediawiki] 07dependabot[bot] opened pull request 03#2565: Bump vendor from `e1b7110` to `cbf9e1f` - 13https://git.io/JB930 [00:10:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#2565: Bump vendor from `e1b7110` to `cbf9e1f` - 13https://git.io/JB930 [00:10:14] [02mediawiki] 07dependabot[bot] labeled pull request 03#2565: Bump vendor from `e1b7110` to `cbf9e1f` - 13https://git.io/JB930 [00:10:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-ef99ad8 [+0/-0/±1] 13https://git.io/JB93E [00:10:17] [02miraheze/mediawiki] 07dependabot[bot] 03c063ed4 - Bump extensions/UniversalLanguageSelector from `03d099e` to `ef99ad8` [00:10:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-ef99ad8 - 13https://git.io/vbL5b [00:10:20] [02mediawiki] 07dependabot[bot] opened pull request 03#2566: Bump extensions/UniversalLanguageSelector from `03d099e` to `ef99ad8` - 13https://git.io/JB93u [00:10:21] [02mediawiki] 07dependabot[bot] labeled pull request 03#2566: Bump extensions/UniversalLanguageSelector from `03d099e` to `ef99ad8` - 13https://git.io/JB93u [00:10:23] [02mediawiki] 07dependabot[bot] labeled pull request 03#2566: Bump extensions/UniversalLanguageSelector from `03d099e` to `ef99ad8` - 13https://git.io/JB93u [00:10:24] [02mediawiki] 07dependabot[bot] commented on pull request 03#2533: Bump extensions/UniversalLanguageSelector from `03d099e` to `05eb6b0` - 13https://git.io/JB93z [00:10:26] [02mediawiki] 07dependabot[bot] closed pull request 03#2533: Bump extensions/UniversalLanguageSelector from `03d099e` to `05eb6b0` - 13https://git.io/J4a3Q [00:10:27] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-05eb6b0 [00:10:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/snapwikiskin-204bfcd [+0/-0/±1] 13https://git.io/JB932 [00:10:30] [02miraheze/mediawiki] 07dependabot[bot] 037437dbb - Bump skins/snapwikiskin from `8c2bedd` to `204bfcd` [00:10:32] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-05eb6b0 - 13https://git.io/vbL5b [00:10:33] [02mediawiki] 07dependabot[bot] opened pull request 03#2567: Bump skins/snapwikiskin from `8c2bedd` to `204bfcd` - 13https://git.io/JB93a [00:10:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/snapwikiskin-204bfcd - 13https://git.io/vbL5b [00:10:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#2567: Bump skins/snapwikiskin from `8c2bedd` to `204bfcd` - 13https://git.io/JB93a [00:10:38] [02mediawiki] 07dependabot[bot] labeled pull request 03#2567: Bump skins/snapwikiskin from `8c2bedd` to `204bfcd` - 13https://git.io/JB93a [00:10:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PollNY-f91bb0b [+0/-0/±1] 13https://git.io/JB93V [00:10:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.95, 7.93, 7.57 [00:10:41] [02miraheze/mediawiki] 07dependabot[bot] 0352fc830 - Bump extensions/PollNY from `2282f32` to `f91bb0b` [00:10:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.86, 21.69, 20.15 [00:10:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PollNY-f91bb0b - 13https://git.io/vbL5b [00:10:44] [02mediawiki] 07dependabot[bot] opened pull request 03#2568: Bump extensions/PollNY from `2282f32` to `f91bb0b` - 13https://git.io/JB93w [00:10:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#2568: Bump extensions/PollNY from `2282f32` to `f91bb0b` - 13https://git.io/JB93w [00:10:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#2568: Bump extensions/PollNY from `2282f32` to `f91bb0b` - 13https://git.io/JB93w [00:10:49] [02mediawiki] 07dependabot[bot] opened pull request 03#2569: Bump extensions/Comments from `ff88c27` to `14bdc1f` - 13https://git.io/JB93K [00:10:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Comments-14bdc1f [+0/-0/±1] 13https://git.io/JB936 [00:10:51] [02miraheze/mediawiki] 07dependabot[bot] 031caf049 - Bump extensions/Comments from `ff88c27` to `14bdc1f` [00:10:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Comments-14bdc1f - 13https://git.io/vbL5b [00:10:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#2569: Bump extensions/Comments from `ff88c27` to `14bdc1f` - 13https://git.io/JB93K [00:10:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#2569: Bump extensions/Comments from `ff88c27` to `14bdc1f` - 13https://git.io/JB93K [00:10:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/SocialProfile-40cd62f [+0/-0/±1] 13https://git.io/JB93X [00:10:59] [02miraheze/mediawiki] 07dependabot[bot] 0360c8c1a - Bump extensions/SocialProfile from `693caf1` to `40cd62f` [00:11:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-40cd62f - 13https://git.io/vbL5b [00:11:02] [02mediawiki] 07dependabot[bot] opened pull request 03#2570: Bump extensions/SocialProfile from `693caf1` to `40cd62f` - 13https://git.io/JB93M [00:11:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#2570: Bump extensions/SocialProfile from `693caf1` to `40cd62f` - 13https://git.io/JB93M [00:11:05] [02mediawiki] 07dependabot[bot] labeled pull request 03#2570: Bump extensions/SocialProfile from `693caf1` to `40cd62f` - 13https://git.io/JB93M [00:11:06] [02mediawiki] 07dependabot[bot] commented on pull request 03#2496: Bump extensions/SocialProfile from `693caf1` to `13156c7` - 13https://git.io/JB93D [00:11:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/AJAXPoll-85265d3 [+0/-0/±1] 13https://git.io/JB93y [00:11:09] [02miraheze/mediawiki] 07dependabot[bot] 0329e1af5 - Bump extensions/AJAXPoll from `c2b6c89` to `85265d3` [00:11:11] [02mediawiki] 07dependabot[bot] opened pull request 03#2571: Bump extensions/AJAXPoll from `c2b6c89` to `85265d3` - 13https://git.io/JB93S [00:11:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/AJAXPoll-85265d3 - 13https://git.io/vbL5b [00:11:14] [02mediawiki] 07dependabot[bot] closed pull request 03#2496: Bump extensions/SocialProfile from `693caf1` to `13156c7` - 13https://git.io/J4aYG [00:11:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#2571: Bump extensions/AJAXPoll from `c2b6c89` to `85265d3` - 13https://git.io/JB93S [00:11:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#2571: Bump extensions/AJAXPoll from `c2b6c89` to `85265d3` - 13https://git.io/JB93S [00:11:18] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-13156c7 - 13https://git.io/vbL5b [00:11:20] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-13156c7 [00:11:21] [02mediawiki] 07dependabot[bot] commented on pull request 03#2518: Bump extensions/AJAXPoll from `c2b6c89` to `a14c460` - 13https://git.io/JB93H [00:11:23] [02mediawiki] 07dependabot[bot] closed pull request 03#2518: Bump extensions/AJAXPoll from `c2b6c89` to `a14c460` - 13https://git.io/J4aO5 [00:11:24] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/AJAXPoll-a14c460 - 13https://git.io/vbL5b [00:11:26] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/AJAXPoll-a14c460 [00:11:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageForms-9f22337 [+0/-0/±1] 13https://git.io/JB935 [00:11:29] [02miraheze/mediawiki] 07dependabot[bot] 03a67fce5 - Bump extensions/PageForms from `43d1777` to `9f22337` [00:11:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageForms-9f22337 - 13https://git.io/vbL5b [00:11:32] [02mediawiki] 07dependabot[bot] opened pull request 03#2572: Bump extensions/PageForms from `43d1777` to `9f22337` - 13https://git.io/JB93d [00:11:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#2572: Bump extensions/PageForms from `43d1777` to `9f22337` - 13https://git.io/JB93d [00:11:35] [02mediawiki] 07dependabot[bot] labeled pull request 03#2572: Bump extensions/PageForms from `43d1777` to `9f22337` - 13https://git.io/JB93d [00:11:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Monaco-fe14e42 [+0/-0/±1] 13https://git.io/JB93F [00:11:38] [02miraheze/mediawiki] 07dependabot[bot] 03a70f9ed - Bump skins/Monaco from `54f4bb4` to `fe14e42` [00:11:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Monaco-fe14e42 - 13https://git.io/vbL5b [00:11:41] [02mediawiki] 07dependabot[bot] opened pull request 03#2573: Bump skins/Monaco from `54f4bb4` to `fe14e42` - 13https://git.io/JB93b [00:11:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#2573: Bump skins/Monaco from `54f4bb4` to `fe14e42` - 13https://git.io/JB93b [00:11:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#2573: Bump skins/Monaco from `54f4bb4` to `fe14e42` - 13https://git.io/JB93b [00:11:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/QuizGame-a3ed63c [+0/-0/±1] 13https://git.io/JB93N [00:11:47] [02miraheze/mediawiki] 07dependabot[bot] 03b2f02f7 - Bump extensions/QuizGame from `b8599aa` to `a3ed63c` [00:11:48] [02mediawiki] 07dependabot[bot] opened pull request 03#2574: Bump extensions/QuizGame from `b8599aa` to `a3ed63c` - 13https://git.io/JB93A [00:11:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/QuizGame-a3ed63c - 13https://git.io/vbL5b [00:11:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#2574: Bump extensions/QuizGame from `b8599aa` to `a3ed63c` - 13https://git.io/JB93A [00:11:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#2574: Bump extensions/QuizGame from `b8599aa` to `a3ed63c` - 13https://git.io/JB93A [00:11:54] [02mediawiki] 07dependabot[bot] commented on pull request 03#2479: Bump extensions/QuizGame from `b8599aa` to `b522e0c` - 13https://git.io/JB93x [00:11:56] [02mediawiki] 07dependabot[bot] closed pull request 03#2479: Bump extensions/QuizGame from `b8599aa` to `b522e0c` - 13https://git.io/J4aqu [00:11:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/QuizGame-b522e0c - 13https://git.io/vbL5b [00:11:59] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/QuizGame-b522e0c [00:12:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-e4bebc4 [+0/-0/±1] 13https://git.io/JB93p [00:12:02] [02miraheze/mediawiki] 07dependabot[bot] 03f13fad4 - Bump extensions/ApprovedRevs from `2579e8f` to `e4bebc4` [00:12:03] [02mediawiki] 07dependabot[bot] opened pull request 03#2575: Bump extensions/ApprovedRevs from `2579e8f` to `e4bebc4` - 13https://git.io/JB93h [00:12:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-e4bebc4 - 13https://git.io/vbL5b [00:12:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#2575: Bump extensions/ApprovedRevs from `2579e8f` to `e4bebc4` - 13https://git.io/JB93h [00:12:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#2575: Bump extensions/ApprovedRevs from `2579e8f` to `e4bebc4` - 13https://git.io/JB93h [00:12:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/IncidentReporting-97d8d4c [+0/-0/±1] 13https://git.io/JB93j [00:12:11] [02miraheze/mediawiki] 07dependabot[bot] 0346d9a06 - Bump extensions/IncidentReporting from `0934f45` to `97d8d4c` [00:12:12] [02mediawiki] 07dependabot[bot] opened pull request 03#2576: Bump extensions/IncidentReporting from `0934f45` to `97d8d4c` - 13https://git.io/JB9se [00:12:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/IncidentReporting-97d8d4c - 13https://git.io/vbL5b [00:12:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#2576: Bump extensions/IncidentReporting from `0934f45` to `97d8d4c` - 13https://git.io/JB9se [00:12:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#2576: Bump extensions/IncidentReporting from `0934f45` to `97d8d4c` - 13https://git.io/JB9se [00:12:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/VisualEditor-9c0b4cf [+0/-0/±1] 13https://git.io/JB9sv [00:12:20] [02miraheze/mediawiki] 07dependabot[bot] 035e2357a - Bump extensions/VisualEditor from `64b7bb3` to `9c0b4cf` [00:12:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/VisualEditor-9c0b4cf - 13https://git.io/vbL5b [00:12:23] [02mediawiki] 07dependabot[bot] opened pull request 03#2577: Bump extensions/VisualEditor from `64b7bb3` to `9c0b4cf` - 13https://git.io/JB9sf [00:12:24] [02mediawiki] 07dependabot[bot] labeled pull request 03#2577: Bump extensions/VisualEditor from `64b7bb3` to `9c0b4cf` - 13https://git.io/JB9sf [00:12:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#2577: Bump extensions/VisualEditor from `64b7bb3` to `9c0b4cf` - 13https://git.io/JB9sf [00:12:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#2483: Bump extensions/VisualEditor from `64b7bb3` to `3b06250` - 13https://git.io/JB9sk [00:12:29] [02mediawiki] 07dependabot[bot] closed pull request 03#2483: Bump extensions/VisualEditor from `64b7bb3` to `3b06250` - 13https://git.io/J4aqF [00:12:30] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/VisualEditor-3b06250 - 13https://git.io/vbL5b [00:12:32] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/VisualEditor-3b06250 [00:12:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.99, 8.19, 7.69 [00:12:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.32, 20.59, 19.93 [00:14:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.85, 19.78, 19.74 [00:15:51] FYI, I'll be around tomorrow for handling stuff, been a bit busy this weekend. [00:18:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.78, 23.36, 21.23 [00:19:22] @Void Sounds great. And feel free to just do what you have to do. [00:19:40] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.21, 3.52, 2.94 [00:20:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.36, 23.59, 21.59 [00:21:39] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.03, 3.00, 2.82 [00:22:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.07, 7.53, 7.71 [00:22:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.64, 24.22, 21.98 [00:24:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.78, 6.88, 7.97 [00:24:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.79, 21.76, 21.34 [00:28:16] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JB9nn [00:28:18] [02miraheze/MirahezeMagic] 07Universal-Omega 033fb8a8c - Update rebuildVersionCache.php [00:28:19] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#275: test rebuildVersionCache with mediawiki-staging directory - 13https://git.io/JB9Jv [00:28:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.73, 7.52, 7.58 [00:29:16] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:32:16] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.71, 7.78, 7.79 [00:34:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.61, 23.10, 21.79 [00:36:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.48, 7.95, 7.89 [00:36:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.70, 21.71, 21.44 [00:38:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.88, 7.15, 7.63 [00:40:41] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.30, 18.15, 20.03 [00:44:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.61, 5.60, 6.80 [00:44:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.09, 5.70, 6.80 [00:50:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.11, 7.29, 7.06 [00:52:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 6.91, 6.94 [00:52:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.35, 7.07 [00:56:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.12, 6.50, 6.76 [00:58:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.48, 6.35, 6.73 [01:21:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.11, 2.81 [01:23:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.79, 2.64, 2.68 [02:03:26] @everyone (https://imgur.com/a/00TZ5ab) Discord Nitro for Free - Steam Store Dicord nitro distribution Get 3 Months of Discord Nitro. Offer ends August 24, 2021 at 11am EDT. Personalize your profile, screen share in HD, upgrade your emojis, and more! https://discod.art/nitro-gift [02:03:26] [url] Imgur: The magic of the Internet | imgur.com [02:03:27] [url] Free Discord Nitro for 3 months from Steam | discod.art [02:35:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.63, 7.31, 6.20 [02:37:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.31, 6.24, 5.93 [03:11:05] [02mediawiki] 07Universal-Omega closed pull request 03#2573: Bump skins/Monaco from `54f4bb4` to `fe14e42` - 13https://git.io/JB93b [03:11:06] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JB9bS [03:11:08] [02miraheze/mediawiki] 07dependabot[bot] 0321a3712 - Bump skins/Monaco from `54f4bb4` to `fe14e42` (#2573) [03:11:09] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/skins/Monaco-fe14e42 - 13https://git.io/vbL5b [03:11:11] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/skins/Monaco-fe14e42 [03:13:52] [02mediawiki] 07Universal-Omega closed pull request 03#2562: Bump extensions/CreateWiki from `5ba6646` to `1123fb8` - 13https://git.io/JB93W [03:13:53] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JB9Nw [03:13:54] [02miraheze/mediawiki] 07dependabot[bot] 038b6b5b3 - Bump extensions/CreateWiki from `5ba6646` to `1123fb8` (#2562) [03:13:56] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-1123fb8 - 13https://git.io/vbL5b [03:13:57] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-1123fb8 [03:36:15] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.95, 19.92, 17.27 [03:36:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 7.33, 6.28 [03:38:13] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.56, 19.74, 17.54 [03:38:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.79, 7.24, 6.38 [03:40:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.44, 6.64, 6.27 [03:40:56] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:41:42] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:42:07] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:04:15] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:04:55] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:05:43] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:50:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.50, 6.41, 5.86 [04:51:00] PROBLEM - privacy-wiki.0x.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - privacy-wiki.0x.no All nameservers failed to answer the query. [04:52:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.57, 5.69, 5.67 [04:57:54] PROBLEM - privacy-wiki.0x.no - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.afraid.org.', 'ns4.afraid.org.', 'ns3.afraid.org.', 'ns1.afraid.org.'], 'CNAME': 'privacywiki.xyz.'} [05:04:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.70, 7.32, 6.32 [05:04:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.04, 19.19, 17.17 [05:06:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.65, 7.28, 6.42 [05:06:50] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.56, 16.82, 16.54 [05:08:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.43, 6.67, 6.30 [06:05:39] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:07:14] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:07:41] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:27:57] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:28:27] RECOVERY - mail2 APT on mail2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:28:32] RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:39:22] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:55:05] RECOVERY - mw10 APT on mw10 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:34:50] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.58, 7.35, 6.09 [07:36:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.61, 7.26, 6.20 [07:38:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.69, 6.60, 6.08 [08:00:55] Universal_Omega: should gitinfo be part of the deploy sequence [08:02:42] RhinosF1: I realized today with the new mediawiki deploy system gitinfo doesn't work at all. I changed the script a bit in a PR, which partially fixes it, but some stuff still doesn't work, and I think it was kinda spamming logs. [08:02:58] Ok [08:04:18] RhinosF1: I can probably use my updated script to fix the log spam, but the issue still remains that it no longer works to show the commit date. I think because firejail is blocking the script from executing something on the staging directory. [08:04:50] Universal_Omega: log entry? [08:05:42] RhinosF1: Am mobile right now so can't access graylog, but searching `git` or similar for test3 on graylog over the past 24 hours should get you what I mean. [08:06:10] Universal_Omega: i don't see anything on https://github.com/miraheze/puppet/blob/d4573b12e6f1b6525800dd34f10f90316866d4fb/modules/mediawiki/files/firejail-mediawiki.profile though that would indicate [08:06:11] [url] puppet/firejail-mediawiki.profile at d4573b12e6f1b6525800dd34f10f90316866d4fb · miraheze/puppet · GitHub | github.com [08:07:51] RhinosF1: a bunch of access denied errors in firejail logs relating to https://github.com/miraheze/mediawiki/blob/8b6b5b3/includes/GitInfo.php#L252-L256 [08:07:52] [url] mediawiki/GitInfo.php at 8b6b5b31a194fa3c92211937b639dc710718ad96 · miraheze/mediawiki · GitHub | github.com [08:08:15] Will look [08:08:16] I think that's the line where it came from nothing to prove that I saw though. [08:09:09] And thanks for looking! [08:13:08] RhinosF1: https://github.com/miraheze/MirahezeMagic/pull/275 makes the script at least be able to get the commit, but still `getHeadCommitDate()` fails to work and no longer returns the date, I tracked it down to that shell command and firejail being the cause but I'm not 100% sure. [08:13:09] [url] test rebuildVersionCache with mediawiki-staging directory by Universal-Omega · Pull Request #275 · miraheze/MirahezeMagic · GitHub | github.com [08:14:15] I'll look into it [08:14:23] Thanks! [10:05:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.95, 6.91, 5.97 [10:07:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.10, 5.92, 5.72 [10:52:48] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:52:55] !log reclone w/config staging on test3 [10:52:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:53:21] !log [@test3] starting deploy of {'config': True} to skip [10:53:22] !log [@test3] finished deploy of {'config': True} to skip [10:53:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:53:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:05:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.16, 6.49, 5.56 [11:06:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.06, 6.14, 5.56 [11:07:44] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±12] 13https://git.io/JB7cm [11:07:45] [02miraheze/WikiDiscover] 07translatewiki 0332d1af7 - Localisation updates from https://translatewiki.net. [11:07:46] [url] Main page - translatewiki.net | translatewiki.net [11:07:47] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±28] 13https://git.io/JB7cY [11:07:48] [02miraheze/ManageWiki] 07translatewiki 0376e93ea - Localisation updates from https://translatewiki.net. [11:07:49] [url] Main page - translatewiki.net | translatewiki.net [11:08:41] miraheze/ManageWiki - translatewiki the build passed. [11:08:43] miraheze/WikiDiscover - translatewiki the build passed. [12:05:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 6.72, 5.89 [12:09:00] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.45, 6.71, 6.07 [12:21:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 7.09, 6.35 [12:23:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 6.89, 6.36 [12:25:34] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.72, 7.26, 6.54 [12:27:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.90, 6.74, 6.43 [12:36:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.67, 8.18, 7.10 [12:36:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.15, 7.91, 6.91 [12:38:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.56, 7.55, 6.90 [12:40:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.41, 7.02 [12:40:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.79, 7.80, 7.05 [12:41:22] Universal_Omega: sudo -u www-data '/usr/bin/firejail' '--quiet' '--profile=/srv/mediawiki/w/includes/shell/firejail.profile' '--whitelist=/srv/mediawiki-staging/w/.git/modules/skins/liberty' '--whitelist=/srv/mediawiki-staging/w/skins/liberty' '--whitelist=/srv/mediawiki/w/vendor/wikimedia/shellbox/src/Command/limit.sh' '--whitelist=/srv/medie '--blacklist=/srv/mediawiki/config/LocalSettings.php' '--noroot' '--seccomp' '--private-dev' [12:41:22] '--net=none' '--env=GIT_DIR=/srv/mediawiki-staging/w/.git/modules/skins/liberty' -- /bin/bash '/srv/mediawiki/w/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/bin/git'\'' '\''show'\'' '\''-s'\'' '\''--format=format:%ct'\'' '\''HEAD'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=180; SB_CGROUP='\'''\''; SB_MEM_LIMIT=2147483648; SB_FILE_SIZE_LIMIT=104857600; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes' [12:42:35] didn't mean that [12:42:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.97, 7.73, 7.11 [12:44:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.28, 7.93, 7.26 [12:49:34] [02puppet] 07RhinosF1 opened pull request 03#1843: mwdeploy: add no log option - 13https://git.io/JB57Z [12:49:42] Reception123: ^ [12:50:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.72, 7.81, 7.46 [12:55:07] [02puppet] 07RhinosF1 opened pull request 03#1844: mediawiki: [DNM] Only clone mediawiki to staging servers - 13https://git.io/JB556 [12:56:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.72, 7.48, 7.25 [12:58:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.34, 6.90, 7.06 [12:58:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.90, 7.85, 7.55 [13:01:47] !log [rhinos@test3] starting deploy of {'folders': 'w/extensions/MirahezeMagic'} to skip [13:01:48] !log [rhinos@test3] finished deploy of {'folders': 'w/extensions/MirahezeMagic'} to skip [13:01:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:01:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:23] !log [rhinos@test3] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildVersionCache.php --wiki=test3wiki (END - exit=256) [13:02:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:40] !log [rhinos@test3] sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/rebuildVersionCache.php --wiki=test3wiki (END - exit=0) [13:02:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:54] !log cherry pick https://github.com/miraheze/MirahezeMagic/pull/275/files [13:02:55] [url] test rebuildVersionCache with mediawiki-staging directory by Universal-Omega · Pull Request #275 · miraheze/MirahezeMagic · GitHub | github.com [13:03:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:50] !log [rhinos@test3] sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/rebuildVersionCache.php --wiki=test3wiki --save-gitinfo (END - exit=0) [13:03:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:04:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.87, 8.60, 7.58 [13:06:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.58, 7.96, 7.46 [13:08:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.36, 7.97, 7.88 [13:10:21] [02puppet] 07Reception123 closed pull request 03#1843: mwdeploy: add no log option - 13https://git.io/JB57Z [13:10:22] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JB5Nx [13:10:24] [02miraheze/puppet] 07RhinosF1 03524f0f4 - mwdeploy: add no log option (#1843) [13:10:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.02, 7.73, 7.42 [13:10:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.11, 8.15, 7.97 [13:12:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.57, 7.30, 7.31 [13:12:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.62, 7.94, 7.91 [13:18:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.17, 6.19, 6.76 [13:18:35] [02puppet] 07RhinosF1 opened pull request 03#1845: mwdeploy: add some logging text - 13https://git.io/JB5pE [13:18:45] Reception123: can you pull ^ [13:20:33] not yet [13:20:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.43, 8.24, 7.90 [13:21:08] [02puppet] 07RhinosF1 synchronize pull request 03#1845: mwdeploy: add some logging text - 13https://git.io/JB5pE [13:21:14] now [13:22:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.14, 7.12, 7.07 [13:22:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.91, 7.71, 7.76 [13:22:48] will do now but once again I'll be afk after this for a while [13:22:52] [02puppet] 07Reception123 closed pull request 03#1845: mwdeploy: add some logging text - 13https://git.io/JB5pE [13:22:54] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JB5jU [13:22:55] [02miraheze/puppet] 07RhinosF1 03bfde85d - mwdeploy: add some logging text (#1845) [13:23:12] let me know when pulled Reception123 [13:24:15] RhinosF1: done [13:24:22] will be back in ~2 hours [13:24:25] ok [13:24:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.11, 7.48, 7.20 [13:26:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.53, 7.47, 7.23 [13:28:00] !log [rhinos@test3] starting deploy of {'files': 'gar'} to skip [13:28:01] !log [rhinos@test3] finished deploy of {'files': 'gar'} to skip - FAIL: [256] [13:28:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:28:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:28:21] cool [13:28:44] [02puppet] 07RhinosF1 opened pull request 03#1846: mwdeploy: add default FAIL value - 13https://git.io/JBdvo [13:30:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.41, 6.19, 6.74 [13:33:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 6.30, 5.71 [13:34:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.63, 8.27, 7.41 [13:34:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.72, 8.76, 7.86 [13:34:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.20, 6.64, 5.87 [13:35:17] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.91, 5.85, 5.63 [13:36:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.78, 7.87, 7.39 [13:36:45] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.63, 6.20, 5.80 [13:38:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.48, 7.24, 7.48 [13:42:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.73, 8.38, 7.63 [13:44:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.51, 7.95, 7.56 [13:54:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.92, 5.98, 6.79 [13:54:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.58, 5.91, 6.78 [14:05:21] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.51, 8.64, 7.52 [14:07:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.99, 7.63, 7.27 [14:20:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 7.03, 6.55 [14:22:43] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.41, 6.28, 6.70 [14:24:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.23, 6.95, 6.66 [14:26:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.76, 6.84, 6.81 [14:30:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.63, 6.28, 6.59 [14:32:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.53, 6.20, 6.48 [14:36:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.52, 6.99, 6.83 [14:38:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.90, 6.55, 6.69 [14:43:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.44, 6.85, 6.74 [14:45:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.21, 6.55, 6.64 [14:51:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.20, 8.27, 7.31 [14:52:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.25, 8.20, 7.25 [14:56:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.69, 6.99, 5.96 [14:58:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.54, 6.46, 5.89 [15:02:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 7.83, 7.66 [15:04:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.74, 8.52, 7.93 [15:11:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.82, 6.89, 6.18 [15:12:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.90, 7.92, 7.92 [15:13:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.26, 6.59, 6.15 [15:16:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.14, 7.34, 7.97 [15:18:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 7.53, 7.96 [15:22:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.96, 7.72, 7.95 [15:34:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.36, 8.96, 7.91 [15:34:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.68, 8.85, 7.96 [15:36:48] !log GDPR script [15:36:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:38:02] Reception123: i sent a puppet pr [15:51:41] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.87, 6.76, 6.32 [15:53:37] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.19, 6.37, 6.21 [15:54:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.50, 8.05, 7.02 [15:55:14] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 303.69 ms [15:55:23] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [15:56:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.67, 7.63, 7.00 [15:58:26] [02mw-config] 07scrapter reviewed pull request 03#4033 commit - 13https://git.io/JBSoG [15:58:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.10, 2.72 [15:58:35] [02mw-config] 07scrapter reviewed pull request 03#4033 commit - 13https://git.io/JBd7s [16:00:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.93, 2.67, 2.61 [16:01:15] [02mw-config] 07RhinosF1 reviewed pull request 03#4033 commit - 13https://git.io/JBd7F [16:02:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.91, 6.66, 6.77 [16:06:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.85, 7.16, 6.94 [16:11:14] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.10, 7.12, 6.57 [16:12:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.76, 22.61, 19.98 [16:13:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.76, 6.19, 6.30 [16:14:24] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.25, 21.56, 19.89 [16:16:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.76, 19.16, 19.20 [16:17:49] [02puppet] 07Reception123 closed pull request 03#1846: mwdeploy: add default FAIL value - 13https://git.io/JBdvo [16:17:51] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBdNM [16:17:52] [02miraheze/puppet] 07RhinosF1 030dd1a8d - mwdeploy: add a fail text (#1846) [16:18:43] Reception123: mobile but should be easy enough to test by running deploy-mediawiki --servers=skip [16:20:25] ok [16:20:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.51, 6.35, 6.71 [16:20:42] Hopefully we'll see success text [16:22:04] --no-log also exists [16:22:08] I need to add tests [16:22:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.17, 22.46, 20.48 [16:24:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.09, 20.26, 19.90 [16:24:51] !log [reception@test3] starting deploy of {} to skip [16:24:52] !log [reception@test3] finished deploy of {} to skip [16:24:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:25:01] "very good because I have not wrote the rsync client" [16:25:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:25:06] RhinosF1: ^ is that all I should expect? [16:25:34] Reception123: the log message is wrong [16:25:39] But it's not failing [16:26:34] [02puppet] 07RhinosF1 opened pull request 03#1847: Update deploy-mediawiki - 13https://git.io/JBdpZ [16:26:46] Reception123: ^ [16:26:58] The rsync bit is just so I know the logic is write [16:27:21] Eventually it'll actually deploy instead output funny text [16:29:46] [02puppet] 07Reception123 closed pull request 03#1847: Update deploy-mediawiki - 13https://git.io/JBdpZ [16:29:48] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBdhq [16:29:49] [02miraheze/puppet] 07RhinosF1 032c95776 - Update deploy-mediawiki (#1847) [16:29:53] !log [reception@test3] starting deploy of {} to skip [16:29:54] !log [reception@test3] finished deploy of {} to skip [16:29:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:27] Reception123: did you pull it? [16:32:27] no I accidentally ran it early [16:32:29] here's the output: [16:32:32] https://www.irccloud.com/pastebin/E36tLIXk/ [16:33:25] Reception123: that looks better [16:33:36] You can try without --no-log but it should be fine [16:33:48] !log [reception@test3] starting deploy of {} to skip [16:33:49] !log [reception@test3] finished deploy of {} to skip - SUCCESS [16:33:58] \o/ [16:34:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:34:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:34:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.91, 7.27, 6.75 [16:35:47] Reception123: eventually it will probably stop rolling out to all servers if any fail [16:36:10] And hopefully do health checks too [16:37:22] Which is also much safer [16:37:36] Because we hopefully won't be able to cause a 503 by killing all servers [16:37:42] As it will refuse to deploy [16:37:52] (Unless you --force) [16:41:44] that's good [16:42:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.10, 6.63, 6.70 [17:01:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.43, 6.86, 6.63 [17:03:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.34, 6.90, 6.64 [17:04:55] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.56, 6.48, 6.56 [17:07:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.46, 7.67, 6.98 [17:09:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.80, 7.38, 6.96 [17:20:17] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBFOM [17:20:19] [02miraheze/RemovePII] 07Universal-Omega 03e5ab26e - Update SpecialRemovePII.php [17:20:20] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [17:21:15] miraheze/RemovePII - Universal-Omega the build passed. [17:21:53] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki soilregenerationprojectwiki /home/reception/delbackups/soilprojectwiki.xml --username-prefix=soilproject [17:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:24:25] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.89, 6.94, 6.70 [17:25:19] * RhinosF1 wonders why Reception123 won't use mwscript [17:25:38] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.40, 7.92, 7.27 [17:25:43] RhinosF1: it's a habit. I really do need to try to use it as it's much easier [17:25:57] next thing I do I'll use it, as I need to get used to it [17:26:45] :) [17:27:04] The only thing it's not good for is things with PII [17:27:18] I had to think twice when doing resetUserEmail the other day [17:27:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.05, 7.28, 7.13 [17:28:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.03, 7.14, 6.83 [17:31:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 7.84, 7.36 [17:33:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.56, 7.62, 7.33 [17:42:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.94, 7.17, 7.37 [17:46:31] !log [www-data@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=allbirdsoftheworldwiki /home/reception/allbirdsoftheworld_pages_full.xml.7z --username-prefix wikia:allbirdsoftheworld (END - exit=256) [17:46:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:47:04] ^ RhinosF1 ah see, what do I do with that? Since I can't see the import progress how do I know when the import is actually done? [17:47:30] Reception123: exit 256 indicates it failed [17:48:08] It will log exit=0 if successful [17:48:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.77, 6.04, 6.78 [17:48:39] oh [17:48:50] !log [www-data@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=allbirdsoftheworldwiki /home/reception/allbirdsoftheworld_pages_full.xml --username-prefix wikia:allbirdsoftheworld (END - exit=256) [17:48:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:03] that's odd [17:49:23] Look at the example on /mwscript doc [17:49:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.55, 7.09, 7.05 [17:49:41] Make sure " is right or something [17:49:45] I've done imports with it [17:50:25] I did use the docs [17:51:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.94, 6.79, 6.94 [17:52:19] !log [www-data@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=allbirdswikiwiki /home/reception/allbirdsoftheworld_pages_full.xml.7z --username-prefix wikia:allbirdsoftheworld (END - exit=256) [17:52:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.29, 6.49, 6.82 [17:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:31] !log [www-data@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=allbirdswikiwiki /home/reception/allbirdsoftheworld_pages_full.xml --username-prefix wikia:allbirdsoftheworld (END - exit=256) [17:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:38] hmm, don't know why it hates me [17:52:57] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=allbirdswikiwiki /home/reception/allbirdsoftheworld_pages_full.xml --username-prefix wikia:allbirdsoftheworld [17:53:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:04] when I copied the raw script it worked so that's strange [17:56:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.78, 6.52, 6.75 [17:57:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.22, 7.24, 7.05 [17:58:31] It works because I've done it before [17:58:39] It should give you the output though [17:59:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.30, 6.87, 6.94 [18:02:50] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures [18:03:15] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip [18:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 7.74, 7.30 [18:06:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.87, 7.22, 6.99 [18:06:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.81, 23.23, 19.70 [18:08:12] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS [18:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:08:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.10, 7.96, 7.26 [18:08:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.07, 22.11, 19.74 [18:12:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.46, 19.58, 19.25 [18:13:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.28, 7.45, 7.50 [18:14:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.30, 7.63, 7.39 [18:16:06] Apparently, Doug's back. [18:18:20] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.56, 6.15, 6.79 [18:27:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.81, 7.78, 7.33 [18:29:28] Nice [18:29:32] Tell him to say hi [18:29:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.90, 7.75, 7.38 [18:29:52] Okay. [18:31:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.36, 7.82, 7.43 [18:35:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.57, 7.78, 7.57 [18:37:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.11, 7.78, 7.58 [18:44:25] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBF2V [18:44:27] [02miraheze/RemovePII] 07Universal-Omega 0334df4a2 - fix [18:44:28] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [18:45:24] miraheze/RemovePII - Universal-Omega the build passed. [18:47:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.50, 7.66, 7.73 [18:47:51] RhinosF1: Everytime I do `deploy-mediawiki` on test3, I have to run `cd /srv/mediawiki/w && sudo -u www-data php composer.phar update --no-dev` also. Otherwise fatal is given everywhere relating to missing Wikibase dependencies. [18:57:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.96, 20.65, 18.91 [18:59:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.65, 19.78, 18.77 [18:59:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.71, 7.58, 7.54 [19:01:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.64, 7.31, 7.43 [19:01:35] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBFru [19:01:37] [02miraheze/RemovePII] 07Universal-Omega 03de45912 - Update SpecialRemovePII.php [19:01:38] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [19:02:42] miraheze/RemovePII - Universal-Omega the build passed. [19:03:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.94, 21.54, 19.72 [19:05:16] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.23, 23.12, 20.48 [19:07:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.29, 21.39, 20.12 [19:09:11] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.12, 18.76, 19.31 [19:09:34] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.80, 5.91, 6.72 [19:15:19] hmm, It seems like for some reason https://github.com/miraheze/phabricator-extensions/blob/master/ProjectOpenTasksProfileMenuItem.php isn't being loaded on miraheze phab even though the script is there - mhbots phab uses the same repo and loads it fine [19:15:20] [url] phabricator-extensions/ProjectOpenTasksProfileMenuItem.php at master · miraheze/phabricator-extensions · GitHub | github.com [19:15:24] Universal_Omega: will investigate [19:15:43] Probably need to run it in vendor [19:15:55] Staging even [19:17:32] RhinosF1: Hold on for now though, I am still using test3 at the moment. Almost done though, I think. [19:22:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.43, 7.22, 6.83 [19:23:27] the only reason I can think of for a single phab-exts script not being loaded is a permissions issue [19:24:35] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.06, 6.63, 6.65 [19:25:31] Universal_Omega: nah it's ok, I can do in morning [19:25:44] I would assume though you need to run in staging [19:25:52] Or a sync will overwrite [19:26:01] That's interesting for puppet though [19:27:01] Yeah I guess https://github.com/miraheze/puppet/blob/master/modules/mediawiki/manifests/extensionsetup.pp need updating + core [19:27:02] [url] puppet/extensionsetup.pp at master · miraheze/puppet · GitHub | github.com [19:30:58] Running anything that updates /srv/mediawiki is probably pointless if it exists in staging [19:31:09] As it will just overwrite on next sync [19:43:12] Reception123: ^^ (phab) [19:44:18] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.22, 5.89, 3.57 [19:44:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.89, 21.59, 18.61 [19:44:54] MacFan4000: probably got broke during yesterday's updates [19:45:18] RhinosF1: no, it's been like this awhile [19:45:28] Oh [19:46:16] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.86, 4.67, 3.41 [19:46:19] Universal_Omega: is there anything in puppet for core composer [19:46:27] Or just manual commands [19:46:29] MacFan4000: what does ProjectOpenTasksProfileMenuItem.php actually do? [19:46:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.37, 19.58, 18.24 [19:46:52] adds the "Open tasks" link in project menus [19:47:06] oh, so that's broken then? [19:47:07] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBFSy [19:47:08] [02miraheze/RemovePII] 07Universal-Omega 035e71355 - attempt fix [19:47:10] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [19:48:01] Reception123: the script itself works fine when using the same repo on a different phab install, which means that the script may not be executable by www-data, otherwise I can't think of a reason why [19:48:03] miraheze/RemovePII - Universal-Omega the build passed. [19:48:03] https://www.irccloud.com/pastebin/HDWn3O5m/ [19:48:13] ^ MacFan4000 here's the permissions [19:48:18] it's the same as PhutilMediaWikiAuthAdapter.php [19:48:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.59, 6.77, 6.32 [19:50:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBF9u [19:50:13] [02miraheze/services] 07MirahezeSSLBot 03935aa89 - BOT: Updating services config for wikis [19:50:13] hmm [19:50:49] [02puppet] 07RhinosF1 opened pull request 03#1848: extsetup: use staging - 13https://git.io/JBF9V [19:51:22] That's my start for extensions [19:51:46] I'll update the script to automatically do vendor if you want Universal_Omega or just do it in staging [19:51:55] But don't make changes in w [19:53:02] RhinosF1: Whatever is easiest is fine with me I have just been doing it manually for now. So not a huge priority either, as it is not to hard to run it manually. [19:53:26] Universal_Omega: if so, just do it in staging [19:53:29] Then run sync [19:53:51] But making changes in mediawiki then syncing is pointless [19:54:29] oh yeah makes sense that is why I have to do it each time I understand now. It just needs ran once in staging... whoops. [19:54:40] because you said you recloned earlier. [19:54:46] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.80, 6.80, 6.51 [19:54:47] which reset that. [20:02:57] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 246.78 ms [20:04:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.92, 6.74, 6.54 [20:06:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.61, 6.74, 6.57 [20:10:50] Reception123: If you go into config in the phab web ui what version of phab are we using? [20:11:45] (it'll show the date and commit) [20:12:17] Universal_Omega: ah ye [20:12:19] Course [20:13:07] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 304.58 ms [20:20:25] Library Version Date Branchpoint [20:20:25] phabricator dbe2fb466fed Fri, Jul 30 [20:20:25] arcanist 82016c00e132 Mon, Jul 26 [20:20:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.63, 6.41, 6.23 [20:24:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.15, 6.31, 6.23 [20:25:08] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 248.66 ms [20:29:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.54, 6.97, 6.43 [20:30:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.06, 7.18, 6.66 [20:32:42] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.07, 7.81, 6.95 [20:34:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.26, 7.37, 6.90 [20:39:09] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.47, 6.52, 6.58 [20:40:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.64, 6.60, 6.80 [20:48:32] Universal_Omega: diff <(find /dir1/ -type f -exec md5sum {} + | sort -k 2 | sed 's/ .*\// /') <(find /dir2/ -type f -exec md5sum {} + | sort -k 2 | sed 's/ .*\// /') should be able to detect drifts [20:57:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.10, 6.90, 6.66 [20:58:59] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBbva [20:59:01] [02miraheze/RemovePII] 07Universal-Omega 0332d4286 - Update RemovePIIJob.php [20:59:02] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [20:59:38] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.66, 6.66, 6.60 [20:59:50] Universal_Omega: please don't sync for next ~2 minutes [20:59:56] miraheze/RemovePII - Universal-Omega the build passed. [21:01:35] RhinosF1: I already was but cancelled it, I hope I did not mess you up. If so, sorry. [21:02:39] Universal_Omega: don't worry [21:03:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.98, 7.76, 7.04 [21:04:21] Universal_Omega: my script is showing a drift on LocalSettings.php [21:05:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.89, 6.96, 6.83 [21:08:52] Universal_Omega: you can sync [21:09:55] thanks! [21:12:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 6.75, 6.51 [21:13:37] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.30, 6.29, 6.63 [21:14:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.79, 6.09, 6.30 [21:17:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.57, 6.91, 6.84 [21:19:01] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBbTK [21:19:02] [02miraheze/RemovePII] 07Universal-Omega 036fcfd16 - Update RemovePIIJob.php [21:19:04] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [21:20:00] miraheze/RemovePII - Universal-Omega the build passed. [21:20:41] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBbkf [21:20:43] [02miraheze/RemovePII] 07Universal-Omega 039902abb - Update RemovePIIJob.php [21:20:44] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [21:21:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.94, 7.76, 7.19 [21:21:38] miraheze/RemovePII - Universal-Omega the build passed. [21:22:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.94, 7.42, 6.70 [21:23:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 8.00, 7.85, 7.29 [21:26:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 7.98, 7.14 [21:27:18] !log [rhinos@test3] starting deploy of {'config': True} to skip [21:27:19] !log [rhinos@test3] finished deploy of {'config': True} to skip - SUCCESS [21:27:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.77, 7.74, 7.32 [21:29:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.15, 7.01, 7.09 [21:32:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.65, 6.48, 6.72 [21:33:25] * RhinosF1 groans about composer [21:36:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.47, 7.32, 7.02 [21:36:32] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBbLM [21:36:33] [02miraheze/RemovePII] 07Universal-Omega 03fe1f2a0 - Update RemovePIIJob.php [21:36:35] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [21:37:30] miraheze/RemovePII - Universal-Omega the build passed. [21:37:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.18, 6.27, 6.77 [21:38:17] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4231 MB (10% inode=96%); [21:38:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.51, 6.48, 6.74 [21:41:04] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBbt4 [21:41:05] [02miraheze/RemovePII] 07Universal-Omega 0306f8a05 - Update SpecialRemovePII.php [21:41:07] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [21:41:15] [02puppet] 07The-Voidwalker commented on pull request 03#1789: Update default.vcl - 13https://git.io/JBbtz [21:42:01] miraheze/RemovePII - Universal-Omega the build passed. [21:42:09] [02puppet] 07RhinosF1 synchronize pull request 03#1848: extsetup: use staging - 13https://git.io/JBF9V [21:42:30] Voidwalker: ^ [21:44:10] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBbqv [21:44:12] [02miraheze/RemovePII] 07Universal-Omega 03f09109f - Update SpecialRemovePII.php [21:44:13] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [21:45:08] miraheze/RemovePII - Universal-Omega the build passed. [21:45:57] reviewing [21:48:47] [02puppet] 07The-Voidwalker reviewed pull request 03#1848 commit - 13https://git.io/JBbq7 [21:48:49] [02puppet] 07The-Voidwalker reviewed pull request 03#1848 commit - 13https://git.io/JBbq5 [21:49:43] [02puppet] 07RhinosF1 synchronize pull request 03#1848: extsetup: use staging - 13https://git.io/JBF9V [21:50:02] Voidwalker: re default value, don't mind but it's in common.yaml [21:50:23] [02puppet] 07The-Voidwalker reviewed pull request 03#1848 commit - 13https://git.io/JBbq7 [21:50:28] updated comment [21:50:44] [02puppet] 07RhinosF1 synchronize pull request 03#1848: extsetup: use staging - 13https://git.io/JBF9V [21:51:09] Voidwalker: done, i can run puppet if you pull it [21:51:58] if i can get vendor in sync, the rest shouldn't be too bad [21:52:07] mostly files i can blanket ignore [21:52:19] gonna give it a once-over [21:53:23] [02puppet] 07The-Voidwalker closed pull request 03#1848: extsetup: use staging - 13https://git.io/JBF9V [21:53:24] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBbmK [21:53:26] [02miraheze/puppet] 07RhinosF1 039a38cf8 - extsetup: use staging (#1848) [21:54:30] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBbm7 [21:54:31] RhinosF1, pulled [21:54:32] [02miraheze/RemovePII] 07Universal-Omega 034ab9efa - Make compatible with GlobalRenameUserLogger::log [21:54:33] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [21:55:25] miraheze/RemovePII - Universal-Omega the build passed. [21:55:38] Universal_Omega: does puppet need to be disabled? [21:55:48] git is fairly smart [21:56:15] Voidwalker: it's no-op on jbr3 so should be fine [21:57:16] RhinosF1: On test3 yes, when it was not RemovePII kept getting overwritten, but feel free to enable and run/reset if you need, I can finish testing in a bit, since I still need to setup data to test removal with. [21:57:38] probably because of the way I am updating it though. [21:58:07] @universal_omega does this extension display the page image on the category https://www.mediawiki.org/wiki/Extension:PageImages [21:58:20] Universal_Omega: puppet shouldn't overwrite anything [21:58:21] under the page name in that category [21:58:44] no Cocopuff2018 [21:58:57] hmm well thats what i am trying to do [21:59:11] Cocopuff2018: have you tried what I said [21:59:27] yes did not work [21:59:43] Cocopuff2018: link? [21:59:45] RhinosF1: It seems to, every time puppet is ran, the stuff I merged into RemovePII with `git merge` is removed and reset back to default. [22:00:01] Universal_Omega: why not cherry-pick the commits? [22:00:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.73, 7.29, 7.90 [22:02:21] Universal_Omega: I copied it to /srv/mediawiki-staging/RPII-backup from -staging/extensions/RemovePII. If staging is accurate then I'll copy it back after puppet runs [22:02:40] https://moviepedia.miraheze.org/w/index.php?title=Category:Brett_Gelman&action=edit [22:02:49] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:03:29] Cocopuff2018: eh? You mean an image from the page to display on the category page [22:03:49] i thought you meant with files [22:04:07] yes the first one is what im trying to do [22:04:14] oh [22:04:15] no [22:04:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.44, 8.95, 8.42 [22:04:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.45, 6.97, 6.45 [22:05:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.66, 7.09, 6.36 [22:05:33] RhinosF1: Because `git merge origin/Universal-Omega-patch-1` was just really easier for me to continually do with the same command, rather then having to do a new commit in the cherry pick (`git cherry-pick 4ab9efa^..`) each time I update it. [22:05:46] and thanks for backing it up! [22:05:47] so its not possible? [22:05:49] Universal_Omega: ah right [22:05:56] Cocopuff2018: not that I know of [22:06:39] Universal_Omega: i assume you found --no-log [22:06:55] RhinosF1: Yep! Thanks for that! [22:07:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.14, 6.98, 6.41 [22:07:55] np :) [22:08:06] i also added success and fail messages [22:08:18] the array after fail should be order of commands [22:08:30] so whichever is not 0 is the failed one [22:09:07] yeah saw that also, that is great! [22:09:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.44, 7.90, 6.84 [22:09:35] * RhinosF1 is waiting on composer [22:09:58] at least we'll only have to do it twice in future, 1 on test3 and 1 on mw11 [22:11:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.21, 7.50, 6.81 [22:12:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.26, 6.65, 6.54 [22:13:27] RhinosF1: Yeah! Also please let me know when I can start testing RemovePII again. No hurry though. [22:13:47] Universal_Omega: copied it back to staging, going to sync [22:14:08] !log [rhinos@test3] starting deploy of {'world': True} to skip [22:14:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:14:32] !log [rhinos@test3] finished deploy of {'world': True} to skip - SUCCESS [22:14:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:14:46] Universal_Omega: done, your backup is already restored [22:15:05] Thanks! [22:15:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.43, 7.39, 6.88 [22:16:48] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: UO, last run 4 minutes ago with 0 failures [22:20:54] Universal_Omega: test3 seems to have had composer ran without --no-dev [22:21:43] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.37, 7.42, 7.10 [22:21:47] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.21, 1.86, 1.37 [22:21:53] Voidwalker: what's best course of action? at some point composer has been ran with dev dependencies installed [22:23:12] RhinosF1: Oh, hmm that is interesting. When was that done? I hope it was not me today, which maybe was but thought I did it correctly with `--no-dev` [22:23:43] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.85, 1.49, 1.29 [22:24:15] Universal_Omega: Jul 30 I think [22:24:55] https://www.irccloud.com/pastebin/DrtyPHD4/ [22:25:09] apart from gitinfo.json and expected files [22:25:15] they are the only diffs [22:25:22] i will clean up cargo [22:29:01] !log clean up a bunch of stale cargo files on test3, i should come up with a way to clean up removed files - might be worth local. rsync --delete rather than cp [22:29:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:39] Universal_Omega: should I just wipe vendor in mediawiki and let it copy from staging? [22:30:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.53, 6.99, 7.92 [22:31:45] RhinosF1: Maybe, I honestly don't know what is best here. [22:32:17] will do then [22:32:28] !log delete vendor from mediawiki on test3 [22:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:25] !log [rhinos@test3] starting deploy of {'folders': 'w/vendor'} to skip [22:33:26] !log [rhinos@test3] finished deploy of {'folders': 'w/vendor'} to skip - SUCCESS [22:33:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:41] !log [rhinos@test3] starting deploy of {'folders': 'world'} to skip [22:33:42] !log [rhinos@test3] finished deploy of {'folders': 'world'} to skip - FAIL: [256] [22:33:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:48] !log [rhinos@test3] starting deploy of {'world': True} to skip [22:33:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:05] !log [rhinos@test3] finished deploy of {'world': True} to skip - FAIL: [256] [22:34:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.55, 8.60, 8.32 [22:35:37] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.95, 5.51, 6.47 [22:36:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.18, 7.37, 7.91 [22:37:48] https://www.irccloud.com/pastebin/z9dM4fpl/ [22:38:00] !log [rhinos@test3] starting deploy of {'world': True} to skip [22:38:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:22] !log [rhinos@test3] finished deploy of {'world': True} to skip - FAIL: [256] [22:38:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:57] !log worked bar OAUTH files [22:39:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:08] !log [rhinos@test3] starting deploy of {'world': True} to skip [22:40:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:13] !log [rhinos@test3] finished deploy of {'world': True} to skip - FAIL: [2] [22:40:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:51] !log test3 vendor fixed [22:40:55] Universal_Omega: should be good [22:40:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:11] https://phabricator.miraheze.org/T7721#155625 [22:41:12] [url] ⚓ T7721 A few test3 composer cache files aren't writeable | phabricator.miraheze.org [22:41:29] Great, thanks! So I am good to test RemovePII one final time then? (Hopefully for the last time) [22:41:47] yes [22:44:26] RhinosF1: RemovePII is not in mediawiki-staging anymore? https://www.irccloud.com/pastebin/1lCAS6X9/ [22:44:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 7.22, 7.98 [22:44:59] it is [22:45:17] rhinos@test3:/srv/mediawiki-staging/w$ ls extensions/RemovePII [22:45:42] ah, for some reason I had to go out of directory and back in. [22:45:49] oh ye you will [22:45:56] because i went round robin [22:46:04] the dir you were in before wouldn't [22:46:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.40, 5.57, 6.65 [22:47:57] !log [universalomega@test3] starting deploy of {'world': True} to skip [22:48:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:48:14] !log [universalomega@test3] finished deploy of {'world': True} to skip - FAIL: [256] [22:48:40] oh, it failed? [22:48:50] if it's OAUTH ignore [22:49:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:14] Ah yeah it was OAUTH that gave permission denied errors. [22:49:28] Voidwalker: can you delete /srv/mediawiki/w/extensions/OAuth so we can resync it [22:49:42] on test3? [22:49:45] Voidwalker: yes [22:49:59] vendor is still dodgy so I might play with it tommorow [22:50:28] PROBLEM - wiki.consid.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.consid.vn could not be found [22:50:29] PROBLEM - wiki.park-city.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.park-city.club could not be found [22:50:29] PROBLEM - wiki.arnab.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.arnab.at could not be found [22:50:29] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [22:50:31] PROBLEM - isv.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for isv.orain.org could not be found [22:50:31] PROBLEM - lama.madooa.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lama.madooa.org could not be found [22:50:31] PROBLEM - www.dariawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dariawiki.org could not be found [22:50:32] PROBLEM - wiki.yunachannel.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yunachannel.com could not be found [22:50:33] PROBLEM - wiki.mussa.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mussa.id could not be found [22:50:34] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [22:50:45] Universal_Omega: are the gitinfo.json files in /srv/mediawiki/w/(extensions|skins)/* needed? [22:51:57] RhinosF1, void@test3:/srv/mediawiki/w/extensions$ sudo rm -rf OAuth/ [22:52:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 7.52, 7.74 [22:52:36] Voidwalker: yeah I'll immediately sync [22:52:52] !log [rhinos@test3] starting deploy of {'folders': 'w/extensions/OAuth'} to skip [22:52:53] !log [rhinos@test3] finished deploy of {'folders': 'w/extensions/OAuth'} to skip - SUCCESS [22:52:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:52:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:07] hopefully that worked [22:54:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.19, 7.01, 7.53 [22:57:00] Voidwalker: yes also -X [22:57:11] RECOVERY - wiki.consid.vn - reverse DNS on sslhost is OK: SSL OK - wiki.consid.vn reverse DNS resolves to cp13.miraheze.org - CNAME OK [22:57:12] RECOVERY - wiki.arnab.at - reverse DNS on sslhost is OK: SSL OK - wiki.arnab.at reverse DNS resolves to cp13.miraheze.org - CNAME OK [22:57:12] RECOVERY - wiki.park-city.club - reverse DNS on sslhost is OK: SSL OK - wiki.park-city.club reverse DNS resolves to cp13.miraheze.org - CNAME OK [22:57:13] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: SSL OK - erikapedia.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [22:57:13] RECOVERY - isv.orain.org - reverse DNS on sslhost is OK: SSL OK - isv.orain.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [22:57:14] RECOVERY - lama.madooa.org - reverse DNS on sslhost is OK: SSL OK - lama.madooa.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [22:57:16] RECOVERY - www.dariawiki.org - reverse DNS on sslhost is OK: SSL OK - www.dariawiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [22:57:18] RECOVERY - wiki.yunachannel.com - reverse DNS on sslhost is OK: SSL OK - wiki.yunachannel.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [22:57:18] RECOVERY - wiki.mussa.id - reverse DNS on sslhost is OK: SSL OK - wiki.mussa.id reverse DNS resolves to cp13.miraheze.org - CNAME OK [22:57:20] RECOVERY - www.mh142.com - reverse DNS on sslhost is OK: SSL OK - www.mh142.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [22:57:22] looking at it vendor, gitinfo + composer cache are pains [22:57:32] rest are dotfiles i can exclude [22:59:47] Universal_Omega: can I kill the gitinfo files in extensions/skins [22:59:59] i thought they went to cache [23:01:18] RhinosF1: I'm actually not sure, they do go to cache, the ones in cache are different I think. You can try and I will test to make sure the rebuildVersionCache script still works (as much as it does because of firejail) [23:01:29] ok [23:01:44] if you can let's keep them only in cache [23:01:51] saves a big thing to exclude [23:02:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.83, 5.79, 6.74 [23:04:51] somehow magic is missing from staging [23:05:07] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBbCa [23:05:08] [02miraheze/RemovePII] 07Universal-Omega 03d1918b8 - Update RemovePIIJob.php [23:05:10] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [23:05:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.98, 7.72, 6.89 [23:05:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.21, 22.88, 18.84 [23:06:02] miraheze/RemovePII - Universal-Omega the build passed. [23:06:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.27, 7.69, 7.34 [23:07:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.81, 20.84, 18.58 [23:08:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.21, 7.50, 7.31 [23:09:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.61, 20.35, 18.69 [23:10:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.09, 8.39, 7.68 [23:12:29] PROBLEM - www.highdeologywiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.highdeologywiki.com' expires in 15 day(s) (Wed 18 Aug 2021 23:05:55 GMT +0000). [23:13:12] @Doug: I can barely even load meta [23:13:38] A rename is gonna be laggy [23:13:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.54, 20.60, 19.23 [23:15:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.39, 20.12, 19.22 [23:16:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBblY [23:16:30] [02miraheze/ssl] 07MirahezeSSLBot 030ad51bb - Bot: Update SSL cert for www.highdeologywiki.com [23:18:54] @R4356th: sorry for the job queue being slow on your rename [23:21:18] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 7.57, 6.17 [23:21:26] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBb8m [23:21:28] [02miraheze/RemovePII] 07Universal-Omega 031824a87 - Update RemovePIIJob.php [23:21:29] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [23:21:49] !log set off runJobs.php on all wikis [23:21:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:26] miraheze/RemovePII - Universal-Omega the build passed. [23:23:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 7.97, 7.97 [23:23:14] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.26, 7.61, 6.36 [23:24:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.63, 7.35, 7.99 [23:24:32] ping if needed [23:26:12] Universal_Omega: can you hold on setting off any more jobs please [23:26:54] Oh yeah sorry. [23:27:38] Universal_Omega: looks like GUP [23:28:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.56, 7.05, 7.63 [23:30:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.81, 6.21, 7.26 [23:31:01] only 43 unclaimed jobs in redis [23:31:01] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.55, 6.48, 6.44 [23:32:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.23, 5.35, 6.80 [23:32:37] Voidwalker: grafana is saying ~2k [23:32:50] it's slowly dropping [23:32:54] there's some other bug going on where jobs are being incorrectly entered as h-data, h-idBySha1, h-attempts, and a few others [23:33:02] looks like refreskLinks could be slow [23:33:07] grafana seems perpetually overreporting by about 2k [23:33:25] RECOVERY - www.highdeologywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.highdeologywiki.com' will expire on Sun 31 Oct 2021 22:16:20 GMT +0000. [23:34:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.39, 8.53, 7.80 [23:36:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.29, 8.09, 7.68 [23:36:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.67, 20.48, 18.81 [23:36:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.23, 7.99, 7.70 [23:36:58] Voidwalker: that's helpful [23:38:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.74, 7.46, 7.50 [23:38:30] https://meta.miraheze.org/wiki/Special:GlobalRenameProgress?username=Redmin is actually reporting a failed status [23:38:31] [url] Global rename progress - Miraheze Meta | meta.miraheze.org [23:38:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.55, 9.10, 8.09 [23:38:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.19, 20.25, 18.93 [23:38:47] https://afterlife.miraheze.org/wiki/ throws [ad206832bee66588010decdf] 2021-08-02 23:38:15: Fatal exception of type MediaWiki\Storage\NameTableAccessException [23:38:47] [url] Internal error - AfterLife GDR | afterlife.miraheze.org [23:39:06] we're down to 359 [23:39:10] Voidwalker: now it's run [23:39:38] i guess we need the silly inserts [23:40:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.46, 7.74, 7.59 [23:40:52] Voidwalker: try INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main') [23:42:54] hi dmehus [23:43:37] !log INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main'); -- afterlifewiki [23:43:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:09] still broke [23:44:26] * RhinosF1 opens graylog [23:44:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 7.86, 7.72 [23:45:05] hey RhinosF1 [23:45:05] [tell] dmehus: 2021-07-27 - 17:06:43UTC ask dmehus How are you today? [23:45:26] .tell darkmatterman450 pretty well thanks, you? [23:45:26] dmehus: I'll pass that on when darkmatterman450 is around. [23:46:09] Voidwalker: error says slot main is missing [23:46:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.68, 8.10, 7.81 [23:46:28] so not sure why that didn't fix [23:46:39] > SELECT * FROM slot_roles; [23:46:39] stdClass Object [23:46:39] ( [23:46:39] [role_id] => 1 [23:46:40] [role_name] => main [23:46:42] ) [23:47:02] ye so it's there [23:47:07] hrm, strange, yeah, that looks right (namespace ID and name). Weird. [23:47:37] I have to go for a few, so I'll leave this to someone else to figure out [23:47:37] nothing to do with namespaces [23:48:19] It's an italian wiki [23:48:31] would that matter in terms of the name we inserted? [23:48:43] no [23:48:48] oh ok [23:48:49] hrm [23:49:40] yeah, still getting `[608786db3b65684ee6ef5453] 2021-08-02 23:49:03: Fatal exception of type MediaWiki\Storage\NameTableAccessException` on `afterlifewiki`. Assuming you already know that, but pasting it nonetheless [23:49:45] yes [23:50:21] Universal_Omega: you can carry on testing [23:52:23] !log do the usual content model inserts on that wiki [23:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:14] hrm, very strange... that normally resolves that error, but this time no. [23:53:20] nope [23:54:55] !log [rhinos@jobrunner3] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=afterlifewiki --logwiki=metawiki R4356th Redmin (END - exit=256) [23:54:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:13] dmehus: fix script failed [23:56:03] i have 0 ideas [23:56:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.05, 7.28, 7.84 [23:58:25] i'm gonna nap [23:58:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.03, 8.89, 8.35 [23:58:34] hopefully i can dream a fix