[00:02:04] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5116 MB (21% inode=94%); [00:02:23] PROBLEM - cp12 Disk Space on cp12 is CRITICAL: DISK CRITICAL - free space: / 2288 MB (5% inode=96%); [00:03:48] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.50, 2.02, 1.80 [00:04:23] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 7288 MB (18% inode=96%); [00:05:23] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.28, 20.43, 16.35 [00:05:43] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.32, 1.74, 1.72 [00:07:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.05, 19.87, 16.70 [00:07:38] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.27, 1.60, 1.67 [00:10:42] [02mediawiki] 07dependabot[bot] opened pull request 03#2579: Bump skins/Citizen from `63d21cf` to `334e23d` - 13https://git.io/JRRwm [00:10:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Citizen-334e23d [+0/-0/±1] 13https://git.io/JRRwY [00:10:45] [02miraheze/mediawiki] 07dependabot[bot] 03f35b789 - Bump skins/Citizen from `63d21cf` to `334e23d` [00:10:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#2579: Bump skins/Citizen from `63d21cf` to `334e23d` - 13https://git.io/JRRwm [00:10:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#2579: Bump skins/Citizen from `63d21cf` to `334e23d` - 13https://git.io/JRRwm [00:10:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Citizen-334e23d - 13https://git.io/vbL5b [00:10:51] [02mediawiki] 07dependabot[bot] commented on pull request 03#2561: Bump skins/Citizen from `63d21cf` to `159198d` - 13https://git.io/JRRwO [00:10:53] [02mediawiki] 07dependabot[bot] closed pull request 03#2561: Bump skins/Citizen from `63d21cf` to `159198d` - 13https://git.io/JB93Z [00:10:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/BlogPage-36929b1 [+0/-0/±1] 13https://git.io/JRRw3 [00:10:56] [02miraheze/mediawiki] 07dependabot[bot] 03db8dc68 - Bump extensions/BlogPage from `2d7b581` to `36929b1` [00:10:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Citizen-159198d - 13https://git.io/vbL5b [00:10:59] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Citizen-159198d [00:11:00] [02mediawiki] 07dependabot[bot] opened pull request 03#2580: Bump extensions/BlogPage from `2d7b581` to `36929b1` - 13https://git.io/JRRws [00:11:02] [02mediawiki] 07dependabot[bot] labeled pull request 03#2580: Bump extensions/BlogPage from `2d7b581` to `36929b1` - 13https://git.io/JRRws [00:11:03] [02mediawiki] 07dependabot[bot] labeled pull request 03#2580: Bump extensions/BlogPage from `2d7b581` to `36929b1` - 13https://git.io/JRRws [00:11:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/BlogPage-36929b1 - 13https://git.io/vbL5b [00:11:06] [02mediawiki] 07dependabot[bot] commented on pull request 03#2556: Bump extensions/BlogPage from `2d7b581` to `ac66322` - 13https://git.io/JRRwG [00:11:08] [02mediawiki] 07dependabot[bot] closed pull request 03#2556: Bump extensions/BlogPage from `2d7b581` to `ac66322` - 13https://git.io/JB9Oj [00:11:09] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/BlogPage-ac66322 - 13https://git.io/vbL5b [00:11:11] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/BlogPage-ac66322 [00:11:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MintyDocs-99ba92d [+0/-0/±1] 13https://git.io/JRRwZ [00:11:14] [02miraheze/mediawiki] 07dependabot[bot] 035d36e4f - Bump extensions/MintyDocs from `5d6347c` to `99ba92d` [00:11:15] [02mediawiki] 07dependabot[bot] opened pull request 03#2581: Bump extensions/MintyDocs from `5d6347c` to `99ba92d` - 13https://git.io/JRRwn [00:11:16] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.39, 21.98, 18.12 [00:11:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MintyDocs-99ba92d - 13https://git.io/vbL5b [00:11:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#2581: Bump extensions/MintyDocs from `5d6347c` to `99ba92d` - 13https://git.io/JRRwn [00:11:20] [02mediawiki] 07dependabot[bot] labeled pull request 03#2581: Bump extensions/MintyDocs from `5d6347c` to `99ba92d` - 13https://git.io/JRRwn [00:11:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/Cosmos-47bf4c4 [+0/-0/±1] 13https://git.io/JRRwc [00:11:23] [02miraheze/mediawiki] 07dependabot[bot] 03a4d3804 - Bump skins/Cosmos from `c4668fc` to `47bf4c4` [00:11:24] [02mediawiki] 07dependabot[bot] opened pull request 03#2582: Bump skins/Cosmos from `c4668fc` to `47bf4c4` - 13https://git.io/JRRwC [00:11:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/Cosmos-47bf4c4 - 13https://git.io/vbL5b [00:11:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#2582: Bump skins/Cosmos from `c4668fc` to `47bf4c4` - 13https://git.io/JRRwC [00:11:29] [02mediawiki] 07dependabot[bot] labeled pull request 03#2582: Bump skins/Cosmos from `c4668fc` to `47bf4c4` - 13https://git.io/JRRwC [00:11:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageForms-64ed777 [+0/-0/±1] 13https://git.io/JRRwW [00:11:32] [02miraheze/mediawiki] 07dependabot[bot] 037cb2519 - Bump extensions/PageForms from `43d1777` to `64ed777` [00:11:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageForms-64ed777 - 13https://git.io/vbL5b [00:11:35] [02mediawiki] 07dependabot[bot] opened pull request 03#2583: Bump extensions/PageForms from `43d1777` to `64ed777` - 13https://git.io/JRRwl [00:11:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#2583: Bump extensions/PageForms from `43d1777` to `64ed777` - 13https://git.io/JRRwl [00:11:38] [02mediawiki] 07dependabot[bot] labeled pull request 03#2583: Bump extensions/PageForms from `43d1777` to `64ed777` - 13https://git.io/JRRwl [00:11:39] [02mediawiki] 07dependabot[bot] commented on pull request 03#2572: Bump extensions/PageForms from `43d1777` to `9f22337` - 13https://git.io/JRRw8 [00:11:41] [02mediawiki] 07dependabot[bot] closed pull request 03#2572: Bump extensions/PageForms from `43d1777` to `9f22337` - 13https://git.io/JB93d [00:11:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/snapwikiskin-329fe48 [+0/-0/±1] 13https://git.io/JRRwB [00:11:44] [02miraheze/mediawiki] 07dependabot[bot] 0343aca6c - Bump skins/snapwikiskin from `8c2bedd` to `329fe48` [00:11:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/snapwikiskin-329fe48 - 13https://git.io/vbL5b [00:11:47] [02mediawiki] 07dependabot[bot] opened pull request 03#2584: Bump skins/snapwikiskin from `8c2bedd` to `329fe48` - 13https://git.io/JRRwR [00:11:48] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/PageForms-9f22337 - 13https://git.io/vbL5b [00:11:50] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/PageForms-9f22337 [00:11:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#2584: Bump skins/snapwikiskin from `8c2bedd` to `329fe48` - 13https://git.io/JRRwR [00:11:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#2584: Bump skins/snapwikiskin from `8c2bedd` to `329fe48` - 13https://git.io/JRRwR [00:11:54] [02mediawiki] 07dependabot[bot] commented on pull request 03#2567: Bump skins/snapwikiskin from `8c2bedd` to `204bfcd` - 13https://git.io/JRRw0 [00:11:56] [02mediawiki] 07dependabot[bot] closed pull request 03#2567: Bump skins/snapwikiskin from `8c2bedd` to `204bfcd` - 13https://git.io/JB93a [00:11:57] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/snapwikiskin-204bfcd [00:11:59] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/snapwikiskin-204bfcd - 13https://git.io/vbL5b [00:12:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ManageWiki-ffa26f8 [+0/-0/±1] 13https://git.io/JRRwE [00:12:02] [02miraheze/mediawiki] 07dependabot[bot] 031fdbe9a - Bump extensions/ManageWiki from `b345e81` to `ffa26f8` [00:12:03] [02mediawiki] 07dependabot[bot] opened pull request 03#2585: Bump extensions/ManageWiki from `b345e81` to `ffa26f8` - 13https://git.io/JRRwu [00:12:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ManageWiki-ffa26f8 - 13https://git.io/vbL5b [00:12:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#2585: Bump extensions/ManageWiki from `b345e81` to `ffa26f8` - 13https://git.io/JRRwu [00:12:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#2585: Bump extensions/ManageWiki from `b345e81` to `ffa26f8` - 13https://git.io/JRRwu [00:12:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/skins/chameleon-2054484 [+0/-0/±1] 13https://git.io/JRRwg [00:12:11] [02miraheze/mediawiki] 07dependabot[bot] 03f7173c7 - Bump skins/chameleon from `074924a` to `2054484` [00:12:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/skins/chameleon-2054484 - 13https://git.io/vbL5b [00:12:14] [02mediawiki] 07dependabot[bot] opened pull request 03#2586: Bump skins/chameleon from `074924a` to `2054484` - 13https://git.io/JRRwa [00:12:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#2586: Bump skins/chameleon from `074924a` to `2054484` - 13https://git.io/JRRwa [00:12:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#2586: Bump skins/chameleon from `074924a` to `2054484` - 13https://git.io/JRRwa [00:12:18] [02mediawiki] 07dependabot[bot] commented on pull request 03#2455: Bump skins/chameleon from `074924a` to `32a78bd` - 13https://git.io/JRRwV [00:12:20] [02mediawiki] 07dependabot[bot] closed pull request 03#2455: Bump skins/chameleon from `074924a` to `32a78bd` - 13https://git.io/J4aLM [00:12:21] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/chameleon-32a78bd [00:12:23] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/chameleon-32a78bd - 13https://git.io/vbL5b [00:12:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Translate-c15b3d3 [+0/-0/±1] 13https://git.io/JRRwr [00:12:26] [02miraheze/mediawiki] 07dependabot[bot] 031351811 - Bump extensions/Translate from `a4e5804` to `c15b3d3` [00:12:27] [02mediawiki] 07dependabot[bot] opened pull request 03#2587: Bump extensions/Translate from `a4e5804` to `c15b3d3` - 13https://git.io/JRRwo [00:12:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Translate-c15b3d3 - 13https://git.io/vbL5b [00:12:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#2587: Bump extensions/Translate from `a4e5804` to `c15b3d3` - 13https://git.io/JRRwo [00:12:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#2587: Bump extensions/Translate from `a4e5804` to `c15b3d3` - 13https://git.io/JRRwo [00:12:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#2560: Bump extensions/Translate from `a4e5804` to `1680211` - 13https://git.io/JRRwK [00:12:35] [02mediawiki] 07dependabot[bot] closed pull request 03#2560: Bump extensions/Translate from `a4e5804` to `1680211` - 13https://git.io/JB93s [00:12:36] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Translate-1680211 [00:12:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/WikiDiscover-5cec28f [+0/-0/±1] 13https://git.io/JRRw6 [00:12:39] [02miraheze/mediawiki] 07dependabot[bot] 03d43b9c9 - Bump extensions/WikiDiscover from `3bc6b10` to `5cec28f` [00:12:41] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Translate-1680211 - 13https://git.io/vbL5b [00:12:42] [02mediawiki] 07dependabot[bot] opened pull request 03#2588: Bump extensions/WikiDiscover from `3bc6b10` to `5cec28f` - 13https://git.io/JRRwi [00:12:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/WikiDiscover-5cec28f - 13https://git.io/vbL5b [00:12:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#2588: Bump extensions/WikiDiscover from `3bc6b10` to `5cec28f` - 13https://git.io/JRRwi [00:12:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#2588: Bump extensions/WikiDiscover from `3bc6b10` to `5cec28f` - 13https://git.io/JRRwi [00:12:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-1c3d31c [+0/-0/±1] 13https://git.io/JRRwP [00:12:50] [02miraheze/mediawiki] 07dependabot[bot] 0398f5984 - Bump extensions/UniversalLanguageSelector from `03d099e` to `1c3d31c` [00:12:51] [02mediawiki] 07dependabot[bot] opened pull request 03#2589: Bump extensions/UniversalLanguageSelector from `03d099e` to `1c3d31c` - 13https://git.io/JRRwX [00:12:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-1c3d31c - 13https://git.io/vbL5b [00:12:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#2589: Bump extensions/UniversalLanguageSelector from `03d099e` to `1c3d31c` - 13https://git.io/JRRwX [00:12:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#2589: Bump extensions/UniversalLanguageSelector from `03d099e` to `1c3d31c` - 13https://git.io/JRRwX [00:12:57] [02mediawiki] 07dependabot[bot] commented on pull request 03#2566: Bump extensions/UniversalLanguageSelector from `03d099e` to `ef99ad8` - 13https://git.io/JRRw1 [00:12:59] [02mediawiki] 07dependabot[bot] closed pull request 03#2566: Bump extensions/UniversalLanguageSelector from `03d099e` to `ef99ad8` - 13https://git.io/JB93u [00:13:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/CreateWiki-4343989 [+0/-0/±1] 13https://git.io/JRRwM [00:13:02] [02miraheze/mediawiki] 07dependabot[bot] 03338cbe2 - Bump extensions/CreateWiki from `1123fb8` to `4343989` [00:13:03] [02mediawiki] 07dependabot[bot] opened pull request 03#2590: Bump extensions/CreateWiki from `1123fb8` to `4343989` - 13https://git.io/JRRwD [00:13:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-4343989 - 13https://git.io/vbL5b [00:13:06] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-ef99ad8 [00:13:08] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/UniversalLanguageSelector-ef99ad8 - 13https://git.io/vbL5b [00:13:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#2590: Bump extensions/CreateWiki from `1123fb8` to `4343989` - 13https://git.io/JRRwD [00:13:11] [02mediawiki] 07dependabot[bot] labeled pull request 03#2590: Bump extensions/CreateWiki from `1123fb8` to `4343989` - 13https://git.io/JRRwD [00:13:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-60889b4 [+0/-0/±1] 13https://git.io/JRRwS [00:13:14] [02miraheze/mediawiki] 07dependabot[bot] 03e19cca2 - Bump extensions/MirahezeMagic from `0563225` to `60889b4` [00:13:15] [02mediawiki] 07dependabot[bot] opened pull request 03#2591: Bump extensions/MirahezeMagic from `0563225` to `60889b4` - 13https://git.io/JRRw9 [00:13:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MirahezeMagic-60889b4 - 13https://git.io/vbL5b [00:13:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#2591: Bump extensions/MirahezeMagic from `0563225` to `60889b4` - 13https://git.io/JRRw9 [00:13:20] [02mediawiki] 07dependabot[bot] labeled pull request 03#2591: Bump extensions/MirahezeMagic from `0563225` to `60889b4` - 13https://git.io/JRRw9 [00:13:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Comments-bc9f2b7 [+0/-0/±1] 13https://git.io/JRRwQ [00:13:23] [02miraheze/mediawiki] 07dependabot[bot] 0378441dc - Bump extensions/Comments from `ff88c27` to `bc9f2b7` [00:13:24] [02mediawiki] 07dependabot[bot] opened pull request 03#2592: Bump extensions/Comments from `ff88c27` to `bc9f2b7` - 13https://git.io/JRRw7 [00:13:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Comments-bc9f2b7 - 13https://git.io/vbL5b [00:13:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#2592: Bump extensions/Comments from `ff88c27` to `bc9f2b7` - 13https://git.io/JRRw7 [00:13:29] [02mediawiki] 07dependabot[bot] labeled pull request 03#2592: Bump extensions/Comments from `ff88c27` to `bc9f2b7` - 13https://git.io/JRRw7 [00:13:30] [02mediawiki] 07dependabot[bot] commented on pull request 03#2569: Bump extensions/Comments from `ff88c27` to `14bdc1f` - 13https://git.io/JRRw5 [00:13:32] [02mediawiki] 07dependabot[bot] closed pull request 03#2569: Bump extensions/Comments from `ff88c27` to `14bdc1f` - 13https://git.io/JB93K [00:13:33] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Comments-14bdc1f - 13https://git.io/vbL5b [00:13:35] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Comments-14bdc1f [00:13:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/RemovePII-5c90a7b [+0/-0/±1] 13https://git.io/JRRwd [00:13:38] [02miraheze/mediawiki] 07dependabot[bot] 039652b42 - Bump extensions/RemovePII from `7ac312d` to `5c90a7b` [00:13:39] [02mediawiki] 07dependabot[bot] opened pull request 03#2593: Bump extensions/RemovePII from `7ac312d` to `5c90a7b` - 13https://git.io/JRRwF [00:13:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/RemovePII-5c90a7b - 13https://git.io/vbL5b [00:13:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#2593: Bump extensions/RemovePII from `7ac312d` to `5c90a7b` - 13https://git.io/JRRwF [00:13:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#2593: Bump extensions/RemovePII from `7ac312d` to `5c90a7b` - 13https://git.io/JRRwF [00:13:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-5daa77e [+0/-0/±1] 13https://git.io/JRRwb [00:13:47] [02miraheze/mediawiki] 07dependabot[bot] 038c21169 - Bump extensions/ApprovedRevs from `2579e8f` to `5daa77e` [00:13:48] [02mediawiki] 07dependabot[bot] opened pull request 03#2594: Bump extensions/ApprovedRevs from `2579e8f` to `5daa77e` - 13https://git.io/JRRwN [00:13:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-5daa77e - 13https://git.io/vbL5b [00:13:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#2594: Bump extensions/ApprovedRevs from `2579e8f` to `5daa77e` - 13https://git.io/JRRwN [00:13:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#2594: Bump extensions/ApprovedRevs from `2579e8f` to `5daa77e` - 13https://git.io/JRRwN [00:13:54] [02mediawiki] 07dependabot[bot] commented on pull request 03#2575: Bump extensions/ApprovedRevs from `2579e8f` to `e4bebc4` - 13https://git.io/JRRwx [00:13:56] [02mediawiki] 07dependabot[bot] closed pull request 03#2575: Bump extensions/ApprovedRevs from `2579e8f` to `e4bebc4` - 13https://git.io/JB93h [00:13:57] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-e4bebc4 [00:13:59] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/ApprovedRevs-e4bebc4 - 13https://git.io/vbL5b [00:14:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/MatomoAnalytics-3d62a38 [+0/-0/±1] 13https://git.io/JRRwj [00:14:02] [02miraheze/mediawiki] 07dependabot[bot] 03a48fb2e - Bump extensions/MatomoAnalytics from `f9f715b` to `3d62a38` [00:14:03] [02mediawiki] 07dependabot[bot] opened pull request 03#2595: Bump extensions/MatomoAnalytics from `f9f715b` to `3d62a38` - 13https://git.io/JRRre [00:14:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/MatomoAnalytics-3d62a38 - 13https://git.io/vbL5b [00:14:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#2595: Bump extensions/MatomoAnalytics from `f9f715b` to `3d62a38` - 13https://git.io/JRRre [00:14:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#2595: Bump extensions/MatomoAnalytics from `f9f715b` to `3d62a38` - 13https://git.io/JRRre [00:14:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/WikiForum-b9b6abc [+0/-0/±1] 13https://git.io/JRRrI [00:14:11] [02miraheze/mediawiki] 07dependabot[bot] 03868b75e - Bump extensions/WikiForum from `0cf5ac4` to `b9b6abc` [00:14:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/WikiForum-b9b6abc - 13https://git.io/vbL5b [00:14:14] [02mediawiki] 07dependabot[bot] opened pull request 03#2596: Bump extensions/WikiForum from `0cf5ac4` to `b9b6abc` - 13https://git.io/JRRrL [00:14:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#2596: Bump extensions/WikiForum from `0cf5ac4` to `b9b6abc` - 13https://git.io/JRRrL [00:14:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#2596: Bump extensions/WikiForum from `0cf5ac4` to `b9b6abc` - 13https://git.io/JRRrL [00:14:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PageSchemas-fabc1a5 [+0/-0/±1] 13https://git.io/JRRrt [00:14:20] [02miraheze/mediawiki] 07dependabot[bot] 0336f91b9 - Bump extensions/PageSchemas from `925ce7f` to `fabc1a5` [00:14:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PageSchemas-fabc1a5 - 13https://git.io/vbL5b [00:14:23] [02mediawiki] 07dependabot[bot] opened pull request 03#2597: Bump extensions/PageSchemas from `925ce7f` to `fabc1a5` - 13https://git.io/JRRrq [00:14:24] [02mediawiki] 07dependabot[bot] labeled pull request 03#2597: Bump extensions/PageSchemas from `925ce7f` to `fabc1a5` - 13https://git.io/JRRrq [00:14:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#2597: Bump extensions/PageSchemas from `925ce7f` to `fabc1a5` - 13https://git.io/JRRrq [00:14:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/WikibaseLocalMedia-5bad136 [+0/-0/±1] 13https://git.io/JRRrY [00:14:29] [02miraheze/mediawiki] 07dependabot[bot] 03306f2d3 - Bump extensions/WikibaseLocalMedia from `6d5d118` to `5bad136` [00:14:30] [02mediawiki] 07dependabot[bot] opened pull request 03#2598: Bump extensions/WikibaseLocalMedia from `6d5d118` to `5bad136` - 13https://git.io/JRRrO [00:14:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/WikibaseLocalMedia-5bad136 - 13https://git.io/vbL5b [00:14:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#2598: Bump extensions/WikibaseLocalMedia from `6d5d118` to `5bad136` - 13https://git.io/JRRrO [00:14:35] [02mediawiki] 07dependabot[bot] labeled pull request 03#2598: Bump extensions/WikibaseLocalMedia from `6d5d118` to `5bad136` - 13https://git.io/JRRrO [00:14:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/PollNY-dc5362f [+0/-0/±1] 13https://git.io/JRRrs [00:14:38] [02miraheze/mediawiki] 07dependabot[bot] 0310d7adc - Bump extensions/PollNY from `2282f32` to `dc5362f` [00:14:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/PollNY-dc5362f - 13https://git.io/vbL5b [00:14:41] [02mediawiki] 07dependabot[bot] opened pull request 03#2599: Bump extensions/PollNY from `2282f32` to `dc5362f` - 13https://git.io/JRRrG [00:14:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#2599: Bump extensions/PollNY from `2282f32` to `dc5362f` - 13https://git.io/JRRrG [00:14:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#2599: Bump extensions/PollNY from `2282f32` to `dc5362f` - 13https://git.io/JRRrG [00:14:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Cargo-ad0ba83 [+0/-0/±1] 13https://git.io/JRRrZ [00:14:47] [02miraheze/mediawiki] 07dependabot[bot] 03f3c1635 - Bump extensions/Cargo from `336ec47` to `ad0ba83` [00:14:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Cargo-ad0ba83 - 13https://git.io/vbL5b [00:14:50] [02mediawiki] 07dependabot[bot] opened pull request 03#2600: Bump extensions/Cargo from `336ec47` to `ad0ba83` - 13https://git.io/JRRrn [00:14:51] [02mediawiki] 07dependabot[bot] commented on pull request 03#2568: Bump extensions/PollNY from `2282f32` to `f91bb0b` - 13https://git.io/JRRrc [00:14:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#2600: Bump extensions/Cargo from `336ec47` to `ad0ba83` - 13https://git.io/JRRrn [00:14:55] [02mediawiki] 07dependabot[bot] labeled pull request 03#2600: Bump extensions/Cargo from `336ec47` to `ad0ba83` - 13https://git.io/JRRrn [00:14:56] [02mediawiki] 07dependabot[bot] closed pull request 03#2568: Bump extensions/PollNY from `2282f32` to `f91bb0b` - 13https://git.io/JB93w [00:14:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/PollNY-f91bb0b - 13https://git.io/vbL5b [00:14:59] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/PollNY-f91bb0b [00:15:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#2563: Bump extensions/Cargo from `336ec47` to `c86a46e` - 13https://git.io/JRRrC [00:15:02] [02mediawiki] 07dependabot[bot] closed pull request 03#2563: Bump extensions/Cargo from `336ec47` to `c86a46e` - 13https://git.io/JB938 [00:15:03] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Cargo-c86a46e - 13https://git.io/vbL5b [00:15:05] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Cargo-c86a46e [00:15:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_36/extensions/Maps-5716264 [+0/-0/±1] 13https://git.io/JRRrW [00:15:08] [02miraheze/mediawiki] 07dependabot[bot] 033aafddd - Bump extensions/Maps from `b43865b` to `5716264` [00:15:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_36/extensions/Maps-5716264 - 13https://git.io/vbL5b [00:15:11] [02mediawiki] 07dependabot[bot] opened pull request 03#2601: Bump extensions/Maps from `b43865b` to `5716264` - 13https://git.io/JRRrl [00:15:13] [02mediawiki] 07dependabot[bot] labeled pull request 03#2601: Bump extensions/Maps from `b43865b` to `5716264` - 13https://git.io/JRRrl [00:15:14] [02mediawiki] 07dependabot[bot] labeled pull request 03#2601: Bump extensions/Maps from `b43865b` to `5716264` - 13https://git.io/JRRrl [00:15:15] [02mediawiki] 07dependabot[bot] commented on pull request 03#2555: Bump extensions/Maps from `b43865b` to `61105dc` - 13https://git.io/JRRr8 [00:15:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.35, 23.13, 19.56 [00:15:17] [02mediawiki] 07dependabot[bot] closed pull request 03#2555: Bump extensions/Maps from `b43865b` to `61105dc` - 13https://git.io/JB9ON [00:15:19] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Maps-61105dc [00:15:20] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/Maps-61105dc - 13https://git.io/vbL5b [00:19:16] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.14, 19.54, 18.93 [00:34:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.25, 2.80 [00:35:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.78, 7.62, 5.86 [00:36:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.28, 3.15, 2.81 [00:36:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.06, 6.78, 5.70 [00:37:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.27, 7.15, 5.91 [00:38:30] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 6.31, 5.67 [00:39:09] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.33, 6.26, 5.74 [00:55:13] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.37, 2.90, 2.04 [00:56:21] [02mediawiki] 07Universal-Omega closed pull request 03#2582: Bump skins/Cosmos from `c4668fc` to `47bf4c4` - 13https://git.io/JRRwC [00:56:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JRRX0 [00:56:24] [02miraheze/mediawiki] 07dependabot[bot] 034dd05f9 - Bump skins/Cosmos from `c4668fc` to `47bf4c4` (#2582) [00:56:25] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/skins/Cosmos-47bf4c4 - 13https://git.io/vbL5b [00:56:27] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/skins/Cosmos-47bf4c4 [01:03:16] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.89, 1.87, 1.84 [01:05:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 5.73, 5.10 [01:07:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.41, 5.26, 5.00 [01:07:16] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.92, 2.55, 2.10 [01:12:20] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:12:38] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:15:14] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.74, 1.57, 1.82 [01:19:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.91, 1.25, 1.63 [01:34:20] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [01:34:38] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [01:34:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.21, 6.15, 4.87 [01:36:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.76, 5.45, 4.76 [01:45:14] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.01, 1.72, 1.46 [01:47:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.79, 1.38, 1.37 [02:28:07] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.25, 1.85, 1.43 [02:30:03] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.28, 1.65, 1.42 [03:15:07] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [03:15:08] ackages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query diamowiki.ga. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed [03:15:08] ; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [03:20:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.09, 2.79 [03:22:03] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [03:22:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 2.74, 2.70 [03:44:28] [02mediawiki] 07Universal-Omega closed pull request 03#2584: Bump skins/snapwikiskin from `8c2bedd` to `329fe48` - 13https://git.io/JRRwR [03:44:30] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JR0sy [03:44:31] [02miraheze/mediawiki] 07dependabot[bot] 03c88bec1 - Bump skins/snapwikiskin from `8c2bedd` to `329fe48` (#2584) [03:44:33] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/skins/snapwikiskin-329fe48 - 13https://git.io/vbL5b [03:44:34] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/skins/snapwikiskin-329fe48 [03:45:01] [02mediawiki] 07Universal-Omega closed pull request 03#2595: Bump extensions/MatomoAnalytics from `f9f715b` to `3d62a38` - 13https://git.io/JRRre [03:45:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JR0s7 [03:45:04] [02miraheze/mediawiki] 07dependabot[bot] 038874527 - Bump extensions/MatomoAnalytics from `f9f715b` to `3d62a38` (#2595) [03:45:06] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/MatomoAnalytics-3d62a38 [03:45:07] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/MatomoAnalytics-3d62a38 - 13https://git.io/vbL5b [04:58:07] PROBLEM - phab.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.bots.miraheze.wiki' expires in 7 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [04:58:11] PROBLEM - phab-storage.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.bots.miraheze.wiki' expires in 7 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [04:59:26] PROBLEM - phabdigests.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.bots.miraheze.wiki' expires in 7 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [04:59:32] PROBLEM - sopel.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.bots.miraheze.wiki' expires in 7 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [05:00:19] PROBLEM - wildcard.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.bots.miraheze.wiki' expires in 7 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [05:00:48] PROBLEM - icinga.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.bots.miraheze.wiki' expires in 7 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [05:35:13] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.42, 1.83, 1.38 [05:35:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.49, 21.52, 15.65 [05:37:13] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.31, 1.63, 1.36 [05:37:31] PROBLEM - halloween.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for halloween.gyaanipedia.com could not be found [05:37:33] PROBLEM - www.puzzles.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.puzzles.wiki could not be found [05:37:41] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.landev.vn could not be found [05:38:22] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.32, 6.96, 4.04 [05:38:22] PROBLEM - nenawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nenawiki.org could not be found [05:38:43] PROBLEM - beaconspace.unrestrictedlorefare.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for beaconspace.unrestrictedlorefare.com could not be found [05:41:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.62, 22.09, 18.21 [05:42:13] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.17, 5.17, 3.99 [05:43:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.95, 23.22, 19.04 [05:44:17] RECOVERY - halloween.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - halloween.gyaanipedia.com reverse DNS resolves to cp12.miraheze.org - CNAME OK [05:44:19] RECOVERY - www.puzzles.wiki - reverse DNS on sslhost is OK: SSL OK - www.puzzles.wiki reverse DNS resolves to cp12.miraheze.org - NS RECORDS OK [05:44:34] RECOVERY - wiki.landev.vn - reverse DNS on sslhost is OK: SSL OK - wiki.landev.vn reverse DNS resolves to cp12.miraheze.org - CNAME OK [05:45:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.41, 22.46, 19.21 [05:45:19] RECOVERY - nenawiki.org - reverse DNS on sslhost is OK: SSL OK - nenawiki.org reverse DNS resolves to cp12.miraheze.org - NS RECORDS OK [05:45:42] RECOVERY - beaconspace.unrestrictedlorefare.com - reverse DNS on sslhost is OK: SSL OK - beaconspace.unrestrictedlorefare.com reverse DNS resolves to cp12.miraheze.org - CNAME OK [05:47:16] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.68, 19.15, 18.39 [05:48:06] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.53, 2.04, 1.61 [05:48:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.96, 4.52, 4.20 [05:50:02] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.11, 1.66, 1.52 [05:51:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.47, 20.35, 18.95 [05:53:20] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.50, 24.25, 20.60 [05:53:55] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.10, 2.46, 1.86 [05:54:15] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.19, 5.75, 4.78 [05:54:36] PROBLEM - wiki.creaturathegame.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.creaturathegame.com could not be found [05:55:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.02, 23.42, 20.71 [05:56:14] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.28, 5.05, 4.66 [05:57:12] PROBLEM - mw8 Disk Space on mw8 is WARNING: DISK WARNING - free space: / 2049 MB (10% inode=72%); [05:59:10] RECOVERY - mw8 Disk Space on mw8 is OK: DISK OK - free space: / 3073 MB (16% inode=72%); [05:59:16] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.13, 20.12, 20.01 [05:59:41] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.19, 1.92, 1.84 [06:01:26] RECOVERY - wiki.creaturathegame.com - reverse DNS on sslhost is OK: SSL OK - wiki.creaturathegame.com reverse DNS resolves to cp12.miraheze.org - CNAME OK [06:03:12] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [06:03:34] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.74, 1.28, 1.59 [06:05:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.29, 20.87, 20.14 [06:09:16] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.07, 19.27, 19.61 [06:09:21] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.94, 1.83, 1.74 [06:10:28] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [06:11:16] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.14, 2.03, 1.83 [06:11:48] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JREeU [06:11:50] [02miraheze/dns] 07Reception123 03b2355f6 - add rww2.org zone [06:12:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.13, 6.01, 4.61 [06:13:51] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JREeE [06:13:52] [02miraheze/dns] 07Reception123 03790c4bf - add otcg.ml zone [06:14:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.27, 5.42, 4.57 [06:19:14] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.28, 1.80, 1.85 [06:20:28] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/JREfq [06:20:29] [02miraheze/ssl] 07Reception123 03d517fa6 - add www.chinatech.wiki cert [06:20:31] [02miraheze/ssl] 07Reception123 0319c20bb - add otcg.ml cert [06:20:32] [02miraheze/ssl] 07Reception123 03f679251 - add rww2.org cert [06:25:15] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.53, 1.16, 1.57 [06:29:13] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.37, 2.15, 1.90 [06:30:18] PROBLEM - mw9 Disk Space on mw9 is WARNING: DISK WARNING - free space: / 1231 MB (6% inode=73%); [06:32:17] RECOVERY - mw9 Disk Space on mw9 is OK: DISK OK - free space: / 2255 MB (11% inode=73%); [06:32:37] PROBLEM - ping4 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 104.17 ms [06:34:16] PROBLEM - mw10 Disk Space on mw10 is WARNING: DISK WARNING - free space: / 1724 MB (9% inode=72%); [06:34:40] RECOVERY - ping4 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 81.74 ms [06:40:00] RECOVERY - mw10 Disk Space on mw10 is OK: DISK OK - free space: / 2748 MB (14% inode=72%); [06:43:14] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.42, 1.94, 2.00 [06:43:54] PROBLEM - mw8 Disk Space on mw8 is WARNING: DISK WARNING - free space: / 2071 MB (10% inode=72%); [06:45:57] RECOVERY - mw8 Disk Space on mw8 is OK: DISK OK - free space: / 3014 MB (15% inode=72%); [06:45:59] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [06:45:59] ckages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.003986835479736 seconds [06:46:14] PROBLEM - mw9 Disk Space on mw9 is WARNING: DISK WARNING - free space: / 1230 MB (6% inode=73%); [06:47:14] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.97, 2.48, 2.17 [06:48:13] RECOVERY - mw9 Disk Space on mw9 is OK: DISK OK - free space: / 2254 MB (11% inode=73%); [06:50:59] PROBLEM - ping6 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 114.60 ms [06:51:41] PROBLEM - mw10 Disk Space on mw10 is WARNING: DISK WARNING - free space: / 1724 MB (9% inode=72%); [06:53:11] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [06:53:43] RECOVERY - mw10 Disk Space on mw10 is OK: DISK OK - free space: / 2748 MB (14% inode=72%); [06:54:23] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.79, 4.96, 3.95 [06:54:24] PROBLEM - mrlove.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mrlove.wiki could not be found [06:54:29] PROBLEM - wikidiffs.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikidiffs.ga could not be found [06:54:59] RECOVERY - ping6 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 81.49 ms [06:55:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.43, 21.37, 18.77 [06:56:09] PROBLEM - mw9 Disk Space on mw9 is WARNING: DISK WARNING - free space: / 1230 MB (6% inode=73%); [06:56:22] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.52, 4.98, 4.09 [06:57:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.45, 19.90, 18.57 [06:58:08] RECOVERY - mw9 Disk Space on mw9 is OK: DISK OK - free space: / 2254 MB (11% inode=73%); [06:59:07] PROBLEM - ping6 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 103.28 ms [07:01:06] RECOVERY - mrlove.wiki - reverse DNS on sslhost is OK: SSL OK - mrlove.wiki reverse DNS resolves to cp12.miraheze.org - NS RECORDS OK [07:01:19] RECOVERY - wikidiffs.ga - reverse DNS on sslhost is OK: SSL OK - wikidiffs.ga reverse DNS resolves to cp12.miraheze.org - CNAME FLAT [07:03:06] RECOVERY - ping6 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 86.24 ms [07:04:02] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [07:04:02] ackages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.0016508102417 seconds [07:04:04] PROBLEM - mw9 Disk Space on mw9 is WARNING: DISK WARNING - free space: / 1230 MB (6% inode=73%); [07:06:02] RECOVERY - mw9 Disk Space on mw9 is OK: DISK OK - free space: / 2254 MB (11% inode=73%); [07:14:41] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.90, 1.25, 1.93 [07:18:39] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.98, 1.94, 2.03 [07:20:39] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.82, 1.70, 1.92 [07:22:39] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.37, 1.78, 1.92 [07:26:39] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.00, 1.76, 1.90 [07:32:41] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.53, 1.06, 1.56 [07:39:44] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:47:06] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [07:47:06] ackages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query diamowiki.ga. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed [07:47:06] ; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [08:01:36] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [08:05:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JREaO [08:05:13] [02miraheze/services] 07MirahezeSSLBot 03c565a5d - BOT: Updating services config for wikis [08:11:32] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [08:18:45] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, [08:18:46] 0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.003917455673218 seconds [08:33:17] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [08:40:44] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [08:40:44] ckages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query diamowiki.ga. IN CNAME: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation tim [08:40:44] t.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [08:55:52] PROBLEM - diamowiki.ga - reverse DNS on sslhost is UNKNOWN: [08:58:42] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 1.96, 2.15, 1.56 [09:03:08] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [09:03:08] ckages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query diamowiki.ga. IN CNAME: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [09:08:40] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.78, 1.82, 1.81 [09:10:44] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.75, 1.45, 1.67 [09:17:57] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [09:24:55] PROBLEM - ping4 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 109.41 ms [09:25:30] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [09:25:30] ckages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query diamowiki.ga. IN CNAME: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation tim [09:25:30] t.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [09:26:37] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.84, 2.95, 2.19 [09:26:58] RECOVERY - ping4 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 95.72 ms [09:29:30] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.12, 21.58, 17.19 [09:31:29] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.68, 20.28, 17.26 [09:36:23] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.83, 1.69, 1.97 [09:40:12] PROBLEM - diamowiki.ga - reverse DNS on sslhost is UNKNOWN: [09:40:19] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.96, 2.27, 2.12 [09:42:16] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.94, 1.72, 1.93 [09:46:11] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.80, 1.70, 1.84 [09:47:30] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [09:47:31] ackages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.002935647964478 seconds [09:53:24] PROBLEM - ping6 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 108.83 ms [09:54:26] PROBLEM - mw10 Disk Space on mw10 is WARNING: DISK WARNING - free space: / 1721 MB (9% inode=72%); [09:55:23] RECOVERY - ping6 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 74.96 ms [09:56:22] RECOVERY - mw10 Disk Space on mw10 is OK: DISK OK - free space: / 2745 MB (14% inode=72%); [09:59:58] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.94, 1.63, 1.91 [10:04:44] PROBLEM - mw9 Disk Space on mw9 is WARNING: DISK WARNING - free space: / 1807 MB (9% inode=73%); [10:07:49] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.39, 1.83, 1.83 [10:08:27] PROBLEM - ping6 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 118.33 ms [10:10:27] RECOVERY - ping6 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 98.06 ms [10:10:44] RECOVERY - mw9 Disk Space on mw9 is OK: DISK OK - free space: / 2251 MB (11% inode=73%); [10:15:46] PROBLEM - mw10 Disk Space on mw10 is WARNING: DISK WARNING - free space: / 1721 MB (9% inode=72%); [10:17:30] !log sudo rm -rf /tmp/* on mw* [10:17:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:42] RECOVERY - mw10 Disk Space on mw10 is OK: DISK OK - free space: / 5889 MB (31% inode=72%); [10:19:32] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.41, 1.82, 1.92 [10:21:29] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.17, 1.91, 1.94 [10:29:27] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.22, 1.71, 1.90 [10:31:24] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.44, 2.08, 2.01 [10:31:27] PROBLEM - ping4 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 112.07 ms [10:31:32] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [10:33:21] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.67, 1.86, 1.93 [10:33:30] RECOVERY - ping4 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 82.08 ms [10:35:18] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.00, 2.15, 2.02 [10:39:12] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.31, 1.81, 1.92 [10:41:39] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [10:41:39] ackages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.00082564353943 seconds [10:45:05] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.93, 1.35, 1.67 [11:03:35] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [11:08:46] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRuzG [11:08:47] [02miraheze/landing] 07translatewiki 034c4ab04 - Localisation updates from https://translatewiki.net. [11:08:48] [url] Main page - translatewiki.net | translatewiki.net [11:08:49] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±7] 13https://git.io/JRuzn [11:08:50] [02miraheze/CreateWiki] 07translatewiki 0333a7b14 - Localisation updates from https://translatewiki.net. [11:08:51] [url] Main page - translatewiki.net | translatewiki.net [11:08:52] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JRuzc [11:08:53] [02miraheze/ManageWiki] 07translatewiki 0348226fc - Localisation updates from https://translatewiki.net. [11:08:54] [url] Main page - translatewiki.net | translatewiki.net [11:08:55] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRuz8 [11:08:56] [02miraheze/MirahezeMagic] 07translatewiki 03e172e84 - Localisation updates from https://translatewiki.net. [11:08:57] [url] Main page - translatewiki.net | translatewiki.net [11:08:58] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JRuz4 [11:08:59] [02miraheze/WikiDiscover] 07translatewiki 035f71ded - Localisation updates from https://translatewiki.net. [11:09:00] [url] Main page - translatewiki.net | translatewiki.net [11:09:41] miraheze/landing - translatewiki the build passed. [11:09:47] miraheze/CreateWiki - translatewiki the build passed. [11:09:54] miraheze/ManageWiki - translatewiki the build passed. [11:09:56] miraheze/MirahezeMagic - translatewiki the build passed. [11:10:03] miraheze/WikiDiscover - translatewiki the build passed. [11:13:53] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [11:13:53] ackages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.00344443321228 seconds [11:16:39] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.55, 2.27, 1.68 [11:23:11] PROBLEM - ping6 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 111.00 ms [11:24:11] PROBLEM - ping4 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 106.08 ms [11:27:09] RECOVERY - ping6 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 74.55 ms [11:28:22] RECOVERY - ping4 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 78.18 ms [11:28:44] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.20, 1.89, 1.89 [11:30:42] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.35, 1.96, 1.91 [11:56:39] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.20, 1.68, 1.98 [11:58:41] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.35, 1.83, 1.99 [12:00:35] PROBLEM - ping4 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 118.71 ms [12:02:38] RECOVERY - ping4 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 77.86 ms [12:02:40] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.39, 1.75, 1.93 [12:08:39] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.07, 1.38, 1.70 [12:22:40] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 4.16, 2.50, 1.86 [12:23:01] PROBLEM - wikimoma.art - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikimoma.art' expires in 15 day(s) (Wed 25 Aug 2021 12:17:35 GMT +0000). [12:28:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRuQQ [12:28:09] [02miraheze/ssl] 07MirahezeSSLBot 03b308f88 - Bot: Update SSL cert for wikimoma.art [12:29:31] PROBLEM - ping6 on cp12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 107.57 ms [12:31:33] RECOVERY - ping6 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 75.15 ms [12:33:36] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4221 MB (10% inode=96%); [12:36:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.59, 19.40, 17.39 [12:36:39] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.69, 1.60, 2.00 [12:38:11] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.30, 19.44, 17.71 [12:41:57] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [12:44:40] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.99, 1.13, 1.62 [12:50:46] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.11, 2.23, 1.98 [12:52:44] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.70, 2.00, 1.92 [12:56:52] RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Sun 07 Nov 2021 11:27:59 GMT +0000. [13:01:56] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [13:01:56] ckages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.003808736801147 seconds [13:02:40] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.96, 1.35, 1.63 [13:22:33] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.30, 1.79, 1.61 [13:24:32] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.54, 1.72, 1.61 [13:26:29] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.93, 1.49, 1.54 [13:31:13] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [13:41:47] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [13:41:47] ckages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query diamowiki.ga. IN CNAME: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [13:54:00] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.07, 1.82, 1.56 [13:56:05] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp13.miraheze.org - CNAME OK [13:57:56] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.94, 1.93, 1.66 [13:59:52] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.02, 1.64, 1.58 [14:01:51] Reception123: are we aware 1 cp is down and 1 running out of disk [14:01:56] .op [14:01:56] Attempting to OP... [14:06:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [14:06:34] ckages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query diamowiki.ga. IN CNAME: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [14:07:13] If anyone wonders, Africa is shifting towards Singapore. EU & Default towards Canada [14:07:40] * RhinosF1 wonders if that could be an issue as Americans / Asians wake up [14:09:54] Looking at load we could be in trouble if we get much more load on Canada [14:11:02] RhinosF1: let me look now [14:11:18] Reception123: fix is on task [14:11:29] and heh you made the same cp13 typo as me [14:11:44] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.82, 1.70, 1.60 [14:11:55] Reception123: /usr/bin/rm -rf /var/lib/varnish/* [14:11:58] No cp13 [14:12:24] GB caching OOM'd overnight [14:12:27] Not Asia [14:12:32] ah so this time it's actually cp13 [14:12:47] Yes [14:13:01] !log sudo rm -rf /var/lib/varnish/* on cp13 [14:13:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:13:19] Reception123: hopefully if you restart varnish it'll come back [14:13:37] trying that yeah, otherwise I'll try a reboot [14:13:40] Then maybe see if you can review disk usage on cp12 as that's warning because 16 hours of extra traffic [14:13:40] !log restarted varnish on cp13 [14:13:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:13:56] hm yeah, maybe I can figure out what to delete [14:14:12] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [14:14:39] there we go [14:14:40] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:14:41] That's better [14:14:54] RECOVERY - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is OK: OK - NGINX Error Rate is 3% [14:15:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:15:06] RECOVERY - cp13 HTTPS on cp13 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2256 bytes in 0.097 second response time [14:15:41] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.22, 1.39, 1.49 [14:20:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.96, 19.08, 16.21 [14:21:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 6.27, 5.36 [14:22:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.34, 17.03, 15.82 [14:23:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.87, 5.55, 5.19 [14:29:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 7.14, 5.94 [14:30:27] JohnLewis: we do route Europe then to CA, my point is CA had points where cpu usage was maxed out [14:31:06] So given most of America are only just waking up, could we ca cope with the load of Americas + Europe [14:31:13] GB -> CA -> SG is the appropriate order and what you were asking. Nothing can be done if 50% capacity is lost and then it becomes overwhelmed [14:31:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.19, 7.65, 6.31 [14:31:33] Routing any non Asian traffic via SG is degrading performance [14:31:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.93, 7.14, 6.20 [14:31:58] AF goes gb, sg, ca [14:32:32] Which is generally fine in urgent cases [14:32:39] JohnLewis: hi. Do you have any idea what I could do to reduce disk size on cp12 (due to cp13 being down and traffic being redirected)? I see an increase in access logs but it's not really significant [14:32:40] Nothing can be done reads as yes there's a risk we could end up being overwhelmed if we lost GB [14:33:03] Which is true [14:33:17] If you lose 50% capacity and the main caching centre, you will be overwhelmed [14:33:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.38, 9.14, 7.02 [14:33:59] Having some redundancy should be an aim [14:34:08] Reception123: not without looking at what is causing the consumption [14:34:25] I'd assume just traffic tbh [14:34:38] It's spent 16 hours taking more than normal [14:34:40] I proposed getting a second cp in Canada and SPF declined it [14:34:42] well yes it's clearly because of the increased traffic but where is the space being consumed? [14:35:15] du? [14:35:17] Reception123: you’d need to use du -sh to see on / then track it down [14:35:40] yeah I've been trying that so far but was rather unsuccessful, but will keep going [14:35:44] My guess would be either access logs or disk caching [14:35:49] I was expecting /var/log/nginx or varnish to be the culprits [14:36:20] though I mean I could still probably delete some access logs but they're not *that* large [14:36:23] https://www.irccloud.com/pastebin/vaUf4StA/ [14:36:39] But disk caching should be set at a reasonable percentage of the total disk space so shouldn’t really ever be a problem [14:37:07] Reception123: 4.9 GB on a cp is large [14:37:18] oh yeah definitely but that's the latest access log [14:37:27] if I were to delete something I'd delete the oldest ones not the newest [14:37:31] They're only 20GB [14:37:37] Zip it [14:37:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.40, 7.48, 6.72 [14:38:31] gzip should help, as long as it's enough that it doesn't go critical [14:38:55] 8.5GB today is stupidly high though [14:39:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.60, 7.59, 7.13 [14:40:24] Moving access logs off cp* would help for disk space, which has been part of the central logging task for a while but never done [14:40:58] Though capacity concerns that Paladox raised in the past about graylog do need addressing before such a move can happen either way [14:41:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.22, 6.74, 6.58 [14:42:44] Capacity concerns everywhere need addressing [14:43:01] The only user facing system I don't know of a capacity issue with is db* [14:44:07] Capacity all over needs addressing, indeed but that lies with the DSRE to release funding for such capacity works to be able to take place otherwise you need to cut back in non essential areas [14:45:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.58, 6.12, 6.65 [14:48:14] We're at a very serious risk of facing outages simply because of resource issues [14:48:52] And the slight fact that our systems don't recover well [14:49:27] https://phabricator.miraheze.org/T7323 is where the decision to reduce caching capacity in London was made [14:49:28] [url] ⚓ T7323 New Server Resource Request for Cache Proxies | phabricator.miraheze.org [14:52:05] The reason it was made was to increase disk performance on cloud4/5 and increase performance of caching in London. But obviously it had no impact in cloud performance, minimal impact on caching performance and all it did was make us more prone to outage as shown above [14:52:21] Well summed up [15:04:07] -rwxr-x--- 1 vcache varnish 15G Aug 9 15:03 cache_storage.bin [15:04:12] ^ JohnLewis is there any harm in deleting that completely? [15:06:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.54, 6.96, 6.17 [15:08:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.81, 6.12, 5.95 [15:26:04] Reception123: harm no, use, also no [15:26:13] As you would delete all of the cache stored [15:27:07] That file is set at 15G and you should avoid deleting it unless needed to clear cache as you’d be invalidating the purpose of the server. Log deletion would always be better [15:31:53] Ok, I see. Thanks! [15:34:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.20, 9.27, 7.47 [15:35:09] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.70, 8.95, 7.08 [15:40:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.43, 7.16, 7.13 [15:40:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.28, 7.19, 6.91 [15:42:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.43, 6.23, 6.80 [15:42:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.24, 6.60, 6.73 [15:57:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.16, 7.56, 6.71 [15:59:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.35, 7.56, 6.83 [16:01:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.85, 6.70, 6.61 [16:05:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.45, 8.24, 7.26 [16:07:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.14, 6.80, 6.86 [16:10:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.60, 7.18, 6.66 [16:10:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.09, 20.16, 18.46 [16:13:18] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.78, 6.69, 6.76 [16:14:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.35, 6.28, 6.42 [16:14:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.75, 19.86, 18.82 [16:22:04] PROBLEM - cp13 Varnish Backends on cp13 is WARNING: No backends detected. If this is an error, see readme.txt [16:22:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb [16:22:52] PROBLEM - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is CRITICAL: CRITICAL - NGINX Error Rate is 97% [16:22:59] Reception123: ^ [16:23:06] PROBLEM - cp13 HTTPS on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.013 second response time [16:23:08] OOM again I guess [16:23:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb [16:25:22] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.53, 2.22, 1.39 [16:25:38] PROBLEM - cp12 Disk Space on cp12 is CRITICAL: DISK CRITICAL - free space: / 2273 MB (5% inode=96%); [16:27:19] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.60, 2.00, 1.42 [16:28:05] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.66, 5.47, 4.43 [16:29:17] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.20, 2.03, 1.49 [16:31:14] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.90, 1.93, 1.52 [16:31:37] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 3235 MB (8% inode=96%); [16:31:46] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.69, 22.48, 20.34 [16:32:05] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.33, 5.93, 4.89 [16:33:10] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.25, 1.66, 1.47 [16:35:09] RhinosF1: have OOMs been more frequent since the task I linked above was done? Or are the OOMs just recent in nature? (i.e. no relation to that task's date) [16:35:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.13, 23.30, 21.28 [16:35:59] JohnLewis: not that I can recall [16:36:28] Hm, mkay [16:36:48] Would require more close server side monitoring then for sure [16:36:50] JohnLewis: last few days maybe but not long term [16:37:03] Also do you mind me PMing you an email we just got? [16:37:22] No objections [16:38:05] Sent [16:39:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.16, 18.57, 19.80 [16:40:07] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.88, 4.98, 4.88 [16:42:20] I got a few database errors on meta, they seem to have resolved but it did appear for a few minutes [16:42:53] by database errors I mean that the database could not be accessed [16:43:39] @Agent: only aware of caching issues but noted [16:43:47] Voidwalker: cp13 back down [16:43:52] Looks like OOM [16:44:56] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.01, 1.81, 1.61 [16:46:16] cache-worker invoked oom-killer, yeah [16:46:53] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.40, 1.69, 1.59 [16:47:23] Voidwalker: can you clean the var/lib/varnish up and restart it [16:48:06] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [16:48:33] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:48:41] Yey [16:48:54] RECOVERY - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is OK: OK - NGINX Error Rate is 4% [16:49:07] RECOVERY - cp13 HTTPS on cp13 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2256 bytes in 0.411 second response time [16:49:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:49:52] is fixed? [16:50:44] icinga-miraheze says so [16:51:05] cp12 disk space is tight fyi [16:51:11] looks like restarting varnish clears anyway, due to systemd script [16:51:18] Yeah [16:51:27] It just doesn't do it after an OOM right [16:59:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.44, 6.59, 5.89 [17:01:22] looks like the problem is probably not related to ramdisk, but instead the OOM killed a child process but did not restart the master process [17:01:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.85, 6.31, 5.87 [17:04:16] Voidwalker: PR in coming that may fix it [17:05:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 7.34, 6.42 [17:06:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.77, 7.00, 6.34 [17:07:11] Ah, so after the child gets killed for OOM, it fails to start a new child with: Child (1244) said errno = 28 (No space left on device) [17:07:13] [02puppet] 07JohnFLewis opened pull request 03#1857: T7319: Change KillMode to control-group for Varnish - 13https://git.io/JRgNZ [17:08:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.25, 6.73, 6.32 [17:09:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.92, 6.26, 6.20 [17:10:32] [02puppet] 07The-Voidwalker closed pull request 03#1857: T7319: Change KillMode to control-group for Varnish - 13https://git.io/JRgNZ [17:10:34] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRgNb [17:10:35] [02miraheze/puppet] 07JohnFLewis 0345b39ce - T7319: Change KillMode to control-group for Varnish (#1857) [17:10:51] JohnLewis, thanks, appreciate it [17:11:41] Essentially the reason the new child can't start is because the old process is killed but resources don't get re-allocated [17:12:26] So there is no physical space left, because the file isn't cleaned up, but the system.d service just sees no process running and expects the file to not exist because system.d clears the resources in its memory only [17:12:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 7.13, 6.59 [17:12:49] JohnLewis: there's a task if you want to close it [17:13:44] Done [17:13:55] Ty [17:14:05] At least hopefully they'll be much shorter outages now [17:14:08] Not 16 hours [17:14:35] And if that doesn't work, I have another thing to try [17:14:47] But that killmode setting should [17:17:37] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.51, 6.99, 6.52 [17:18:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.69, 7.97, 7.06 [17:19:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.47, 7.23, 6.68 [17:20:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 7.75, 7.09 [17:21:34] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.56, 8.10, 7.05 [17:25:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.92, 7.04, 6.90 [17:32:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.95, 6.47, 6.80 [17:33:16] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 6.93, 6.93 [17:36:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 6.87, 8.51, 7.69 [17:37:08] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.63, 7.53, 7.26 [17:38:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.84, 7.26, 7.33 [17:44:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.37, 6.06, 6.66 [17:46:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.36, 5.94, 6.65 [18:05:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.96, 7.24, 6.68 [18:07:43] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2642 MB (10% inode=94%); [18:10:40] Oh loveky [18:10:46] Go away cp3 [18:11:09] Voidwalker: can you watch cache proxy disk usage in case they do go funny [18:11:29] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.01, 6.50, 6.56 [18:11:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.98, 6.74, 6.43 [18:13:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.56, 6.18, 6.26 [18:19:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.89, 9.10, 7.33 [18:21:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.84, 7.07, 6.71 [18:23:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.44, 6.77, 6.66 [18:23:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.54, 7.64, 7.13 [18:27:38] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.30, 6.49, 6.75 [18:30:08] looks fine at the moment, not really sure what I could do if it fills up though, besides dumping logs (which is not ideal) [18:30:52] I may just compress some of the other log files if needed [18:33:28] Reducing the size of disk caching is one option, though that reduces caching efficiency really. Moving to graylog for cp* access/error logs may work but as mentioned before Graylog needs quite a few changes to prevent being overwhelmed [18:35:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.95, 6.73, 6.68 [18:37:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.19, 6.51, 6.61 [18:41:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.83, 6.99, 6.76 [18:45:35] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.65, 8.05, 7.19 [18:47:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.02, 7.46, 7.13 [18:48:52] PROBLEM - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is CRITICAL: CRITICAL - NGINX Error Rate is 99% [18:49:07] PROBLEM - cp13 HTTPS on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.010 second response time [18:49:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb [18:49:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.31, 6.45, 6.80 [18:49:40] PROBLEM - cp12 Disk Space on cp12 is CRITICAL: DISK CRITICAL - free space: / 2168 MB (5% inode=96%); [18:49:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb [18:50:06] PROBLEM - cp13 Varnish Backends on cp13 is WARNING: No backends detected. If this is an error, see readme.txt [18:51:09] Voidwalker: ^ [18:51:16] Reception123: ^ [18:51:29] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 5.31, 2.82, 1.61 [18:52:14] Urgh again? [18:52:42] looking [18:53:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.61, 7.22, 6.54 [18:54:09] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw9 [18:54:13] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.10, 5.92, 4.36 [18:54:33] Voidwalker: was system.d reloaded and the service restarted when my change was deployed? [18:54:52] RECOVERY - cp13 HTTP 4xx/5xx ERROR Rate on cp13 is OK: OK - NGINX Error Rate is 2% [18:55:06] RECOVERY - cp13 HTTPS on cp13 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2256 bytes in 0.034 second response time [18:55:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.18, 20.73, 18.81 [18:55:33] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.84, 7.80, 7.24 [18:55:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:55:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:56:06] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 7 backends are healthy [18:56:13] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.40, 5.47, 4.38 [18:57:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 7.34, 7.13 [18:57:43] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.85, 7.92, 6.95 [18:58:14] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.29, 5.75, 4.62 [18:58:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.33, 6.19, 5.30 [18:59:29] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.26, 19.54, 18.87 [18:59:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.87, 7.13, 6.78 [19:00:13] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.39, 5.13, 4.53 [19:00:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.39, 5.78, 5.25 [19:00:41] JohnLewis, puppet had deployed, but I don't think so. I've done so now manually to cp13 [19:01:27] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.10, 1.75, 1.68 [19:01:37] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.02, 6.68, 6.65 [19:02:55] The change would now then be deployed :P [19:03:25] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.52, 1.69, 1.67 [19:03:30] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.21, 23.21, 20.52 [19:03:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.85, 7.98, 7.37 [19:03:37] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 2615 MB (6% inode=96%); [19:04:41] heh, well, we'll see if it worked [19:04:54] Nice, then it works again 👍 [19:05:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.91, 23.54, 20.99 [19:05:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.05, 6.85, 6.11 [19:05:38] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.15, 9.27, 7.72 [19:06:17] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.42, 5.01, 4.72 [19:07:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.73, 6.58, 6.11 [19:11:28] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 7.83, 7.78 [19:11:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.77, 7.65, 7.53 [19:11:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.64, 19.44, 20.11 [19:17:21] Voidwalker: hi, you probably want to restart cp12: https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=cp12.miraheze.org&var-port=9100 [19:17:22] [url] Grafana | grafana.miraheze.org [19:20:01] paladox, restart device or varnish? [19:20:09] restart varnish [19:21:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.70, 5.96, 6.75 [19:22:30] !log restart varnish on cp12 [19:22:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:52] might also compress access log [19:23:00] disk utilization peaking at 98% [19:23:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.37, 6.18, 6.80 [19:25:36] thanks! [19:31:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.87, 6.71, 6.69 [19:32:45] no problem, good to see you paladox :) [19:32:59] :) [19:33:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.10, 8.58, 7.35 [19:35:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.41, 8.64, 7.36 [19:36:48] JohnLewis: paladox welcome back! it's official :) [19:36:59] thanks! [19:37:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.79, 7.68, 7.17 [19:37:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.97, 7.85, 7.36 [19:39:36] !log blocked IP in cp13 firewall [19:39:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:40] it's official 🙂 [19:42:47] paladox, just to confirm, your previous SSH key(s) are still valid? [19:42:54] Yeh [19:45:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.50, 6.29, 6.75 [19:47:44] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-2 [+0/-0/±1] 13https://git.io/JR2B9 [19:47:45] [02miraheze/puppet] 07The-Voidwalker 030f70c60 - restore users for John and Paladox [19:47:47] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-2 - 13https://git.io/vbiAS [19:47:51] [02puppet] 07The-Voidwalker opened pull request 03#1858: restore users for John and Paladox - 13https://git.io/JR2BH [19:48:28] It seems like we don't yet have explicit approval from Reception123 though [19:48:56] MacFan4000: I discussed it with Void in a private conversation but sure I'll confirm on the task if that makes it clearer. [19:49:10] that would be best [19:49:29] done [19:49:32] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-2 [+0/-0/±1] 13https://git.io/JR2Rf [19:49:33] [02miraheze/puppet] 07The-Voidwalker 032287932 - add to monitoring as well [19:49:35] [02puppet] 07The-Voidwalker synchronize pull request 03#1858: restore users for John and Paladox - 13https://git.io/JR2BH [19:50:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.29, 8.73, 7.69 [19:51:38] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.04, 7.65, 7.16 [19:52:10] [02mw-config] 07MacFan4000 opened pull request 03#4044: re-add paladox and John to staffwiki - 13https://git.io/JR2Rc [19:53:06] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.70, 5.37, 4.46 [19:53:12] miraheze/mw-config - MacFan4000 the build passed. [19:53:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.50, 7.27, 7.09 [19:54:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.57, 7.34, 7.39 [19:55:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.00, 6.36, 6.77 [19:57:05] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.95, 5.30, 4.71 [20:01:05] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.46, 4.09, 4.38 [20:01:46] added checklists to task for onboarding [20:02:12] I don't have access to -feed for IRC [20:02:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.46, 7.08, 6.94 [20:03:00] Reception123: ^^ [20:04:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.76, 8.91, 7.63 [20:04:48] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.27, 9.93, 8.34 [20:05:39] dmehus: ^^ as well [20:06:47] [02puppet] 07The-Voidwalker closed pull request 03#1858: restore users for John and Paladox - 13https://git.io/JR2BH [20:06:48] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JR2Eq [20:06:50] [02miraheze/puppet] 07The-Voidwalker 03d1582ed - restore users for John and Paladox (#1858) [20:06:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.48, 6.69, 6.16 [20:08:52] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.73, 7.64, 6.56 [20:10:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.81, 7.34, 6.59 [20:10:57] let me do -feed [20:12:51] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.96, 8.11, 6.95 [20:13:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.35, 7.54, 6.28 [20:15:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.28, 7.31, 6.35 [20:16:05] doing ldap [20:17:35] PROBLEM - cp12 Disk Space on cp12 is CRITICAL: DISK CRITICAL - free space: / 2304 MB (5% inode=96%); [20:18:47] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.79, 7.74, 7.19 [20:19:51] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 17 minutes ago with 0 failures [20:20:49] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.95, 7.34 [20:20:49] JohnLewis, paladox: if you want a first action, can we see if cp12 can not run out of space before the logs rotate [20:22:50] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.81, 7.33, 7.17 [20:22:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.27, 6.50, 6.37 [20:24:46] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.59, 20.66, 18.64 [20:26:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.25, 19.36, 18.40 [20:30:49] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.70, 6.36, 6.79 [20:32:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.30, 6.74, 7.88 [20:32:37] JohnLewis, paladox see tasks for LDAP instructions and let me know if you are missing access to anything [20:34:11] Voidwalker: thanks! [20:34:14] RhinosF1: ack [20:34:35] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.00, 8.42, 8.38 [20:34:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.20, 6.68, 6.85 [20:34:57] And welcome back paladox, glad to have you [20:35:10] thanks! [20:36:36] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.00, 22.41, 19.75 [20:36:52] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.86, 7.52, 7.11 [20:37:56] Voidwalker: LDAP sorted, thank you [20:38:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.26, 22.18, 19.99 [20:39:29] github invites should be out too [20:40:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.68, 20.30, 19.53 [20:40:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.03, 7.85, 7.44 [20:41:01] thanks, accepted! [20:50:50] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.12, 7.22, 6.75 [20:51:46] [02mw-config] 07The-Voidwalker closed pull request 03#4044: re-add paladox and John to staffwiki - 13https://git.io/JR2Rc [20:51:48] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR2wK [20:51:49] [02miraheze/mw-config] 07MacFan4000 03c562f8b - re-add paladox and John to staffwiki (#4044) [20:52:43] miraheze/mw-config - The-Voidwalker the build passed. [20:52:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.49, 7.20, 6.80 [20:54:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.44, 7.35, 6.88 [20:54:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.87, 8.53, 7.65 [20:56:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.97, 7.38, 6.93 [20:56:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.49, 8.00, 7.56 [21:02:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.64, 7.64, 7.13 [21:02:55] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.52, 7.86, 7.57 [21:05:44] !log db12: UPDATE `piwik_user` SET superuser_access = 1 WHERE `login` = 'paladox'; [21:05:53] Voidwalker: your username is void right? not voidwalker? [21:06:00] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.26, 21.72, 19.65 [21:06:01] i'll grant you superadmin access [21:06:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:20] yup [21:06:38] !log db12: UPDATE `piwik_user` SET superuser_access = 1 WHERE `login` = 'void'; [21:06:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:58] thanks! [21:07:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.21, 20.97, 19.62 [21:09:22] JohnLewis: you need to login so i can do yours [21:09:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.82, 19.37, 19.17 [21:10:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.59, 7.58, 7.61 [21:10:36] paladox: in [21:10:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.18, 7.89, 7.73 [21:11:20] thanks! [21:11:36] JohnLewis: done [21:11:51] everytime i login.... matomo changes my email to noreply [21:11:57] What would that mean for John Lewis? [21:13:05] oh i see why [21:15:42] JohnLewis: when you have a chance your phab profile still says that you are a steward [21:16:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.54, 3.02 [21:18:00] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 3.03, 2.90 [21:24:52] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.09, 8.08, 7.52 [21:26:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.92, 7.16, 7.25 [21:27:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.49, 6.36, 6.73 [21:32:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.41, 7.13, 7.14 [21:34:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.68, 7.09, 7.14 [21:40:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.20, 6.10, 6.69 [21:42:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.66, 6.79, 7.94 [21:48:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.49, 6.84, 7.79 [21:54:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.48, 7.59, 7.83 [21:56:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.92, 7.75, 7.87 [21:58:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.20, 8.64, 8.15 [22:02:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.27, 7.81, 7.96 [22:04:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.34, 10.10, 8.79 [22:04:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.39, 8.38, 7.55 [22:06:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.39, 7.20, 7.22 [22:10:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 7.22, 7.89 [22:10:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.49, 5.70, 6.57 [22:18:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.40, 5.22, 6.66 [22:25:51] [02mediawiki] 07Universal-Omega closed pull request 03#2596: Bump extensions/WikiForum from `0cf5ac4` to `b9b6abc` - 13https://git.io/JRRrL [22:25:53] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JR2Ha [22:25:54] [02miraheze/mediawiki] 07dependabot[bot] 0374adc49 - Bump extensions/WikiForum from `0cf5ac4` to `b9b6abc` (#2596) [22:25:56] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/WikiForum-b9b6abc - 13https://git.io/vbL5b [22:25:57] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/WikiForum-b9b6abc [22:27:44] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JR2HP [22:27:46] [02miraheze/mediawiki] 07Universal-Omega 03800afbb - Update Cosmos [22:28:05] [02mediawiki] 07Universal-Omega closed pull request 03#2592: Bump extensions/Comments from `ff88c27` to `bc9f2b7` - 13https://git.io/JRRw7 [22:28:06] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JR2HD [22:28:08] [02miraheze/mediawiki] 07dependabot[bot] 03306b128 - Bump extensions/Comments from `ff88c27` to `bc9f2b7` (#2592) [22:28:09] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/Comments-bc9f2b7 - 13https://git.io/vbL5b [22:28:11] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/Comments-bc9f2b7 [22:35:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.28, 7.03, 6.39 [22:36:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.83, 6.83, 6.16 [22:37:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.83, 7.29, 6.56 [22:40:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.01, 7.70, 6.66 [22:40:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.68, 7.97, 6.99 [22:41:24] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:41:55] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:42:16] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:42:34] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:42:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.30, 6.73, 6.43 [22:42:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.03, 7.72, 7.03 [22:43:30] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:46:48] Universal_Omega, ^ might be you? [22:48:13] Voidwalker: Yeah it was. It should fix by itself on next run, but I can manually fix now if needed. [22:48:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.12, 6.33, 6.67 [22:48:54] all right, no worries as long as it fixes [22:51:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JR25g [22:51:19] [02miraheze/mw-config] 07Universal-Omega 036708dde - Update ManageWikiSettings.php [22:51:20] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [22:51:23] [02mw-config] 07Universal-Omega opened pull request 03#4045: Add Cosmos WVUI related configs to ManageWiki - 13https://git.io/JR25V [22:52:25] miraheze/mw-config - Universal-Omega the build passed. [22:54:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JR25N [22:54:30] [02miraheze/mw-config] 07Universal-Omega 034625feb - Update LocalSettings.php [22:54:31] [02mw-config] 07Universal-Omega synchronize pull request 03#4045: Add Cosmos WVUI related configs to ManageWiki - 13https://git.io/JR25V [22:55:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR2dv [22:55:16] [02miraheze/services] 07MirahezeSSLBot 030b59371 - BOT: Updating services config for wikis [22:55:26] miraheze/mw-config - Universal-Omega the build passed. [22:57:56] [02mw-config] 07Universal-Omega closed pull request 03#4045: Add Cosmos WVUI related configs to ManageWiki - 13https://git.io/JR25V [22:57:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JR2dl [22:57:59] [02miraheze/mw-config] 07Universal-Omega 0323690bb - Add Cosmos WVUI related configs to ManageWiki (#4045) [22:58:00] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [22:58:02] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [22:58:50] miraheze/mw-config - Universal-Omega the build passed. [23:00:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.33, 7.38, 6.66 [23:01:55] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:02:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.28, 6.50, 6.31 [23:02:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.20, 7.01, 6.61 [23:04:16] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:04:35] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.64, 7.92, 6.86 [23:04:38] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:04:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.06, 9.70, 7.67 [23:05:32] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:05:34] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:06:49] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:07:16] !log [@test3] starting deploy of {'config': True} to skip [23:07:17] !log [@test3] finished deploy of {'config': True} to skip - FAIL: [256] [23:07:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.65, 7.71, 7.06 [23:10:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.07, 7.32, 7.32 [23:12:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.89, 8.01, 7.24 [23:12:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.01, 8.13, 7.62 [23:14:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.86, 7.28 [23:14:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 7.19, 7.33 [23:16:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.33, 7.83, 7.31 [23:18:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 7.41, 7.21 [23:25:10] !log rm spf+void from sre-mgmt group in LDAP + add myself [23:25:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.57, 6.01, 6.63 [23:28:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.09, 5.87, 6.62 [23:31:20] So, you're not becoming a Steward again? Then again, I was disappointed to see you resign from your position. [23:31:58] Nvm, saw the update. [23:36:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.07, 8.30, 7.28 [23:36:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.98, 7.06, 6.83 [23:38:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.12, 7.50, 7.09 [23:38:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.62, 6.43, 6.62 [23:44:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.74, 6.19, 6.65 [23:48:49] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 15 minutes ago with 0 failures