[00:02:49] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:02:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.67, 6.57, 5.80 [00:05:24] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.15, 8.49, 6.35 [00:06:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.37, 8.22, 6.25 [00:07:33] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:08:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.34, 7.06, 6.07 [00:08:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.16, 21.48, 17.52 [00:08:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.73, 6.26, 5.99 [00:09:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.16, 7.19, 6.36 [00:10:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.42, 6.17, 5.86 [00:10:44] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.34, 19.81, 17.39 [00:15:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.80, 6.14, 6.19 [00:21:24] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [00:30:49] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [00:33:47] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.31, 8.29, 6.99 [00:34:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.97, 6.64, 6.11 [00:35:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.40, 7.55, 6.56 [00:35:36] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.80, 7.99, 6.97 [00:36:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.42, 6.59, 6.16 [00:37:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.31, 6.54, 6.31 [00:37:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.53, 7.50, 6.92 [00:39:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.36, 7.47, 7.10 [00:39:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.15, 6.61, 6.66 [00:41:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.15, 6.50, 6.80 [00:44:09] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [00:55:02] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.79, 1.75, 1.40 [00:57:01] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.32, 1.62, 1.39 [01:18:53] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [01:41:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [01:55:35] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:11:25] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:25:16] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:26:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.88, 5.93, 5.51 [02:28:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.42, 5.86, 5.54 [02:34:12] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:48:05] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:56:59] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [03:10:51] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [03:33:39] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [03:47:30] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [04:17:14] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:12:43] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [05:21:36] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:35:30] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [05:51:17] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:51:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.20, 6.38, 5.42 [05:53:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.68, 5.96, 5.38 [06:10:02] !log [reception@jobrunner3] sudo -u www-data php initSiteStats.php --wiki internetpediawiki --update [06:10:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:38] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [08:05:05] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:39:46] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [08:48:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:51:47] PROBLEM - mrlove.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mrlove.wiki could not be found [08:58:40] RECOVERY - mrlove.wiki - reverse DNS on sslhost is OK: rDNS OK - mrlove.wiki reverse DNS resolves to cp13.miraheze.org [09:23:19] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [09:32:11] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:46:04] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [10:15:45] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [10:34:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.49, 6.82, 5.71 [10:34:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.06, 6.30, 5.47 [10:35:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.17, 6.97, 5.75 [10:36:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.39, 6.14, 5.59 [10:36:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.15, 5.52 [10:37:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.41, 6.46, 5.72 [11:11:13] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [11:20:06] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [11:34:01] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [11:49:50] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [12:03:42] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [12:12:35] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [12:40:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JGL9B [12:40:13] [02miraheze/services] 07MirahezeSSLBot 03e5331c7 - BOT: Updating services config for wikis [12:47:15] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [12:56:12] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:02:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.82, 6.46, 5.77 [13:04:34] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.29, 6.55, 5.89 [13:04:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.95, 6.59, 5.90 [13:07:11] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JGLdk [13:07:13] [02miraheze/dns] 07Reception123 03749adaf - add wikimoma.art zone [13:08:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.54, 5.90, 5.80 [13:18:56] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JGLbG [13:18:57] [02miraheze/ssl] 07Reception123 036cff645 - add wikimoma.art cert [13:35:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.31, 7.14, 6.28 [13:35:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.88, 18.23, 15.76 [13:36:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.14, 7.20, 6.50 [13:37:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.12, 6.63, 6.21 [13:37:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.43, 16.55, 15.46 [13:46:08] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 7.37, 6.82 [13:46:21] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JGLjj [13:46:23] [02miraheze/IncidentReporting] 07translatewiki 03c883458 - Localisation updates from https://translatewiki.net. [13:46:24] [ Main page - translatewiki.net ] - translatewiki.net [13:46:24] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JGtev [13:46:26] [02miraheze/CreateWiki] 07translatewiki 03d182961 - Localisation updates from https://translatewiki.net. [13:46:27] [ Main page - translatewiki.net ] - translatewiki.net [13:46:27] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JGteU [13:46:29] [02miraheze/DataDump] 07translatewiki 03358df0d - Localisation updates from https://translatewiki.net. [13:46:30] [ Main page - translatewiki.net ] - translatewiki.net [13:46:30] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JGteT [13:46:32] [02miraheze/landing] 07translatewiki 03563dcb6 - Localisation updates from https://translatewiki.net. [13:46:33] [ Main page - translatewiki.net ] - translatewiki.net [13:46:33] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JGtek [13:46:35] [02miraheze/MirahezeMagic] 07translatewiki 038068d57 - Localisation updates from https://translatewiki.net. [13:46:36] [ Main page - translatewiki.net ] - translatewiki.net [13:46:36] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JGteI [13:46:38] [02miraheze/ManageWiki] 07translatewiki 0334398d9 - Localisation updates from https://translatewiki.net. [13:46:39] ... [13:46:39] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JGteL [13:46:41] [02miraheze/WikiDiscover] 07translatewiki 03814ef81 - Localisation updates from https://translatewiki.net. [13:46:42] ... [13:47:20] miraheze/CreateWiki - translatewiki the build passed. [13:47:24] miraheze/IncidentReporting - translatewiki the build passed. [13:47:31] miraheze/DataDump - translatewiki the build passed. [13:47:38] miraheze/MirahezeMagic - translatewiki the build passed. [13:47:45] miraheze/WikiDiscover - translatewiki the build passed. [13:47:47] miraheze/landing - translatewiki the build passed. [13:47:55] miraheze/ManageWiki - translatewiki the build passed. [13:48:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.43, 6.98, 6.75 [13:51:05] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [13:52:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.51, 6.60, 6.66 [13:53:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.44, 6.73, 6.09 [13:55:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.46, 6.49, 6.08 [13:59:46] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:03:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.63, 7.09, 6.47 [14:05:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.20, 7.13, 6.52 [14:05:51] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.96, 8.10, 7.21 [14:05:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.13, 7.63, 6.50 [14:07:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 7.41, 6.75 [14:07:49] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.50, 7.87, 7.24 [14:07:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.93, 7.07, 6.44 [14:09:13] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.62, 6.78, 6.61 [14:09:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.20, 6.30, 6.34 [14:09:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.85, 6.68, 6.37 [14:13:19] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [14:25:38] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.12, 7.17, 7.03 [14:29:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.46, 7.24, 7.13 [14:34:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.59, 7.14, 6.71 [14:34:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.32, 7.71, 6.74 [14:36:47] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.30, 7.46, 6.79 [14:36:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.75, 7.26, 6.70 [14:37:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.92, 7.65, 7.29 [14:38:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.47, 6.18, 6.44 [14:39:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.02, 6.68, 6.98 [14:40:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.06, 6.35, 6.48 [14:40:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.51, 6.64, 6.57 [14:45:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.26, 7.35, 7.14 [14:47:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.41, 7.00, 7.04 [14:51:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.41, 7.01, 6.64 [14:55:54] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:57:18] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.48, 6.46, 6.74 [14:57:55] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.45, 6.46, 6.60 [15:00:35] dmehus: I pushed a patch for the echo/comments issue [15:01:20] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.02, 1.72, 1.19 [15:03:15] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.40, 1.65, 1.23 [15:03:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.98, 7.31, 6.98 [15:05:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.41, 7.19, 6.99 [15:08:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.61, 7.68, 6.97 [15:09:26] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [15:10:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.75, 6.83, 6.74 [15:12:51] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.76, 6.65, 6.70 [15:13:19] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.92, 6.36, 6.72 [15:21:15] dmehus: merges! [15:21:17] Merged! [15:21:22] [02mediawiki] 07Universal-Omega closed pull request 03#1990: Bump browserslist from 4.12.0 to 4.16.6 - 13https://git.io/JGku1 [15:21:24] [02mediawiki] 07dependabot[bot] commented on pull request 03#1990: Bump browserslist from 4.12.0 to 4.16.6 - 13https://git.io/JGt82 [15:21:28] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/browserslist-4.16.6 [15:21:29] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/browserslist-4.16.6 - 13https://git.io/vbL5b [15:22:44] dmehus: I'm on the Jenkins whitelist [15:29:22] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03comments-fix-localurl [+0/-0/±1] 13https://git.io/JGtRr [15:29:23] [02miraheze/mediawiki] 07RhinosF1 032ab487a - Update Comments [15:29:25] [02mediawiki] 07RhinosF1 created branch 03comments-fix-localurl - 13https://git.io/vbL5b [15:30:07] [02mediawiki] 07RhinosF1 opened pull request 03#1991: Update Comments - 13https://git.io/JGtRS [15:30:19] Reception123: ^ [15:30:44] today is stupid busy day [15:31:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [15:34:35] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.09, 7.21, 6.12 [15:34:36] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.65, 7.20, 6.28 [15:34:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 6.28, 6.11 [15:36:17] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.61, 7.17, 6.70 [15:36:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.62, 6.23, 5.90 [15:36:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.06, 6.74, 6.23 [15:36:41] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.46, 6.12, 6.08 [15:38:15] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.97, 6.26, 6.42 [15:42:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.19, 7.27, 6.35 [15:42:48] RhinosF1, ah, cool :) [15:44:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.63, 6.97, 6.38 [15:45:08] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.74, 7.80, 7.01 [15:45:27] Crepusculum: Phab doesn't allow you to like comments, but that's a good reply you have - thank you :) [15:45:32] *gave [15:46:37] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.95, 6.64, 6.33 [15:47:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.66, 7.37, 6.95 [15:48:10] JohnLewis: you're welcome. And yes I agree it should've been prioritised higher for a bug like that. I apologise for not looking into it sooner. Being "busy" shouldn't have been an excuse in it as it should've been prioritised higher then other tasks due to it breaks the extension entirely. I'm already looking now after giving it some more thought. [15:49:04] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.57, 6.77, 6.78 [15:49:39] Crepusculum: as long as it's being dealt with in the end - that's all that matters! I'm sure you'll take away a learning point for next time ;) [15:54:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGtgQ [15:54:34] [02miraheze/mw-config] 07Universal-Omega 03ccbac34 - Set wgOnlyUserEditUserPage [15:54:35] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [15:54:45] [02mw-config] 07Universal-Omega opened pull request 03#3925: Set wgOnlyUserEditUserPage - 13https://git.io/JGtg7 [15:55:50] miraheze/mw-config - Universal-Omega the build passed. [15:58:33] RhinosF1: miraheze-github needs connecting to NickServ on Libera like it was on Freenode, I think. [16:04:04] JohnLewis: yes I will. Task should've been better prioritised based on impact, and this should've been higher prioritised because it had a larger impact being that it made the extension become useless. I will better prioritise next time. A week for waiting on looking into a task that had breaking effect shouldn't had happened, whether it's low priority or not. Especially since I said I'd look into it within the next couple days [16:04:04] but didn't until they mentioned it again after a week, so once again apologies for the false promise as well. I acknowledge it was handled poorly and I'll try to not do that again in the future. Thank you for bringing the issue up. [16:05:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.84, 7.13, 6.35 [16:05:35] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [16:06:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.17, 7.33, 6.42 [16:06:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.53, 7.32, 6.90 [16:07:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.62, 6.47, 6.20 [16:08:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.02, 6.55, 6.25 [16:08:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.33, 6.40, 6.60 [16:14:17] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [16:17:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.40, 7.78, 7.08 [16:19:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.79, 7.04, 6.90 [16:19:39] Crepusculum: I think it was on icinga-Miraheze's account so ask someone with the PW to group [16:25:34] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.17, 1.96, 1.32 [16:27:29] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.27, 1.56, 1.24 [16:27:52] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [16:29:23] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.80, 6.57, 6.80 [16:33:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.80, 6.85, 6.90 [16:33:35] * dmehus thinks someone should file a Phabricator.com Phabricator feature request to add a "like" feature to Phabricator comments and tasks [16:34:20] s/comments and tasks/comments [16:34:20] dmehus meant to say: thinks someone should file a Phabricator.com Phabricator feature request to add a "like" feature to Phabricator comments [16:34:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.98, 6.79, 6.29 [16:35:00] * RhinosF1 does not think that will get very far [16:35:57] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 10.63, 7.96, 3.87 [16:36:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.93, 6.68, 6.21 [16:36:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.18, 6.43, 6.21 [16:36:35] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [16:37:57] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 2.00, 5.52, 3.47 [16:38:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.10, 6.36, 6.14 [16:43:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.04, 7.42, 7.17 [16:44:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.07, 7.12, 6.52 [16:45:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.31, 7.05, 7.08 [16:46:20] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.95, 6.47, 6.35 [16:49:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.04, 6.90, 6.95 [16:50:11] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [16:51:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.34, 7.00, 6.98 [16:51:37] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.91, 6.56, 6.08 [16:53:37] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.04, 6.52, 6.14 [16:55:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.52, 7.37, 7.11 [16:57:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.93, 6.84, 6.95 [16:57:28] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.03, 1.82, 1.50 [16:59:23] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.54, 1.65, 1.47 [17:05:40] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [17:09:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.08, 6.06, 6.59 [17:28:37] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.24, 2.18, 1.65 [17:30:37] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.04, 1.78, 1.56 [17:32:37] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.88, 1.46, 1.47 [17:34:29] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGtFN [17:34:31] [02miraheze/IncidentReporting] 07Universal-Omega 036f22790 - Fix statistics [17:34:32] [02IncidentReporting] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fh5YJ [17:34:34] [02IncidentReporting] 07Universal-Omega opened pull request 03#22: Fix statistics - 13https://git.io/JGtFA [17:35:36] miraheze/IncidentReporting - Universal-Omega the build passed. [17:42:34] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JGtNu [17:42:36] [02miraheze/IncidentReporting] 07Universal-Omega 032af87ce - Update SpecialIncidentReports.php [17:42:37] [02IncidentReporting] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/fh5YJ [17:42:39] [02IncidentReporting] 07Universal-Omega opened pull request 03#23: Update SpecialIncidentReports.php - 13https://git.io/JGtNa [17:43:35] miraheze/IncidentReporting - Universal-Omega the build passed. [17:45:27] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JGtAc [17:45:28] [02miraheze/IncidentReporting] 07Universal-Omega 03c457372 - Update SpecialIncidentReports.php [17:45:30] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#23: Update SpecialIncidentReports.php - 13https://git.io/JGtNa [17:46:24] miraheze/IncidentReporting - Universal-Omega the build passed. [17:49:30] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JGtxU [17:49:32] [02miraheze/IncidentReporting] 07Universal-Omega 036480f6c - Update SpecialIncidentReports.php [17:49:33] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#23: Update SpecialIncidentReports.php - 13https://git.io/JGtNa [17:50:42] miraheze/IncidentReporting - Universal-Omega the build passed. [17:53:43] [02IncidentReporting] 07Universal-Omega closed pull request 03#23: Update SpecialIncidentReports.php - 13https://git.io/JGtNa [17:53:45] [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [17:53:47] [02IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/fh5YJ [17:59:56] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [18:03:50] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGthF [18:03:52] [02miraheze/IncidentReporting] 07Universal-Omega 03030c637 - Update SpecialIncidentReports.php [18:03:53] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:04:56] miraheze/IncidentReporting - Universal-Omega the build passed. [18:05:25] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGtjm [18:05:27] [02miraheze/IncidentReporting] 07Universal-Omega 0342d5608 - Update SpecialIncidentReports.php [18:05:28] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:05:47] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.26, 7.13, 6.19 [18:05:55] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.16, 7.85, 6.49 [18:06:18] miraheze/IncidentReporting - Universal-Omega the build passed. [18:06:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.66, 7.22, 6.34 [18:06:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.04, 7.62, 6.58 [18:07:55] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.63, 7.18, 6.41 [18:08:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.73, 7.33, 6.61 [18:09:04] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGtj9 [18:09:05] [02miraheze/IncidentReporting] 07Universal-Omega 03389e6f5 - Update SpecialIncidentReports.php [18:09:07] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:09:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.10, 7.30, 6.53 [18:10:04] miraheze/IncidentReporting - Universal-Omega the build passed. [18:10:37] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.66, 6.77, 6.49 [18:11:38] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:11:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.78, 7.57, 6.73 [18:12:47] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGqeg [18:12:48] [02miraheze/IncidentReporting] 07Universal-Omega 03a88936e - Update SpecialIncidentReports.php [18:12:50] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:13:41] miraheze/IncidentReporting - Universal-Omega the build passed. [18:14:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.85, 7.31, 6.81 [18:15:53] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.00, 6.11, 6.37 [18:16:36] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.79, 6.33, 6.39 [18:17:47] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.93, 6.61, 6.47 [18:18:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.33, 6.31, 6.51 [18:18:32] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGqvz [18:18:34] [02miraheze/IncidentReporting] 07Universal-Omega 03975d208 - Update SpecialIncidentReports.php [18:18:35] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:19:28] miraheze/IncidentReporting - Universal-Omega the build passed. [18:20:51] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGqfv [18:20:52] [02miraheze/IncidentReporting] 07Universal-Omega 03157e662 - Update SpecialIncidentReports.php [18:20:54] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:21:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.77, 7.33, 6.77 [18:21:50] miraheze/IncidentReporting - Universal-Omega the build passed. [18:23:47] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.45, 6.97, 6.71 [18:25:36] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGqJT [18:25:37] [02miraheze/IncidentReporting] 07Universal-Omega 03d0b1ad2 - Update SpecialIncidentReports.php [18:25:39] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:25:47] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.43, 6.70, 6.63 [18:26:28] miraheze/IncidentReporting - Universal-Omega the build passed. [18:28:18] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGqU2 [18:28:19] [02miraheze/IncidentReporting] 07Universal-Omega 03e0a38e6 - Update SpecialIncidentReports.php [18:28:21] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:29:24] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#22: Fix statistics - 13https://git.io/JGtFA [18:29:26] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JGqUy [18:29:27] [02miraheze/IncidentReporting] 07Universal-Omega 0365b4c9c - Update SpecialIncidentReports.php [18:29:35] miraheze/IncidentReporting - Universal-Omega the build passed. [18:30:24] miraheze/IncidentReporting - Universal-Omega the build passed. [18:35:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.45, 6.92, 6.37 [18:35:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.51, 7.27, 6.79 [18:36:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.19, 7.09, 6.79 [18:37:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.83, 6.35, 6.22 [18:42:07] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.18, 6.65, 6.70 [18:42:33] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [18:43:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.59, 6.55, 6.71 [18:52:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 6.79, 6.67 [18:54:59] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.32, 7.31, 6.87 [18:56:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.83, 7.31, 6.93 [19:02:51] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.36, 6.59, 6.73 [19:05:35] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:06:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.99, 6.93, 6.84 [19:08:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.62, 6.07, 6.53 [19:16:25] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [19:25:07] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [19:31:46] Crepusculum: does ldapwikiwiki use its own table for 2FA? [19:32:52] RhinosF1: I'm actually not sure about that, but maybe. I did test ldapwikiwiki by proxy to test3 on 1.36, and log in + 2FA worked. [19:33:13] Crepusculum: oh yeah it's a for future thing [19:33:18] It's ran in prod [19:33:25] But it'll work on 1.36 [19:33:42] They added a script in 1.36 to migrate old ways of doing it [19:33:44] oh. ok then. yeah not sure then. [19:34:00] Not sure when it'll stop supporting old token storage method [19:34:20] RhinosF1: https://phabricator.miraheze.org/T7359 updated that by the way, to add every single extension to the list and... that is a large list... [19:34:22] [ ⚓ T7359 Test all extensions for 1.36 ] - phabricator.miraheze.org [19:35:00] Crepusculum: I saw, it won't harm to just run the script I did from test3 with --wiki [19:35:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.65, 8.25, 6.94 [19:35:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 7.01, 6.25 [19:36:34] Crepusculum: flagged revs is super important because it's got config changes too applied [19:36:36] RhinosF1, I assume `ldapwikiwiki` would have its own table for two-factor authentication as the user table is completely separate from the rest of the farm and I did have to separately enable 2FA on that account [19:36:59] dmehus: yes I'm more asking on what db [19:37:07] RhinosF1, ah [19:37:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.64, 7.94, 6.98 [19:37:42] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.03, 6.47, 6.16 [19:37:45] https://github.com/miraheze/mw-config/blob/37544fa8b73484e3cbdf1fd11acd6096166a9ad3/LocalSettings.php#L3359 [19:37:45] [ mw-config/LocalSettings.php at 37544fa8b73484e3cbdf1fd11acd6096166a9ad3 · miraheze/mw-config · GitHub ] - github.com [19:37:52] Yeah it's ldapwikiwiki [19:38:02] oh yeah. [19:38:04] Might be worth seeing if someone can dump it [19:38:19] ah [19:38:31] Crepusculum: do you mind testing logging in w/ scratch token from prod + test3? [19:38:40] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [19:38:50] I know it'll use them up annoying but we should double check it still works [19:38:55] Reedy said it should [19:39:14] RhinosF1, can MWEs generate an SQL database dump with `sql.php`, or does that have to be done directly on the db server? [19:39:54] dmehus: we can't generate an SQL dump but we can take copies of data [19:40:14] yeah, SQL dumps can only be done by doing mysqldump on a db server [19:40:29] I ended up getting https://phabricator.wikimedia.org/T283595 down to https://github.com/wikimedia/mediawiki-extensions-ApprovedRevs/blob/master/includes/ApprovedRevsHooks.php#L959 as well [19:40:30] [ ⚓ T283595 GlobalPrefrences: ErrorException: PHP Deprecated: A ParserGetVariableValueSwitch hook handler bypassed the cache, this is deprecated since MediaWiki 1.35 ] - phabricator.wikimedia.org [19:40:30] [ mediawiki-extensions-ApprovedRevs/ApprovedRevsHooks.php at master · wikimedia/mediawiki-extensions-ApprovedRevs · GitHub ] - github.com [19:40:37] Or I'm fairly sure that's to blame [19:41:35] RhinosF1: yeah if I can find them. [19:42:49] We stand at 5 blockers so far [19:43:16] RhinosF1, ah, okay, yeah that's what I thought but wasn't sure [19:43:16] Crepusculum: I know I'm giving 50 million tasks but there's upstreaming the title issues too [19:43:29] Reception123: ack [19:43:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.58, 6.42, 6.61 [19:43:41] RhinosF1: I will get to it. [19:43:51] I am doing comments first. [19:44:10] RhinosF1: Does comments need updated on both REL1_35 and REL1_36? [19:44:16] Ty ty [19:44:26] Crepusculum: it uses master on both but ye [19:44:34] ok, thanks! [19:44:37] Theme needs 1.36 too [19:45:37] RhinosF1, ty for apparently solving #T283595 [19:46:59] dmehus: if that's the comments bug then I copied what void said and read a doc [19:47:21] It sounds like we got caught out by a change in title behaviour [19:47:35] Because parsoid [19:47:42] RhinosF1: Tweeki needs updated also if not already done. [19:47:56] * RhinosF1 is starting to hate the parsoid switch given that's causing most of our issues [19:48:13] Crepusculum: I didn't see that so go ahead [19:55:01] RhinosF1, no the ApprovedRevs bug, I saw Void's good spot with the Comments extension yeah [19:55:15] but yeah that's interesting [19:55:27] re: Parsoid switch [19:58:00] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±2] 13https://git.io/JGqnb [19:58:01] [02miraheze/mediawiki] 07Universal-Omega 031f683da - Update Comments and Theme [19:58:18] RhinosF1: ^ [19:59:28] [02mediawiki] 07Universal-Omega closed pull request 03#1991: Update Comments - 13https://git.io/JGtRS [19:59:30] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JGqck [19:59:31] [02miraheze/mediawiki] 07RhinosF1 039291435 - Update Comments (#1991) [19:59:33] [02mediawiki] 07Universal-Omega deleted branch 03comments-fix-localurl - 13https://git.io/vbL5b [19:59:34] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03comments-fix-localurl [20:00:55] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [20:02:04] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 6.98, 6.56 [20:04:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.69, 6.98, 6.60 [20:05:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.83, 7.32, 6.46 [20:05:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.67, 6.85, 6.12 [20:06:01] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.89, 7.69, 6.91 [20:06:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.69, 6.97, 6.37 [20:06:43] Crepusculum: feel free to do Tweeki too :) [20:06:44] But thanks [20:06:55] dmehus: switching a parser is hard work [20:07:20] RhinosF1: Tweeki's PR from paladox was only done on master actually so still needs backported to REL1_36 [20:07:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.59, 7.25, 6.54 [20:07:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.17, 6.75, 6.18 [20:07:41] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [20:07:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.01, 7.29, 6.85 [20:08:32] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.13, 6.76, 6.37 [20:11:21] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:11:25] Crepusculum: okay, feel free to chase up / act [20:11:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.66, 6.00, 6.21 [20:11:33] Or dump in a list to do [20:11:33] RhinosF1, ack [20:11:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.68, 21.68, 17.29 [20:11:57] dmehus: they're doing a really good job tbh, it's just that them doing a good job is annoying for everyone else [20:12:08] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:12:22] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:12:40] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:13:14] Switching to a new parser is long overdue to make MediaWiki a modern piece of software keeping up with technology [20:13:35] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:13:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.26, 20.57, 17.38 [20:13:49] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:13:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.00, 7.28, 6.92 [20:15:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.74, 17.27, 16.56 [20:15:53] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.40, 7.45, 7.04 [20:16:28] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [20:17:10] JohnLewis: I fully agree. It's just triggers many many decepration (I can't spell) warnings [20:18:43] [02mediawiki] 07Universal-Omega opened pull request 03#1992: Update REL1_36 from upstream - 13https://git.io/JGqWN [20:20:21] RhinosF1: how do I update the REL1_36 branch from upstream? keep getting conflicts in submodules when trying. [20:20:47] Crepusculum: manually rebase [20:21:00] I normally pull the branch and then let GitHub desktop do the rebase [20:21:04] Less commands [20:22:08] JohnLewis, oh is there a large upstream task about switching MediaWiki to a new parser? [20:22:24] is "rebasing" the same thing as resolving a GitHub merge conflict? [20:22:48] RhinosF1: doing. [20:23:17] dmehus: somewhere and yes [20:23:48] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.93, 6.38, 6.68 [20:24:37] dmehus: https://phabricator.wikimedia.org/T236809 [20:24:38] [ ⚓ T236809 Refactor Parser.php to allow alternate parser (Parsoid) ] - phabricator.wikimedia.org [20:30:00] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [20:30:01] RhinosF1, ack [20:30:06] thanks [20:33:20] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:34:40] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:36] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:42] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.67, 8.46, 6.72 [20:35:42] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.32, 7.68, 6.97 [20:35:42] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.64, 7.83, 6.47 [20:35:52] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:36:07] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:36:12] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.39, 7.43, 6.39 [20:36:21] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:37:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.52, 7.92, 6.73 [20:39:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.31, 7.85, 6.83 [20:40:27] PROBLEM - wiki.hrznstudio.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hrznstudio.com could not be found [20:41:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.00, 7.47, 6.82 [20:42:48] given that the tagging patch has now been merged, I'd say 1.36 is just about to be released [20:43:19] yeah I saw that, that's why I am rebasing our 1.36 branch. [20:45:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.95, 6.89, 6.90 [20:45:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [20:45:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.78, 6.10, 6.41 [20:47:13] RECOVERY - wiki.hrznstudio.com - reverse DNS on sslhost is OK: rDNS OK - wiki.hrznstudio.com reverse DNS resolves to cp13.miraheze.org [20:47:28] MacFan4000: not seeing any chatter in releng [20:47:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.14, 6.28, 6.67 [20:54:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.97, 7.31, 7.63 [20:54:20] Crepusculum: 2 open blockers :) [20:54:26] For us [20:54:37] We got them WikiBase schema changes Southparkfan found to run too [21:10:06] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.58, 5.97, 6.78 [21:13:10] [02mediawiki] 07wmfgerrit synchronize pull request 03#1992: Update REL1_36 from upstream - 13https://git.io/JGqWN [21:19:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.51, 5.92, 7.73 [21:19:34] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [21:28:16] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [21:29:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.88, 5.57, 6.72 [21:34:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.62, 6.98, 5.73 [21:36:34] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.99, 6.37, 5.65 [21:36:40] RhinosF1: Great! Just created another though (https://github.com/lkucharczyk/mediawiki-PortableInfobox/pull/41) [21:36:42] [ MWTidy::factory no longer exists; gives fatal by Universal-Omega · Pull Request #41 · lkucharczyk/mediawiki-PortableInfobox · GitHub ] - github.com [21:37:01] I saw! [21:37:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 6.88, 6.85 [21:37:27] that is fixable by disabling a config though RhinosF1, though don't know the impact of it. [21:37:39] One blocker down, another added, Crepusculum? [21:37:54] Crepusculum: let's see what response we get upstream [21:37:58] dmehus: yep, that's a way to put it lol [21:38:03] RhinosF1: yeah. [21:38:07] Crepusculum, ah, heh [21:38:13] We can look further if it's too slow [21:38:20] yep. [21:38:26] maybe you can fix it upstream, do you do much work with PortableInfobox? [21:38:32] if we have to I will fork it. [21:38:34] Already done [21:38:40] dmehus: I submitted PR. [21:38:43] Yeah sounds fine [21:38:50] Crepusculum, ah that's great, thanks :) [21:38:54] dmehus: https://github.com/lkucharczyk/mediawiki-PortableInfobox/pull/41 [21:38:54] [ MWTidy::factory no longer exists; gives fatal by Universal-Omega · Pull Request #41 · lkucharczyk/mediawiki-PortableInfobox · GitHub ] - github.com [21:39:02] Crepusculum: ack [21:39:02] We've got a few other blockers so it's got time [21:39:09] And a lot more testing to do [21:39:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.02, 6.52, 6.72 [21:39:43] yep. [21:41:48] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [21:46:13] paladox: are you able to assign https://phabricator.miraheze.org/T7352 to yourself? [21:46:14] [ ⚓ T7352 Significant drop in cache hit rate caused all wikis to go down about 2am UK time 24 May 2021 ] - phabricator.miraheze.org [21:46:42] Southparkfan: not sure what I would do with it? [21:47:08] pinpointing the cause of this incident [21:47:32] an unknown cause caused a chain of issues [21:49:00] varnish is my expertise, but I have no time to focus on it this week [21:49:53] Ok [22:01:31] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.30, 1.78, 1.35 [22:03:26] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.57, 1.40, 1.27 [22:04:04] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [22:37:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [22:41:38] 100% of high priority tasks being assigned to one person is surely concerning, no? How can all three tasks get appropriate attention and resolution if one person is expected to prioritise all three over each other? [22:51:44] I have freed up one task [23:05:56] Crepusculum, paladox: I'm going to try and help test a few extensions on `test3wiki`, and when I tried to view [[mh:test3:Template:Interwiki redirect]], I get: `[512f9af13e86f63e2d2ec48b] 2021-05-27 23:04:59: Fatal exception of type "MediaWiki\Storage\NameTableAccessException"` [23:06:07] do you want me to create a task for that? [23:06:24] Yes [23:06:28] okay [23:06:31] doing now [23:06:41] how should I link it to the MW 1.36 upgrade task? [23:07:05] that is not related to 1.36 upgrade, just a test3 issue that has existed even when 1.35. [23:11:21] okay [23:11:26] so how do we resolve it though? [23:11:50] I'd like to use that template for testing SoftRedirector [23:11:57] but I guess I could just add the magic word [23:15:27] Crepusculum: uhh... why is the magic word __SOFTREDIR__ visible on the page? It's not supposed to be, and it links to SoftRedir. That's not expected and not supposed to happen [23:17:27] [02mediawiki] 07wmfgerrit synchronize pull request 03#1992: Update REL1_36 from upstream - 13https://git.io/JGqWN [23:29:59] [02miraheze/mediawiki] 07Universal-Omega pushed 03112 commits to 03REL1_36_merge [+2/-2/±135] 13https://git.io/JGq9u [23:30:01] [02miraheze/mediawiki] 07AaronSchulz 03589fa3c - rdbms: Use server time in DatabaseMysqlBase::getLagFromPtHeartbeat() [23:30:02] [02miraheze/mediawiki] 07reedy 038d661ff - apihelp-query+iwlinks-param-prop: s/interlanguage/interwiki/ [23:30:04] [02miraheze/mediawiki] 07reedy 034f76eb3 - Delete maintenance/cleanupAncientTables.php [23:30:05] [02miraheze/mediawiki] ... and 109 more commits. [23:30:07] [02mediawiki] 07Universal-Omega created branch 03REL1_36_merge - 13https://git.io/vbL5b [23:30:23] [02mediawiki] 07Universal-Omega deleted branch 03REL1_36_merge - 13https://git.io/vbL5b [23:30:25] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03REL1_36_merge [23:31:33] [02miraheze/mediawiki] 07Universal-Omega pushed 03112 commits to 03REL1_36_merge [+2/-2/±135] 13https://git.io/JGq9u [23:31:34] [02miraheze/mediawiki] 07AaronSchulz 03589fa3c - rdbms: Use server time in DatabaseMysqlBase::getLagFromPtHeartbeat() [23:31:36] [02miraheze/mediawiki] 07reedy 038d661ff - apihelp-query+iwlinks-param-prop: s/interlanguage/interwiki/ [23:31:37] [02miraheze/mediawiki] 07reedy 034f76eb3 - Delete maintenance/cleanupAncientTables.php [23:31:39] [02miraheze/mediawiki] ... and 109 more commits. [23:31:40] [02mediawiki] 07Universal-Omega created branch 03REL1_36_merge - 13https://git.io/vbL5b [23:32:17] [02mediawiki] 07Universal-Omega opened pull request 03#1993: Rel1 36 merge - 13https://git.io/JGq9N [23:40:53] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [23:48:12] [02mediawiki] 07Universal-Omega closed pull request 03#1993: Rel1 36 merge - 13https://git.io/JGq9N [23:48:15] [02mediawiki] 07Universal-Omega deleted branch 03REL1_36_merge - 13https://git.io/vbL5b [23:48:16] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03REL1_36_merge [23:54:29] [02miraheze/mediawiki] 07Universal-Omega pushed 03111 commits to 03REL1_36_update [+3/-3/±187] 13https://git.io/JGq79 [23:54:30] [02miraheze/mediawiki] 07AaronSchulz 03589fa3c - rdbms: Use server time in DatabaseMysqlBase::getLagFromPtHeartbeat() [23:54:32] [02miraheze/mediawiki] 07reedy 038d661ff - apihelp-query+iwlinks-param-prop: s/interlanguage/interwiki/ [23:54:33] [02miraheze/mediawiki] 07reedy 034f76eb3 - Delete maintenance/cleanupAncientTables.php [23:54:35] [02miraheze/mediawiki] ... and 108 more commits. [23:54:36] [02mediawiki] 07Universal-Omega created branch 03REL1_36_update - 13https://git.io/vbL5b [23:55:27] [02mediawiki] 07Universal-Omega opened pull request 03#1994: Merge remote-tracking branch 'upstream/REL1_36' into REL1_36_update - 13https://git.io/JGq7N