[00:01:14] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.77, 6.56, 6.37 [00:02:06] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.52, 2.06, 1.80 [00:02:12] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.69, 6.68, 6.70 [00:03:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 34.13, 24.03, 18.90 [00:03:28] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:45] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.50, 8.29, 7.01 [00:03:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.40, 7.99, 6.66 [00:04:13] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.52, 1.92, 1.79 [00:04:44] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.93, 8.15, 7.02 [00:04:52] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.49, 3.66, 2.70 [00:05:00] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 14.96, 20.86, 21.30 [00:06:15] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.13, 1.58, 1.68 [00:06:54] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.87, 3.36, 2.70 [00:10:56] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.43, 17.75, 19.82 [00:12:13] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.12, 1.91, 1.79 [00:14:20] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.97, 1.54, 1.67 [00:14:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.40, 7.01, 6.60 [00:15:45] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.63, 7.93, 7.93 [00:16:25] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.13, 6.20, 6.35 [00:17:24] [02puppet] 07Universal-Omega edited pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEQyr [00:19:28] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.57, 6.16, 4.65 [00:19:51] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.43, 9.50, 8.48 [00:21:31] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.74, 3.75, 3.17 [00:22:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.66, 7.51, 7.87 [00:23:34] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.01, 3.58, 3.19 [00:25:35] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.20, 3.09, 3.05 [00:25:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.04, 5.57, 5.02 [00:25:58] [02puppet] 07Universal-Omega synchronize pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEQyr [00:26:56] [02puppet] 07Universal-Omega synchronize pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEQyr [00:26:58] [02puppet] 07Universal-Omega synchronize pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEQyr [00:27:14] [02puppet] 07Universal-Omega synchronize pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEQyr [00:27:34] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.33, 4.63, 4.75 [00:28:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.60, 3.17 [00:28:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.22, 6.56, 7.16 [00:29:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.40, 7.31, 6.25 [00:29:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.55, 7.48, 7.94 [00:29:42] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.40, 7.36, 6.53 [00:29:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.11, 6.50, 6.00 [00:30:34] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.87, 3.26, 3.09 [00:30:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.10, 6.79, 7.18 [00:31:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.23, 7.49, 6.46 [00:31:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 6.72, 6.15 [00:32:09] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.03, 7.98, 8.04 [00:33:40] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.81, 7.27, 6.71 [00:33:42] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.94, 6.55, 6.16 [00:34:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.79, 3.37 [00:34:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.99, 7.66, 7.39 [00:35:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.97, 6.42, 6.30 [00:37:31] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.60, 7.79, 7.99 [00:37:41] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.66, 6.65, 6.58 [00:38:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.82, 6.90, 7.21 [00:40:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.75, 3.50 [00:41:32] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.13, 7.27, 7.93 [00:42:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 3.96, 3.61 [00:42:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.26, 6.88, 6.62 [00:43:28] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.37, 7.95, 8.10 [00:44:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.50, 3.48 [00:44:28] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.70, 6.13, 6.37 [00:45:24] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.84, 7.33, 7.87 [00:46:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.87, 2.95, 3.28 [00:46:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.98, 5.95, 6.80 [00:47:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.66, 5.08, 6.60 [00:48:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.14, 19.81, 23.26 [00:49:18] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.17, 6.76, 7.43 [00:53:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.90, 3.70, 3.51 [00:53:14] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.65, 6.22, 7.11 [00:57:07] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.93, 5.19, 6.50 [00:58:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.87, 15.75, 19.74 [01:02:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.22, 3.39 [01:05:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.04, 4.36, 3.49 [01:06:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.22, 22.90, 20.99 [01:07:44] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.46, 3.73, 3.35 [01:08:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.18, 21.88, 20.83 [01:12:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.72, 19.27, 19.97 [01:22:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.56, 3.32 [01:24:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.87, 3.46 [01:26:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.42, 3.35 [01:30:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.27, 3.32 [01:34:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.26, 18.65, 17.68 [01:36:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.44, 3.31 [01:36:30] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.71, 17.95, 17.56 [01:40:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.37, 3.40, 3.33 [01:55:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.65, 3.39 [01:55:28] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.79, 1.60, 1.36 [01:57:27] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.88, 1.32, 1.29 [02:04:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.60, 4.09, 3.62 [02:06:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.17, 7.07, 5.45 [02:06:30] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.42, 2.37, 1.74 [02:06:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.98, 3.65 [02:08:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.33, 6.28, 5.34 [02:08:31] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.32, 1.91, 1.64 [02:14:29] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.57, 1.65, 1.61 [02:16:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.71, 3.74, 3.60 [02:18:28] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.51, 1.72, 1.66 [02:20:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.78, 3.66 [02:20:32] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.32, 1.59, 1.62 [02:24:33] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.23, 2.56, 2.07 [02:28:29] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.75, 1.68, 1.83 [02:32:27] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.87, 1.32, 1.65 [02:35:34] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.57, 6.61, 5.49 [02:36:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.07, 6.13, 5.01 [02:36:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.65, 3.61 [02:37:23] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.98, 6.94, 5.58 [02:37:29] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.99, 6.50, 5.60 [02:38:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.19, 5.39, 4.88 [02:38:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.44, 3.32, 3.51 [02:39:21] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.42, 6.08, 5.43 [02:42:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.02, 3.35 [02:52:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JE7Z0 [02:52:11] [02miraheze/puppet] 07paladox 03872d209 - db: Increase table_definition_cache and table_open_cache to 50000 [02:52:12] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:52:14] [02puppet] 07paladox opened pull request 03#1920: db: Increase table_definition_cache and table_open_cache to 50000 - 13https://git.io/JE7ZE [02:52:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JE7Z2 [02:52:40] [02miraheze/puppet] 07paladox 03e16ba60 - Update db12.yaml [02:52:42] [02puppet] 07paladox synchronize pull request 03#1920: db: Increase table_definition_cache and table_open_cache to 50000 - 13https://git.io/JE7ZE [02:54:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JE7ZQ [02:54:24] [02miraheze/puppet] 07paladox 03c47a880 - Update db13.yaml [02:54:25] [02puppet] 07paladox synchronize pull request 03#1920: db: Increase table_definition_cache and table_open_cache to 50000 - 13https://git.io/JE7ZE [02:54:32] [02puppet] 07paladox closed pull request 03#1920: db: Increase table_definition_cache and table_open_cache to 50000 - 13https://git.io/JE7ZE [02:54:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JE7Z5 [02:54:35] [02miraheze/puppet] 07paladox 0303db48b - db: Increase table_definition_cache and table_open_cache to 50000 (#1920) [02:54:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [02:54:38] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:55:15] !log db11: >set global table_open_cache=50000; [02:55:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:22] !log db11: >set global table_definition_cache=50000; [02:55:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:45] !log db12: >set global table_open_cache=50000; [02:55:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:49] !log db12: >set global table_definition_cache=50000; [02:55:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:13] !log db13: >set global table_open_cache=50000; [02:56:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:15] !log db13: >set global table_definition_cache=50000; [02:56:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.18, 3.84, 3.51 [02:59:20] [02puppet] 07Universal-Omega synchronize pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEQyr [02:59:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.47, 3.41 [03:03:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.23, 3.33 [03:06:29] !log restart prometheus-blackbox-exporter [03:06:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JE78y [03:24:39] [02miraheze/puppet] 07paladox 03e5f6aff - mediawiki: Update composer to version 2 [03:24:41] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [03:24:42] [02puppet] 07paladox opened pull request 03#1921: mediawiki: Update composer to version 2 - 13https://git.io/JE789 [03:41:53] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JE7Rh [03:41:54] [02miraheze/CreateWiki] 07paladox 038bd00c2 - Add support for composer version 2 [03:41:56] [02CreateWiki] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vpJTL [03:41:57] [02CreateWiki] 07paladox opened pull request 03#238: Add support for composer version 2 - 13https://git.io/JE7Rj [03:42:18] [02CreateWiki] 07paladox closed pull request 03#238: Add support for composer version 2 - 13https://git.io/JE7Rj [03:42:19] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JE70f [03:42:21] [02miraheze/CreateWiki] 07paladox 0336a1f88 - Add support for composer version 2 (#238) [03:42:22] [02CreateWiki] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vpJTL [03:42:24] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-2 [03:42:53] miraheze/CreateWiki - paladox the build passed. [03:43:12] miraheze/CreateWiki - paladox the build passed. [03:43:38] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JE70Z [03:43:39] [02miraheze/mediawiki] 07paladox 032df7659 - Update CreateWiki [03:45:06] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [03:45:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:12] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - FAIL: [2, 2] in 6s [03:45:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:25] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [03:45:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:09] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - FAIL: [256, 0] in 104s [03:47:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:44] !log [paladox@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [03:47:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:21] !log [paladox@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 36s [03:48:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:46] [02puppet] 07Universal-Omega commented on pull request 03#1921: mediawiki: Update composer to version 2 - 13https://git.io/JE7E4 [03:48:50] !log [paladox@test3] starting deploy of {'config': True, 'world': True, 'gitinfo': True} to skip [03:48:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:30] !log [paladox@test3] finished deploy of {'config': True, 'world': True, 'gitinfo': True} to skip - SUCCESS in 40s [03:49:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:52:07] [02puppet] 07paladox commented on pull request 03#1921: mediawiki: Update composer to version 2 - 13https://git.io/JE7uk [03:55:10] [02puppet] 07paladox commented on pull request 03#1921: mediawiki: Update composer to version 2 - 13https://git.io/JE7uF [03:56:37] [02puppet] 07paladox commented on pull request 03#1892: (DO NOT MERGE) Update to composer v2 and use the Debian native package - 13https://git.io/JE7z3 [04:11:58] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:12:36] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:14:54] [02puppet] 07paladox synchronize pull request 03#1892: (DO NOT MERGE) Update to composer v2 and use the Debian native package - 13https://git.io/JE02y [04:15:45] [02puppet] 07paladox synchronize pull request 03#1892: (DO NOT MERGE) Update to composer v2 and use the Debian native package - 13https://git.io/JE02y [04:16:03] [02puppet] 07paladox edited pull request 03#1892: Update to composer v2 - 13https://git.io/JE02y [04:22:20] !log [paladox@test3] starting deploy of {'config': True, 'world': True, 'gitinfo': True} to skip [04:22:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:22:56] !log [paladox@test3] finished deploy of {'config': True, 'world': True, 'gitinfo': True} to skip - SUCCESS in 35s [04:23:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:24:32] [02puppet] 07paladox commented on pull request 03#1921: mediawiki: Update composer to version 2 - 13https://git.io/JE7wy [04:24:34] [02puppet] 07paladox closed pull request 03#1921: mediawiki: Update composer to version 2 - 13https://git.io/JE789 [04:24:43] [02puppet] 07paladox edited a comment on pull request 03#1921: mediawiki: Update composer to version 2 - 13https://git.io/JE7wy [04:24:45] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [04:24:46] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [04:25:15] [02puppet] 07paladox closed pull request 03#1892: Update to composer v2 - 13https://git.io/JE02y [04:25:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JE7wd [04:25:18] [02miraheze/puppet] 07MacFan4000 03834ef94 - Update to composer v2 (#1892) [04:27:59] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [04:29:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JE7rS [04:29:31] [02miraheze/puppet] 07paladox 03aab6457 - mediawiki: chmod composer as 0755 [04:29:33] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [04:29:34] [02puppet] 07paladox opened pull request 03#1922: mediawiki: chmod composer as 0755 - 13https://git.io/JE7rQ [04:29:40] [02puppet] 07paladox closed pull request 03#1922: mediawiki: chmod composer as 0755 - 13https://git.io/JE7rQ [04:29:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JE7rF [04:29:43] [02miraheze/puppet] 07paladox 032e92988 - mediawiki: chmod composer as 0755 (#1922) [04:29:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [04:29:46] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [04:32:12] [02puppet] 07Universal-Omega opened pull request 03#1923: CSP Whitelist: don't use unnecessary wildcards - 13https://git.io/JE7oc [04:34:34] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:35:37] !log depool mw8 [04:35:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:35:46] !log upgrade mw8 to composer version 2 [04:35:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:46] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw8 [04:38:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8 [04:38:47] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw8 [04:39:27] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8 [04:39:56] !log repool mw8 [04:39:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:44] !log depool mw9 [04:40:46] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [04:40:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:47] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [04:40:50] !log upgrade mw9 to composer version 2 [04:40:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:42:47] [02puppet] 07Reception123 closed pull request 03#1923: CSP Whitelist: don't use unnecessary wildcards - 13https://git.io/JE7oc [04:42:49] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JE76c [04:42:50] [02miraheze/puppet] 07Universal-Omega 03ad61cec - CSP Whitelist: don't use unnecessary wildcards (#1923) [04:43:39] paladox: thanks for upgrading to composer v2 :) [04:43:47] yw [04:44:49] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw9 [04:44:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw9 [04:45:22] !log repool mw9 [04:45:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:45:35] !log depool mw10 [04:45:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:47:52] !log upgrade mw10 to composer version 2 [04:47:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:49:28] !log repool mw10 [04:49:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:49:57] !log depool mw11 [04:50:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:50:02] !log upgrade mw11 to composer version 2 [04:50:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:21] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [04:53:33] !log repool mw11 [04:53:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:57] !log depool mw12 [04:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:54:01] !log upgrade mw12 to composer version 2 [04:54:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:09] !log repool mw12 [04:57:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:12] !log depool mw13 [04:57:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:17] !log upgrade mw13 to composer version 2 [04:57:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:58:47] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [05:00:32] !log repool mw13 [05:00:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:00:45] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [05:00:50] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [05:08:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JE7Me [05:08:05] [02miraheze/CreateWiki] 07Universal-Omega 030128416 - Send email if wiki is closed when already marked inactive [05:08:06] [02CreateWiki] 07Universal-Omega synchronize pull request 03#233: Cleanup Echo notifications code and fix Echo email notifications - 13https://git.io/JBSa3 [05:08:19] [02puppet] 07paladox closed pull request 03#1915: Revert "Remove db12 from bacula" - 13https://git.io/JEHo1 [05:08:21] [02puppet] 07paladox deleted branch 03revert-1716-paladox-patch-4 - 13https://git.io/vbiAS [05:08:22] [02miraheze/puppet] 07paladox deleted branch 03revert-1716-paladox-patch-4 [05:08:40] [02CreateWiki] 07Universal-Omega edited pull request 03#233: Cleanup Echo notifications code and fix Echo email notifications - 13https://git.io/JBSa3 [05:08:45] [02puppet] 07paladox closed pull request 03#1908: mediawiki: Remove ocaml - 13https://git.io/JEogC [05:08:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [05:08:49] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [05:09:04] miraheze/CreateWiki - Universal-Omega the build passed. [05:30:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JE79C [05:30:37] [02miraheze/puppet] 07paladox 03f0e7ec0 - varnish: Set max threads to at least 500+ [05:30:39] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [05:30:40] [02puppet] 07paladox opened pull request 03#1924: varnish: Set max threads to at least 500+ - 13https://git.io/JE79W [05:31:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JE79r [05:32:00] [02miraheze/puppet] 07paladox 03df45b02 - Update init.pp [05:32:02] [02puppet] 07paladox synchronize pull request 03#1924: varnish: Set max threads to at least 500+ - 13https://git.io/JE79W [05:32:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JE79X [05:32:32] [02miraheze/puppet] 07paladox 0371be565 - Update varnish.systemd.erb [05:32:33] [02puppet] 07paladox synchronize pull request 03#1924: varnish: Set max threads to at least 500+ - 13https://git.io/JE79W [05:33:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JE79H [05:33:09] [02miraheze/puppet] 07paladox 032944f7a - Update init.pp [05:33:10] [02puppet] 07paladox synchronize pull request 03#1924: varnish: Set max threads to at least 500+ - 13https://git.io/JE79W [05:33:22] [02puppet] 07paladox edited pull request 03#1924: varnish: Set max threads to at least 500+ - 13https://git.io/JE79W [05:33:25] [02puppet] 07paladox closed pull request 03#1924: varnish: Set max threads to at least 500+ - 13https://git.io/JE79W [05:33:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JE79x [05:33:28] [02miraheze/puppet] 07paladox 03b1140b8 - varnish: Set max threads to at least 500+ (#1924) [05:33:30] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [05:33:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [05:36:39] !log restart varnish on cp12 [05:36:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:42:02] !log sudo chown -R www-data:www-data /mnt/mediawiki-static/402611wiki [05:42:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:40] paladox: did you also update composer on mwtask1 and test3? I just didn't see any logs for those so wasn't sure if you did or not. I understand test3 doesn't really need logged either so maybe it was but unsure of mwtask1. [08:30:26] https://www.irccloud.com/pastebin/ZcIjgAoJ/ [08:30:34] hmm getting this again for an import [08:34:26] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=polandballhiwiki /home/reception/hipolandball_pages_full.xml --username-prefix wikia:polandball --report 1 (END - exit=0) [08:34:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:49] Use --no-updates or whatever it is if it wants to be persistent [08:35:56] Then rebuildall after [09:48:36] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEd8z [09:48:38] [02miraheze/puppet] 07Reception123 0327e786a - remove cloudytheology.com CSP entry per T7893 [09:54:33] [02puppet] 07JohnFLewis commented on pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEd4S [10:06:05] [02puppet] 07RhinosF1 commented on pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEd0l [10:18:32] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEdzX [10:18:34] [02miraheze/puppet] 07Reception123 039ebe8db - remove facebook from CSP per T7897 [10:19:37] [02puppet] 07JohnFLewis commented on pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEdge [10:21:11] [02puppet] 07JohnFLewis edited a comment on pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEdge [10:26:55] [02puppet] 07RhinosF1 commented on pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEd2P [10:28:04] JohnLewis: what I wanted to do is reduce me from having to look at multiple dashboards to identify whether any servers had low workers to having to look at one single place. All we've done so far is add an email alert (that we can't get to actually tell us anything useful). [10:29:03] More data = more informed decisions. Reducing the amount of information available to you is not an effective way at debugging and seeking to solve a problem [10:30:07] JohnLewis: how's it reducing data Available [10:30:29] I'm just moving the data so it's on the same screen as the alerts saying servers are depooled [10:30:30] Because you’ve stated you want to look in less places, when one of the places involved contains all the data [10:30:48] JohnLewis: but the checks are showing the same information [10:31:13] And merging the above, would introduce another check showing the same information [10:31:48] JohnLewis: but it's then going to be visible right next to the alert saying we're down [10:31:57] So I have far less searching to do to find why [10:32:21] But you would need to access grafana to find out why, which is where the alert will be… [10:32:56] The use case is not convincing to essentially decide grafana should be binned for alerting over icinga when Grafana is more advanced at handling these kinds of alerts [10:33:46] JohnLewis: my point is we should have one single solution that can feed as icinga does to irc & email [10:33:56] One of the pushes I want to do is more informed and advanced monitoring, unfortunately, you require monitoring in place to be able to find the short comings to resolve them and make it more useful. Pushing alerts away from the ‘new system’ defeats the point [10:34:01] (And I quite like to think one day SMS too for very important things) [10:34:24] My point is I want 1 dashboard that shows all alerts currently active [10:35:14] You can’t have ‘one dashboard’ unless you build it right now I’m afraid, we need to do smart alerting and monitoring, but cheaper and easier [10:35:25] *not cheaper [10:35:56] You have two data sources with valuable information, using both is critical, and both will tell you different things and have their own useful takeaways [10:36:12] Can we look at whether grafana can actually work then in sending useful alerts not an email that doesn't tell me which server is broken and nothing on irc [10:36:23] My ticket explicitly mentioned alerts going to irc [10:36:36] Read my above messages [10:36:44] Where that is addressed [10:37:34] If you need it so badly right now, you are free to take on the work to do so - but unfortunately priorities of other work is taking precedence over ‘nice to haves’, which that monitoring alert in the first place was [10:38:13] JohnLewis: is there any actual issue with that PR until we can implement working grafana alerts [10:38:27] Yes, as mentioned in the PR [10:38:29] Most of the logic is stolen from the stunnel ones [10:39:05] JohnLewis: "Manually updating the list is not a good use of puppet’s power and creates manual work and technical debt." is exactly what the other mediawiki checks do [10:39:12] But I am not merging it, and neither will Infra as it purely duplicates existing monitoring with no added benefit, contrary to what we want to achieve [10:39:56] JohnLewis: it does have added benefit [10:40:08] Providing alerts in the event of an outage in a single place [10:40:12] No, it does not as explained above. [10:40:33] [02puppet] 07JohnFLewis commented on pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEdrJ [10:40:34] [02puppet] 07JohnFLewis closed pull request 03#1919: Add PHP-FPM icinga check - 13https://git.io/JEQyr [10:40:50] Yes and I've said from grafana we have no irc dashboard, no single way to view alerts because there's only one there and broken emails [10:41:33] And as I said, other priorities are taking precedence. If it is so important to you, you are free to take on the work. Otherwise, it will sit until there is sufficient time and resources available like any other low priority work [10:43:07] JohnLewis: can we reopen the task for the alerts so it can be tracked for us as I'm pointing out what you've provided is not useful [10:43:44] The task was to provide monitoring, which it has. If you want to reopen the task, I’ll delete the grafana alert if it’s ‘not useful’ to you then [10:44:44] In the state it's in, no sending an email with no information on is not useful [10:45:07] All that does is email me to tell me I should go looking at the grafana dashboards [10:45:12] You have been told you have the ability to fix it [10:45:29] So we can have a task for tracking fixing it [10:45:32] If it’s so important than it can not wait on Infra to have time available, you are able to resolve it [10:45:53] And adding irc alerts which was an explicit part of my request [10:45:56] Fixing it would fall under MW-SRE as it is MW-SRE monitoring [10:46:12] Then I'll punt it to us [10:54:58] Sounds fine, I’ll take it off Infra’s priorities list then if you’ll deal with it [10:56:36] Then that's that [11:35:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.81, 5.69, 4.87 [11:37:11] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.64, 5.03, 4.72 [11:45:45] RhinosF1: you can display alerts directly on dashboards as an FYI if you didn't know [11:46:36] JohnLewis: yeah I'm aware. I've seen wikidata's [11:47:10] If we move to grafana for alerts something like that covers stunnel status, rendering, php-fpm would be cool [11:47:49] They have https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts?orgId=1&refresh=30s [11:47:50] [url] Grafana | grafana.wikimedia.org [11:48:08] Then what they do is have an alert to irc if any alert goes off [11:49:33] There's a lot of monitoring opportunities in Grafana that have been missed for years [11:50:05] 16:08:15 PROBLEM - wikidata-alerts grafana alert on alert1001 is CRITICAL: CRITICAL: Wikidata Alerts ( https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts ) is alerting: Max Lag above 10 for 1 hour. https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts https://grafana.wikimedia.org/d/TUJ0V-0Zk/ [11:50:06] [url] Grafana | grafana.wikimedia.org [11:50:07] [url] WMDE/Wikidata/Alerts - Wikitech | wikitech.wikimedia.org [11:50:08] [url] Grafana | grafana.wikimedia.org [11:50:50] JohnLewis: if we could get enough alerts in grafana for important in an outage and pop it on a dashboard, that would be much better [11:51:02] We just need to fix the server detection [11:51:32] Alerts are dashboard specific unfortunately [11:51:32] Then we can get rid of a lot of icinga stuff and fold it into one message like I pasted [11:52:34] JohnLewis: I mean more than one dashboard isn't bad if the alerts work and they're linked well [11:53:03] Could we fold mediawiki rendering & php-fpm maybe into a php dashboard [11:53:09] There's numerous dashboard currently per service [11:53:23] Yeah we might need to clean stuff up [11:54:15] They have been cleaned up recently to be fair, the problem is people importing dashboards rather than building them [11:54:40] Which is a problem in general for a lot of things, people importing others work rather than meaningfully building them [11:54:48] Would a single mediawiki dashboard that shows important data be possible [11:55:09] Things like can mediawiki render, response times, php-fpm status [11:55:10] You're in the MW team so... you'd have to answer that one [11:55:20] I know nothing about grafana [11:55:45] I know we have response times & php-fpm in two dashboards [11:56:42] Probably do, I haven't really taken a big look into the metrics available and their usefulness for MW stuff [11:56:44] Something based on https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?orgId=1 but with alerts [11:56:45] [url] Grafana | grafana.wikimedia.org [11:57:10] That's something we should look at if we want to use grafana for alerts [11:57:21] Feel free to [11:57:46] [02puppet] 07MacFan4000 commented on pull request 03#1892: Update to composer v2 - 13https://git.io/JEd7k [11:58:17] [02puppet] 07RhinosF1 commented on pull request 03#1892: Update to composer v2 - 13https://git.io/JEd7Y [12:00:23] JohnLewis: why does https://grafana.miraheze.org/dashboards/f/WokTzdnMz/application-servers only show job queue [12:00:23] [url] Grafana | grafana.miraheze.org [12:00:37] We have php too that should be under it [12:00:56] I see 3 dashboards listed [12:01:21] I see 1 [12:01:38] Oh mobile [12:01:48]