[02:09:50] (03PS11) 10AndyRussG: E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) [04:31:44] (03PS12) 10AndyRussG: E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) [06:14:02] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_35) - 10https://gerrit.wikimedia.org/r/764039 (owner: 10L10n-bot) [06:42:12] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_35) - 10https://gerrit.wikimedia.org/r/764039 (owner: 10L10n-bot) [11:42:14] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 2 [=1],minFraud_endpoint_warning 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [11:47:12] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [11:52:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1],minFraud_endpoint_warning 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [11:57:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 2 [=1],minFraud_endpoint_warning 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [12:02:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 2 [=1],minFraud_endpoint_warning 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [12:07:12] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [12:17:12] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 2 [=1],minFraud_endpoint_warning 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [12:22:16] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [12:37:14] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1],minFraud_endpoint_warning 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [12:47:12] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1],minFraud_endpoint_warning 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [12:52:11] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [12:57:12] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1],minFraud_endpoint_warning 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [13:12:11] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [13:47:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 4 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [13:47:59] (03PS13) 10Damilare Adedoyin: E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [13:48:11] (03CR) 10jerkins-bot: [V: 04-1] E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [13:52:10] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:07:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1],minFraud_endpoint_warning 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:17:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 2 [=1],minFraud_endpoint_warning 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:22:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:25:33] looks like we've got some issues on the minfraud health check [14:25:36] fr-tech [14:26:25] it would be great if we could somehow link the icinga alerts to the job files that they are checking results of to make it easy enough to replicate [14:26:47] I guess quick check could be to try hiting the minfraud endpoint we use in payments [14:27:02] I'm guessing that's in localsettings [14:27:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1],minFraud_endpoint_warning 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:29:28] hmm maybe not [14:29:41] let's check the minfraud filter in the code [14:30:33] it probably wouldn't hurt to also check the error logs to see if anything good is showing up in there [14:31:08] Received a server error (504) for Score [14:32:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 7 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:37:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 2 [=1],minFraud_endpoint_warning 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:42:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 3 [=1],minFraud_endpoint_warning 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:47:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:52:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 4 [=1],minFraud_endpoint_warning 2 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:53:29] ACKNOWLEDGEMENT - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 4 [=1],minFraud_endpoint_warning 2 [=1] Jack_Gleeson minfraud scheduled maintenance https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [15:27:09] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [16:56:09] (03PS14) 10Damilare Adedoyin: E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [16:56:52] (03PS15) 10Damilare Adedoyin: E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [17:01:45] (03CR) 10jerkins-bot: [V: 04-1] E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [17:03:26] (03PS16) 10Damilare Adedoyin: E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [17:07:01] (03CR) 10jerkins-bot: [V: 04-1] E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [17:15:56] (03PS17) 10Damilare Adedoyin: E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [17:18:23] (03CR) 10jerkins-bot: [V: 04-1] E-mail Preferences Center header, footer and styling [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/758283 (https://phabricator.wikimedia.org/T290986) (owner: 10AndyRussG) [20:01:30] 10Fundraising Sprint Cheese String Theory, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email: Civi Searchkit should allow user to export list to acoustic - https://phabricator.wikimedia.org/T301514 (10Eileenmcnaughton) a:03Eileenmcnaughton [20:02:10] 10Fundraising Sprint Cheese String Theory, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising Sprint Black Hole Fillings: Can't view the stock information for stock gifts - https://phabricator.wikimedia.org/T300732 (10Eileenmcnaughton) a:03Eileenmcnaughton [20:09:56] (03CR) 10Eileen: "Elliot this is how you make it available to be managed via the contact summary rather than a mandatory tab - although it might mean we wan" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/763386 (https://phabricator.wikimedia.org/T301289) (owner: 10Ejegg) [20:34:08] hmm - I'm in the frtect/ops meeting but no-one else is - what am I missing [20:43:11] Hi eileen, it's a holiday in the US today. [20:43:19] damilare: ah ok [20:43:21] thanks [22:04:27] 10WMSE-Content-partnerships-support-2021-Fundraising, 10User-Eric_Luth_WMSE: Update the main page of the Wikimedia Sverige wiki - https://phabricator.wikimedia.org/T250665 (10Jopparn) [22:38:05] 10WMSE-Content-partnerships-support-2021-Fundraising: Create handout material for volunteers - https://phabricator.wikimedia.org/T282242 (10Jopparn) [22:46:10] 10WMSE-Content-partnerships-support-2021-Fundraising, 10User-Eric_Luth_WMSE: Propose means of finding out amount of visitors at se.wikimedia.org - https://phabricator.wikimedia.org/T262557 (10Jopparn)