[02:56:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1292 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:01:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1592 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:06:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1893 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:11:14] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 873568 Threads: 14 Questions: 162131604 Slow queries: 2290754 Opens: 3207 Open tables: 1743 Queries per second avg: 185.597 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [06:05:06] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_35) - 10https://gerrit.wikimedia.org/r/942874 (owner: 10L10n-bot) [11:30:07] 10Wikimedia-Fundraising, 10MediaWiki-extensions-CentralNotice, 10MediaWiki-extensions-Translate: Allow community without extended rights to help with CN banner translations going 'live' - https://phabricator.wikimedia.org/T341414 (10Nikerabbit) It seems there is nothing to change in the Translate extension.... [15:23:55] 10Fundraising-Backlog, 10fr-donorservices: Problems donating via PayPal from France 07/24 - https://phabricator.wikimedia.org/T342652 (10MBeat33) [15:38:17] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T342879 (10KHaggard) Got it, Thanks @Cstone - we will continue to monitor this week and work with you where needed! [16:06:59] (03PS1) 10Damilare Adedoyin: Can only flip STRING and INTEGER values. Ensure variable contains array before flipping [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/943608 (https://phabricator.wikimedia.org/T340061) [16:21:52] (03CR) 10CI reject: [V: 04-1] Can only flip STRING and INTEGER values. Ensure variable contains array before flipping [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/943608 (https://phabricator.wikimedia.org/T340061) (owner: 10Damilare Adedoyin) [16:31:24] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10Patch-For-Review, 10acl*WMF-FR: Braintree Refund implement and write a batch maintenance script - https://phabricator.wikimedia.org/T342788 (10AnnWF) https://sandbox.braintreegateway.com/merchants/ms9fqmw9gxrm2bp3/transactions/8wk724... [16:50:43] (03PS1) 10Wfan: Check if payload.details.payerInfo not undefined [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942796 (https://phabricator.wikimedia.org/T340799) [16:52:26] (03CR) 10CI reject: [V: 04-1] Check if payload.details.payerInfo not undefined [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942796 (https://phabricator.wikimedia.org/T340799) (owner: 10Wfan) [17:08:18] 10Fundraising-Backlog, 10fr-donorservices: Problems donating via PayPal from France 07/24 - https://phabricator.wikimedia.org/T342652 (10EMartin) @AMJohnson Paypal did not advise of any issue as they attempted to reproduce the problem. Everything is functioning ok in France, they report: From Paypal: Hi Eve... [17:29:55] (03PS5) 10Wfan: Add refund script for braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942513 (https://phabricator.wikimedia.org/T342788) [17:30:28] (03CR) 10CI reject: [V: 04-1] Add refund script for braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942513 (https://phabricator.wikimedia.org/T342788) (owner: 10Wfan) [19:31:38] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Update segmentation criteria to new field spec - https://phabricator.wikimedia.org/T342165 (10Eileenmcnaughton) 05Open→03Resolved [19:31:40] 10Fundraising Sprint Istanbool (not Constantidouble), 10Fundraising Sprint Last Christmas I gave you my donation, 10Fundraising-Backlog: Investigation: Scope Segmentation in Civi & Acoustic - https://phabricator.wikimedia.org/T335376 (10Eileenmcnaughton) [19:31:45] 10Fundraising-Backlog, 10Recurring-Donations, 10fr-donorservices, 10Epic: How can we enable upgrading of existing recurring donation amount? - https://phabricator.wikimedia.org/T325317 (10XenoRyet) [19:31:48] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog: Fixing recurring amounts fields to account for reactivating expired donations and downgrading amounts - https://phabricator.wikimedia.org/T338917 (10XenoRyet) 05Open→03Resolved [19:31:59] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog: Figure out how to schedule preliminary updating of segment data - https://phabricator.wikimedia.org/T339186 (10Eileenmcnaughton) 05Open→03Resolved [19:32:01] 10Fundraising Sprint Istanbool (not Constantidouble), 10Fundraising Sprint Last Christmas I gave you my donation, 10Fundraising-Backlog: Investigation: Scope Segmentation in Civi & Acoustic - https://phabricator.wikimedia.org/T335376 (10Eileenmcnaughton) [19:32:16] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog: Engage DAF import- duplicate issue - https://phabricator.wikimedia.org/T337894 (10Eileenmcnaughton) 05Open→03Resolved [19:32:43] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Recurring-Donations, and 2 others: Handle recurring upgrade decline from email link and recurring upgrade form - https://phabricator.wikimedia.org/T335912 (10XenoRyet) 05Open→03Resolved [19:32:46] 10Fundraising-Backlog, 10Recurring-Donations, 10fr-donorservices, 10Epic: How can we enable upgrading of existing recurring donation amount? - https://phabricator.wikimedia.org/T325317 (10XenoRyet) [19:32:48] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10fr-donorservices: Paypal EC Batch refund script to smashpig - https://phabricator.wikimedia.org/T342401 (10XenoRyet) 05Open→03Resolved [19:32:52] 10Fundraising Sprint Never gonna git you update, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Matching gift import - Catching duplicates where cannot be confirmed - https://phabricator.wikimedia.org/T339976 (10Eileenmcnaughton) 05Open→03Resolved [19:33:20] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10fr-donorservices: Civi - Recurring Donation Field Change related to Upgrade - https://phabricator.wikimedia.org/T341877 (10Eileenmcnaughton) 05Open→03Resolved [19:33:40] 10Fundraising-Backlog: Need a list of active monthly recurring subscription for India - https://phabricator.wikimedia.org/T342328 (10Cstone) @RKumar_WMF this what we were missing, it's work to be done in {T339181}. We don't have a way to cancel these yet. [19:33:42] 10Fundraising Sprint Never gonna git you update, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Civi Forget Me button error - https://phabricator.wikimedia.org/T341633 (10Eileenmcnaughton) 05Open→03Resolved [19:34:02] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations, 10fr-donorservices: Update CiviProxy config to allow new API action for recurring upgrade - https://phabricator.wikimedia.org/T335794 (10XenoRyet) 05Open→03Resolved [19:34:05] 10Fundraising-Backlog, 10Recurring-Donations, 10fr-donorservices, 10Epic: How can we enable upgrading of existing recurring donation amount? - https://phabricator.wikimedia.org/T325317 (10XenoRyet) [19:34:08] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog: Goal - Sprint L 2 - Handoff - https://phabricator.wikimedia.org/T338271 (10XenoRyet) 05Open→03Resolved [19:34:10] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog: Major Gifts team update - no auto Civi TYL suppression - https://phabricator.wikimedia.org/T341263 (10Eileenmcnaughton) 05Open→03Resolved [19:34:23] 10Fundraising Sprint Never gonna git you update, 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fr-donorservices: Donor relations - likely permissions issue on contribution tracking - https://phabricator.wikimedia.org/T341834 (10XenoRyet) 05Open→03Resolved [19:35:10] 10Fundraising-Backlog, 10Recurring-Donations, 10fr-donorservices, 10Epic: How can we enable upgrading of existing recurring donation amount? - https://phabricator.wikimedia.org/T325317 (10XenoRyet) [19:36:00] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations, and 2 others: Create patch for CiviProxy to support API4 - https://phabricator.wikimedia.org/T336816 (10XenoRyet) 05Open→03Resolved [19:37:50] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Matching import - duplicate contribution error - https://phabricator.wikimedia.org/T339050 (10XenoRyet) 05Open→03Resolved [19:38:24] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Are we creating ANY CiviMail records from the TY mailer? - https://phabricator.wikimedia.org/T340064 (10XenoRyet) 05Open→03Resolved [19:38:52] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Test dlocal India recurring ondemand 2nd charge - https://phabricator.wikimedia.org/T338812 (10XenoRyet) 05Open→03Resolved [19:38:54] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10Epic: India Recurring - Post launch issues epic - https://phabricator.wikimedia.org/T339180 (10XenoRyet) [19:39:21] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Staging in a mess (but don't look until after your public holiday!) - https://phabricator.wikimedia.org/T341029 (10XenoRyet) 05Open→03Resolved [19:39:23] 10fundraising-tech-ops: encrypt fundraising database client->server communication - https://phabricator.wikimedia.org/T170321 (10XenoRyet) [19:39:42] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Handle Adyen Chargeback IPNs - https://phabricator.wikimedia.org/T342021 (10XenoRyet) 05Open→03Resolved [19:40:13] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deprecated function: Function Redis::delete() is deprecated in Redis_Lock_PhpRedis->lockRelease() - https://phabricator.wikimedia.org/T342186 (10XenoRyet) 05Open→03Resolved [19:40:30] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Investigate timeouts from Adyen Soap listener - https://phabricator.wikimedia.org/T317422 (10XenoRyet) 05Open→03Resolved [20:16:35] (03PS2) 10Wfan: Check if payload.details.payerInfo not undefined [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942796 (https://phabricator.wikimedia.org/T340799) [20:18:18] (03CR) 10CI reject: [V: 04-1] Check if payload.details.payerInfo not undefined [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942796 (https://phabricator.wikimedia.org/T340799) (owner: 10Wfan) [20:19:36] 10Fundraising-Backlog: Figure out if we can make the queue delay graph better with audit import lag - https://phabricator.wikimedia.org/T342812 (10XenoRyet) This is probably relevant for some of our OKR metrics. [20:20:01] (03PS3) 10Wfan: Check if payload.details.payerInfo not undefined [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942796 (https://phabricator.wikimedia.org/T340799) [20:22:03] (03PS6) 10Wfan: Add refund script for braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942513 (https://phabricator.wikimedia.org/T342788) [20:22:54] 10Fundraising-Backlog: Civi receipt lag 26 July 2023 - https://phabricator.wikimedia.org/T342719 (10AKanji-WMF) For follow up with Dallas/Jeff re: monitor ongoing TY email performance, and potential investigation of distribution between the two email servers [20:24:38] (03PS3) 10Wfan: Unit test for audit parse venmo [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942532 (https://phabricator.wikimedia.org/T340801) [20:28:09] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Small design issue in Hebrew post-payment monthly convert - https://phabricator.wikimedia.org/T342713 (10AKanji-WMF) [20:29:40] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Problems donating via PayPal from France 07/24 - https://phabricator.wikimedia.org/T342652 (10AKanji-WMF) [20:31:23] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising Sprint Machine Unlearning, 10Fundraising Sprint NaN is a Number, and 4 others: Create IPN listener for Braintree - https://phabricator.wikimedia.org/T303451 (10jgleeson) a:05jgleeson→03None [20:32:13] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising Sprint Machine Unlearning, 10Fundraising Sprint NaN is a Number, and 4 others: Create IPN listener for Braintree - https://phabricator.wikimedia.org/T303451 (10AKanji-WMF) Will need to include t... [21:47:53] (03CR) 10Eileen: "Do we know how this arises? We kinda need to upstream this if we want to have this patch cos otherwise it would have a maintenance onload " [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/943608 (https://phabricator.wikimedia.org/T340061) (owner: 10Damilare Adedoyin) [22:03:17] 10Fundraising-Backlog: Civi receipt lag 26 July 2023 - https://phabricator.wikimedia.org/T342719 (10Eileenmcnaughton) Re the 2 email servers - we seem to be sending about half & half to each of them - despite one being much slower. My understanding is we just need to keep the slower one warm so the question to J... [23:05:31] 10Fundraising-Backlog: Check the Adyen audit files August 25 to September 15 - https://phabricator.wikimedia.org/T343171 (10Cstone) [23:23:48] (03PS1) 10Eileen: Fix option label create bug [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/943644 (https://phabricator.wikimedia.org/T342943) [23:38:30] (03CR) 10CI reject: [V: 04-1] Fix option label create bug [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/943644 (https://phabricator.wikimedia.org/T342943) (owner: 10Eileen) [23:45:32] we seem to have a date-related test fail - which will need some test tweaking but affecing all civi tests I think [23:46:10] at least on jenkins - I'm not seeing it - but it is August for me - [23:46:36] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Problems donating via PayPal from France 07/24 - https://phabricator.wikimedia.org/T342652 (10Cstone) Collecting data here, the ones mentioned all came with the same error back from Paypal. | CID | Order Id | Amount | Error | Donat...