[01:24:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Civi / Adyen mismatch re: recurring retries - https://phabricator.wikimedia.org/T365519#9878013 (10Ejegg) There are 31,528 of them, associated with 9,385 different recurring subscriptions. I've got all the IDs in a table so I should be able to delete the... [01:57:53] (03PS1) 10Ejegg: Clean up wrongly recorded contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041296 (https://phabricator.wikimedia.org/T365519) [02:03:19] (03PS4) 10Ejegg: Fully delete all WSDL/SOAP stuff from Adyen [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037863 (https://phabricator.wikimedia.org/T263621) [02:26:08] (03PS9) 10Cstone: Add acitives for recurring failure and autorescue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1037204 (https://phabricator.wikimedia.org/T361256) [02:43:41] 03Fundraising Sprint K (placeholder name), 14Fundraising Sprint: justWork(), 06Fundraising-Backlog: Enable Retry logic for ACH transactions - https://phabricator.wikimedia.org/T365100#9878050 (10Cstone) ACH is going into the 3 tries and failure flow [03:01:13] (03CR) 10Cstone: [C:03+2] Another fallback for CC submethod under Adyen [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1039296 (owner: 10Ejegg) [03:06:26] (03Merged) 10jenkins-bot: Another fallback for CC submethod under Adyen [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1039296 (owner: 10Ejegg) [03:22:07] (03CR) 10Cstone: [C:03+2] Add test for ACH donation in Audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1040276 (owner: 10Ejegg) [03:22:34] (03Merged) 10jenkins-bot: Add test for ACH donation in Audit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1040276 (owner: 10Ejegg) [03:42:43] (03PS1) 10Cstone: Remove uneeded fields for ACH [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1041331 [03:44:32] (03CR) 10CI reject: [V:04-1] Remove uneeded fields for ACH [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1041331 (owner: 10Cstone) [03:50:14] (03PS2) 10Cstone: Remove uneeded fields for ACH [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1041331 [05:31:01] 06Fundraising-Backlog: Scoping: 2nd Recurring Failure Email in Civi - https://phabricator.wikimedia.org/T365488#9878210 (10MSuijkerbuijk_WMF) @AKanji-WMF Yes, let me add more detail - [[ https://docs.google.com/document/d/15IvOaVsdBNklm9HVMNP9z2oiCLUizj7sOHV5iTcI_Ps/edit#heading=h.1mggabd7oeg0 | Email 1 copy... [05:53:02] 03Fundraising Sprint K (placeholder name), 14Fundraising Sprint: justWork(), 06Fundraising-Backlog: Enable Retry logic for ACH transactions - https://phabricator.wikimedia.org/T365100#9878226 (10RKumar_WMF) Thank you! [12:03:52] (03CR) 10Jgleeson: WIP: Add CiviCRM Standalone (031 comment) [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1039235 (owner: 10Jgleeson) [12:32:58] (03PS1) 10Damilare Adedoyin: Add donor connection request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 [12:48:04] (03PS2) 10Jgleeson: Add CiviCRM Standalone [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1039235 [12:48:21] (03PS3) 10Jgleeson: Add CiviCRM Standalone [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1039235 (https://phabricator.wikimedia.org/T276395) [13:04:15] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Civi / Adyen mismatch re: recurring retries - https://phabricator.wikimedia.org/T365519#9879616 (10MBeat33) Many thanks, @Ejegg [13:13:09] 03Fundraising Sprint K (placeholder name), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021, 13Patch-For-Review: Spike - do practice standalong installs locally to get our heads around it - https://phabricator.wikimedia.org/T276395#9879653 (10jgleeson) After setting up [[... [13:41:59] hi fr-tech! [13:42:39] So I have to run a bunch of deletes in the prod db [13:43:22] and I'd love it if anyone could review my scripts, maybe even watch while I run 'em [13:43:34] let's see, I guess I should do a morning-after self-review first [13:43:49] I could definitely add the 'begin transaction' lines there [13:44:27] and figure out whether the contribution deletes cascade everywhere they need to [13:46:57] ooh, I should also check the other IPNs for bool bugs [13:48:57] ok, looks like that was the only place [13:52:43] (03PS2) 10Damilare Adedoyin: Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 [13:54:17] (03PS3) 10Damilare Adedoyin: Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 [13:54:54] (03CR) 10Jgleeson: "Just started review. Some initial thoughts inline." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) (owner: 10Damilare Adedoyin) [13:55:21] ejegg: will take a look when I get back from skrun! [13:56:06] (03PS20) 10Damilare Adedoyin: Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) [13:56:17] thanks jgleeson|skrun, I'd check it out [13:56:55] (03CR) 10CI reject: [V:04-1] Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 (owner: 10Damilare Adedoyin) [14:42:13] (03PS21) 10Damilare Adedoyin: Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) [14:42:39] (03CR) 10CI reject: [V:04-1] Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) (owner: 10Damilare Adedoyin) [14:47:00] (03PS22) 10Damilare Adedoyin: Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) [14:47:37] (03CR) 10CI reject: [V:04-1] Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) (owner: 10Damilare Adedoyin) [14:50:20] (03PS23) 10Damilare Adedoyin: Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) [14:50:56] (03CR) 10CI reject: [V:04-1] Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) (owner: 10Damilare Adedoyin) [14:51:35] (03PS24) 10Damilare Adedoyin: Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) [14:57:00] (03CR) 10Damilare Adedoyin: "Hi jgleeson, thanks for the CR, I updated the patch and replied with a comment inline." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) (owner: 10Damilare Adedoyin) [15:19:05] I put up another version of the scripts and re-ran the numbers to include all the ones from yesterday before the fix went out [15:19:05] wikibugs seems to be silent though [15:19:05] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1041296 [15:19:10] and I tried the SQL deletes on my local and it seemed to cascade just fine [15:35:32] ejegg: that one doesn't look like much fun - is someone else following it or do you want me to take a look [15:36:02] eileen: I think jgleeson might have a minute, but if you'd like to take a peek I'd appreciate the extra attention! [15:36:10] ok cool [15:36:24] jgleeson: you are feeling OK now? [15:36:37] yeah, a really boneheaded bug with a pretty big effect as far as donation numbers go [15:37:30] fortunately very few TY emails sent as a result [15:42:22] hey eileen! I'm ok thanks, symptoms are pretty mild at this point apart from not being able to smell or taste anything. Unfortunately Oscar has picked it up too so he's off school today. Roxy and Tilly have escaped it so far. [15:43:05] ok - I wondered if I might have had an early version early in the week & given it to everyone but neither Seamus or I lost taste & you & Patrick both did [15:44:18] yeah I have no idea really who i got it off [15:47:11] timing wise it makes sense that I was the one who brought it along to share - but my symptoms were only sore throat & runny nose [15:47:25] & didn't last long [15:53:47] just hope it's all gone before the onsite. :) [15:54:30] that's one thing i don't miss about conferences, the post conference crud. [15:54:36] for sure [16:44:28] 14Fundraising Sprint: justWork(), 03Fundraising Sprint: kevinToldMeToCallThis(), 06Fundraising-Backlog, 10FR-donorservices: ask Civi to use PayPal's API when humans cancel recurrings - https://phabricator.wikimedia.org/T361636#9880920 (10SHust) Hi team, I was testing the new PayPal cancellation process and... [16:53:36] (03PS9) 10Damilare Adedoyin: Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 [16:54:04] (03CR) 10CI reject: [V:04-1] Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 (owner: 10Damilare Adedoyin) [16:55:47] conflu [16:59:45] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Civi / Adyen mismatch re: recurring retries - https://phabricator.wikimedia.org/T365519#9880994 (10MBeat33) @Ejegg are the 68 CIDs with only fails donors who were starting new recurring donations? If so + you can share the list, the... [17:07:45] 06Fundraising-Backlog: Amazon "session expired" donation error when logging in - https://phabricator.wikimedia.org/T120680#9881027 (10MBeat33) 05Open→03Resolved a:03MBeat33 Resolving as we haven't seen this issue in a long time. [17:10:13] 06Fundraising-Backlog, 10FR-donorservices: 11/20 Email donation link pointed to “foundation.wikimedia.org/wiki/Home" - https://phabricator.wikimedia.org/T268502#9881040 (10MBeat33) 05Open→03Resolved a:03MBeat33 [17:11:16] (03PS10) 10Damilare Adedoyin: Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 [17:11:38] !log fundraising civicrm upgraded from ebfbad86 to 7252b1b9 [17:11:38] (03CR) 10CI reject: [V:04-1] Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 (owner: 10Damilare Adedoyin) [17:11:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:11:44] (03PS11) 10Damilare Adedoyin: Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 [17:12:06] (03CR) 10CI reject: [V:04-1] Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 (owner: 10Damilare Adedoyin) [17:13:59] (03PS27) 10Damilare Adedoyin: Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) [17:14:26] (03CR) 10CI reject: [V:04-1] Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) (owner: 10Damilare Adedoyin) [17:14:42] (03PS12) 10Damilare Adedoyin: Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 [17:15:15] (03PS28) 10Damilare Adedoyin: Gravy create payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1037776 (https://phabricator.wikimedia.org/T364506) [17:15:51] (03PS13) 10Damilare Adedoyin: Add donor creation request [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1041627 [17:27:06] 03Fundraising Sprint: kevinToldMeToCallThis(), 06Fundraising-Backlog: Validation checks for ACH in donor checkout - https://phabricator.wikimedia.org/T360549#9881157 (10EMartin) 05Open→03Resolved a:03EMartin Adyen confirm their logic is correct in that they only check for 3 digits. [17:27:24] eileen: so do you see any issue with using SQL to delete at least the bad contributions there? [17:28:03] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Civi / Adyen mismatch re: recurring retries - https://phabricator.wikimedia.org/T365519#9881176 (10Ejegg) Hi @MBeat33 actually a lot of the only-fail donors are CZ online banking donors. For that payment method (and iDEAL) we record... [17:28:51] Thinking of leaving the contact records in place for now. [17:29:20] (03CR) 10Jgleeson: [C:03+1] "I tried some of this on frlog and it worked as expected." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041296 (https://phabricator.wikimedia.org/T365519) (owner: 10Ejegg) [17:29:34] thanks jgleeson [17:30:48] (03CR) 10Ejegg: Clean up wrongly recorded contributions (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041296 (https://phabricator.wikimedia.org/T365519) (owner: 10Ejegg) [17:36:18] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10Mail: Update fundraising mail settings to use new production mx hosts - https://phabricator.wikimedia.org/T366740#9881238 (10Dwisehaupt) 05Open→03Resolved The changes have been pushed to eqiad hosts and log entries look good. Closing. [17:39:15] hmm, deadlocks? [17:40:16] 06Fundraising-Backlog, 10FR-Civi-Dedupe, 10FR-donorservices: Deduper Issue - https://phabricator.wikimedia.org/T363958#9881249 (10SHust) **Update:** This issue seems to be resolved! [17:41:00] dedupe query [17:41:10] but it's got a not-so-crazy amount of IDs [17:41:36] unions tho [17:42:09] killed the query [17:51:08] eileen: I was thinking earlier regarding https://phabricator.wikimedia.org/T276395 if would be have to create a new buildkit build called something like wmff-standalone to pull in our customisations ? [17:52:27] damilare: would it make sense to use CardPaymentProvider if we're only redirecting for Paypal? [17:53:30] ok sure, that works too jgleeson [17:53:37] Fr-tech: Evelyn has a question about what flavor our Adyen integration is, API, Drop-in, or Web, and I don't remember off the top of my head. [17:53:46] Anyone got that answer real quick? [17:53:55] Web XenoRyet I think [17:54:04] I think drop-in was the full hit [17:54:09] Cool, thanks. [17:54:11] and we use Web Components [17:54:17] which was the lighter tough [17:54:19] touch [17:54:21] * [17:54:23] That sounds right [17:55:00] Covid has robbed me of the smell of coffee [17:56:28] that sucks jgleeson [17:56:55] indeed :( [17:57:52] 14Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10FR-Civi-Dedupe, 07Unplanned-Sprint-Work: Sandra can't dedupe Name + address matching contacts (no email match) - https://phabricator.wikimedia.org/T353971#9881347 (10SHust) Are there any updates about the 'Address and name (General)' rule? [17:59:42] 06Fundraising-Backlog, 10FR-Civi-Dedupe, 10FR-donorservices: Merging oddity - https://phabricator.wikimedia.org/T362928#9881357 (10SHust) **Update: **This issue seems to have been solved! [18:01:54] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy Integration - https://phabricator.wikimedia.org/T364501#9881384 (10AKanji-WMF) [18:03:58] ok, deleting those bad contribs [18:04:03] 06Fundraising-Backlog, 07Epic, 07payments-orchestration: Gravy backend approve payment - https://phabricator.wikimedia.org/T367112#9881403 (10AKanji-WMF) [18:04:06] 14Fundraising Sprint: justWork(), 03Fundraising Sprint: kevinToldMeToCallThis(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Gravy frontend createPayment - https://phabricator.wikimedia.org/T364508#9881404 (10AKanji-WMF) [18:04:10] 14Fundraising Sprint: justWork(), 03Fundraising Sprint: kevinToldMeToCallThis(), 06Fundraising-Backlog, 13Patch-For-Review, 07payments-orchestration: Gravy backend createPayment - https://phabricator.wikimedia.org/T364506#9881418 (10AKanji-WMF) [18:05:57] huh, maybe I should have stopped queues [18:08:11] !log stopped fundraising scheduled jobs [18:08:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:11:14] crud, still hanging [18:11:21] well, it's in a transaction [18:11:41] so i should be able to roll back [18:13:15] ohhh wait, am i on the r/o server??? [18:13:21] killed [18:15:19] no... I was able to create the temp tables [18:16:00] ok, will do the deletes in smaller batches [18:16:11] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1328 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [18:17:09] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1385 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [18:17:27] dwisehaupt: looks like my deletes are maybe causing that lag? [18:18:10] i just killed the really big delete so it would have rolled back [18:18:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1451 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [18:18:21] then i started a delete of 3k contribs [18:18:37] but I guess with all the cascades and triggers even that is pretty gnarly [18:18:52] maybe I just need to queue them up in an upgrade [18:21:09] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1628 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [18:22:09] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1685 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [18:23:15] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 2246064 Threads: 5 Questions: 101991444 Slow queries: 1037 Opens: 4799 Open tables: 1162 Queries per second avg: 45.408 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [18:23:36] argh, will kill the small batch query too [18:23:53] ok, killed [18:26:09] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1906 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [18:27:09] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 512066 Threads: 4 Questions: 28521644 Slow queries: 237 Opens: 1163 Open tables: 1141 Queries per second avg: 55.699 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [18:27:25] ejegg: could you use your query to get them into a group then we could use searchkit or would that cause lag on its own? [18:29:21] I think we just need to do smaller batches, right? [18:29:33] would searchkit let us delete in batches? [18:30:13] and can you put contributions in a group? [18:30:22] hmmmmmm [18:31:10] we could add something to the ones you want to delete although that just seems like more work though [18:31:11] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 496865 Threads: 4 Questions: 28001166 Slow queries: 237 Opens: 1161 Open tables: 1139 Queries per second avg: 56.355 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [18:31:42] yeah, i'm just making a patch to get them deleted via coworker [18:39:35] 06Fundraising-Backlog: Add GIACT account verification for ACH - https://phabricator.wikimedia.org/T367078#9881743 (10AKanji-WMF) @EMartin we are on component [18:42:06] 06Fundraising-Backlog, 07payments-orchestration: Enable Amazon Pay through Adyen (via Gr4vy?) - https://phabricator.wikimedia.org/T362991#9881747 (10AKanji-WMF) @EMartin can you please update this (not urgent) with the current status of Amazon Pay via Adyen (via Gravy?) [18:42:25] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 13Patch-For-Review, 10Web-Team-Backlog (Needs Prioritization (Tech)): CentralNotice should catch banner errors and log them to its own channel - https://phabricator.wikimedia.org/T361680#9881750 (10Jdlrobson) I am talking to Sam about this on Thu... [18:46:57] 06Fundraising-Backlog, 07payments-orchestration: Enable Amazon Pay through Adyen (via Gr4vy?) - https://phabricator.wikimedia.org/T362991#9881766 (10EMartin) In a conversation with Gr4vy during the week of 6/10/24 they disclosed that Adyen has deprecated the connection to Amazon Pay. The notion of re-enabli... [18:47:34] ejegg: did you figure out coworker - upgrade_4380 uses coworker [18:49:45] 06Fundraising-Backlog: Add GIACT account verification for ACH - https://phabricator.wikimedia.org/T367078#9881773 (10EMartin) Thanks @Anil. I will ask Adyen what we do since GIACT is only available on the API connection. [18:56:39] yep, we're in the tech talk eileen [18:56:49] oh cool [18:56:52] about to take a stab at writing that batch delete [18:57:42] (03CR) 10Ejegg: [C:03+2] Attempt to recreate and import missed Dlocal trxns through audit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1037094 (owner: 10Damilare Adedoyin) [18:57:57] (03PS7) 10Damilare Adedoyin: Attempt to recreate and import missed Dlocal trxns through audit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1037094 [18:58:04] (03CR) 10Ejegg: [V:03+2 C:03+2] Attempt to recreate and import missed Dlocal trxns through audit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1037094 (owner: 10Damilare Adedoyin) [18:58:17] thanks ejegg [19:19:10] (03PS1) 10Ejegg: Upgrader to delete bad contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041753 (https://phabricator.wikimedia.org/T365519) [19:24:38] (03Merged) 10jenkins-bot: Attempt to recreate and import missed Dlocal trxns through audit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1037094 (owner: 10Damilare Adedoyin) [19:31:27] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Civi / Adyen mismatch re: recurring retries - https://phabricator.wikimedia.org/T365519#9881960 (10MBeat33) Thank you @Ejegg that will let us get ahead of them noticing the funds not transferring despite the TY email [19:36:27] (03PS3) 10Ejegg: Update composer.lock for smashpig upgrade [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/1039291 [19:36:43] (03CR) 10Ejegg: [C:03+2] "recheck" [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/1039291 (owner: 10Ejegg) [19:45:42] 06Fundraising-Backlog: Create button to merge "Unknown: Shares contact information" - https://phabricator.wikimedia.org/T367234 (10NNichols) 03NEW [19:45:53] (03Merged) 10jenkins-bot: Update composer.lock for smashpig upgrade [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/1039291 (owner: 10Ejegg) [20:03:01] (03PS3) 10Ejegg: Find wrongly recorded contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041296 (https://phabricator.wikimedia.org/T365519) [20:03:01] (03PS2) 10Ejegg: Upgrader to delete bad contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041753 (https://phabricator.wikimedia.org/T365519) [20:30:53] (03CR) 10Ejegg: "To test locally: use sql to create the table and insert 1 row:" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041753 (https://phabricator.wikimedia.org/T365519) (owner: 10Ejegg) [20:31:16] fr-tech I added local test instructions to ^^^ if anyone wants to take a look [20:40:20] I'll try it out ejegg [20:42:13] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-init is 2053 2000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 20 keys, up 20 days 3 hours - memory use is 162.20M (peak 162.35M, 2.10% of max, fragmentation 1.06%), connected_slaves is 3, donations is 3034, jobs is 0, jobs-adyen is 2637, jobs-dlocal is 7, jobs-paypal is 566, payments-antifraud is 4380, pending is 955, recurring is 34, refund is 5, unsub [20:42:13] 13 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [20:47:13] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-init is 2122 2000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 20 keys, up 20 days 3 hours - memory use is 162.62M (peak 162.84M, 2.11% of max, fragmentation 1.06%), connected_slaves is 3, donations is 3130, jobs is 0, jobs-adyen is 2720, jobs-dlocal is 7, jobs-paypal is 589, payments-antifraud is 4535, pending is 994, recurring is 34, refund is 5, unsub [20:47:13] 13 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [20:51:05] oops, right, turning those jobs back on! [20:52:05] thanks :) [20:52:17] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-init is 2184 2000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 20 keys, up 20 days 3 hours - memory use is 163.00M (peak 163.26M, 2.11% of max, fragmentation 1.06%), connected_slaves is 3, donations is 3213, jobs is 0, jobs-adyen is 2796, jobs-dlocal is 7, jobs-paypal is 611, payments-antifraud is 4675, pending is 1031, recurring is 34, refund is 6, unsu [20:52:17] 13 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [20:52:51] !log re-enabled fundraising scheduled jobs [20:52:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:54:00] jgleeson: ah i think we need to disable xdebug on cli to make coworker run [20:54:42] hmm, i do have it disabled [20:55:11] what was making that "Received unexpected response line:" error then [20:56:01] let me try to make it log that sql before running it [20:57:13] RECOVERY - check_redis on frqueue1004 is OK: OK: REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 20 days 3 hours - memory use is 154.39M (peak 163.26M, 2.02% of max, fragmentation 1.07%), connected_slaves is 3, donations is 3875, jobs is 0, jobs-adyen is 23, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 2, payments-init is 0, pending is 1065, recurring is 114, refund is 7, unsubscribe is 14 https://icin [20:57:13] ia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [20:57:55] ok, there's a SQL error [21:01:02] (03PS3) 10Ejegg: Upgrader to delete bad contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041753 (https://phabricator.wikimedia.org/T365519) [21:02:49] 06Fundraising-Backlog, 10fundraising-tech-ops, 13Patch-For-Review: On missing audit file, send Notification - https://phabricator.wikimedia.org/T365466#9882248 (10Dwisehaupt) Check enabled on the hosts. Patch provided to prod puppet repo to enable reporting on the icinga side. [21:02:57] fixed in PS3 jgleeson [21:03:29] if you already run the upgrader you need to delete from civicrm_queue_item [21:03:46] and then either edit the code to make that update 4511 or roll back the version in the db [21:05:37] luckily I had to install civi & payments and import a donation so I'm ready now :) [21:07:32] ooh, failmails [21:07:56] weird, email prefs invalid? [21:08:14] db lock [21:08:21] ah, let's find that [21:09:08] oh more dedupe [21:10:39] killed [21:12:59] ok jgleeson I ran the update in PS3 and then coworker, and it deleted the donation [21:18:27] (03CR) 10Jgleeson: [C:03+2] "This is working for me locally. The contribution added was removed after running coworker." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041753 (https://phabricator.wikimedia.org/T365519) (owner: 10Ejegg) [21:20:01] thanks jgleeson ! [21:20:11] want to merge that setup patch with the sql + sh scripts? [21:20:23] it's just in the chain [21:24:54] i supposed I can self-merge that one, as it's not really code that runs on prod [21:25:22] (03CR) 10Ejegg: [C:03+2] Find wrongly recorded contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041296 (https://phabricator.wikimedia.org/T365519) (owner: 10Ejegg) [21:25:50] beat me to it [21:26:05] ok it's pretty late here I'm wrapping up [21:26:12] catch you all tomorrow o/ [21:47:03] (03Merged) 10jenkins-bot: Find wrongly recorded contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041296 (https://phabricator.wikimedia.org/T365519) (owner: 10Ejegg) [21:47:05] (03Merged) 10jenkins-bot: Upgrader to delete bad contributions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1041753 (https://phabricator.wikimedia.org/T365519) (owner: 10Ejegg) [21:47:09] finally! [21:48:17] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1041785 [21:48:18] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1041785 (owner: 10Ejegg) [21:48:25] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1041785 (owner: 10Ejegg) [21:51:08] !log fundraising civicrm upgraded from 7252b1b9 to f7855d25 [21:51:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:53:10] 03Fundraising Sprint: kevinToldMeToCallThis(), 06Fundraising-Backlog, 10fundraising-tech-ops: On missing audit file, send Notification - https://phabricator.wikimedia.org/T365466#9882469 (10Dwisehaupt) The check is showing up for the hosts. Closing. https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?sea... [21:55:09] ok, coworker should start deleting those contributions now [22:02:49] hmm, none deleted yet [22:02:59] let's see if we have anything in logs [22:04:01] huh, 27 things in the queue [22:04:16] is coworker running? [22:04:38] several omnisnooze jobs ahead [22:04:49] ohh, and a few submitted a long time ago! [22:05:34] ok, those omni-snooze/damaged ones all have a snooze_end_date in the past [22:05:38] they just need to be deleted; [22:10:24] ok ,deleted [22:10:30] ohh there's also a release_time of 1 hr after the submit_time on that wmf_data_upgrades job [22:10:38] hmm, will every batch have an hour wait? [22:18:27] oh i see Queue item from wmf_data_upgrades with id=394701 failed with exception="Database lock encountered" [22:23:54] it's trying again [22:25:28] ooh, it got some! [22:25:57] hmm, or maybe not? Looked like that queue item succeeded [22:26:13] but the contribution count seems not to have dropped [22:27:55] 06Fundraising-Backlog: Add GIACT account verification for ACH - https://phabricator.wikimedia.org/T367078#9882541 (10EMartin) Parking some background on the ACH validation requirement here as we consider how to gain compliance: The ruling is famously neutral in its compliance requirements, which means that it’s... [22:28:51] 06Fundraising-Backlog: Compliance requirement re: account validation for ACH - https://phabricator.wikimedia.org/T367078#9882542 (10EMartin) [22:30:18] oh i see, the queue item id changed but the parameters are the same [22:30:23] run count is up to 9 :( [22:30:31] ok, maybe we needed smaller batches? [22:41:09] whaa [22:41:17] looks like it's running two items at once? [22:46:43] the heck... [22:46:49] well that's gonna keep deadlocking [22:50:47] ugh, ok, I'mma just mess with the parameters [22:51:02] so how many contributions should we actually be able to delete at once? [22:54:42] ugh, still running the 200 batch and deadlocking the queues [22:55:12] let's kill it and see if it can run a 100 batch [22:58:18] 06Fundraising-Backlog, 07payments-orchestration: Enable Amazon Pay through Adyen (via Gr4vy?) - https://phabricator.wikimedia.org/T362991#9882564 (10AKanji-WMF) FYI @jgleeson [22:58:27] oh jeez, now 'queue task returned false' [22:58:47] I guess my manual param tweak didn't work? [22:59:21] who'd have thought deleting rows 200 at a time would be so hard? [23:01:07] heh. [23:02:59] ok, I just deleted the queue item so it won't keep trying [23:03:09] gonna head home and try something else tomorrow [23:04:11] 06Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for jschoenig - https://phabricator.wikimedia.org/T364396#9882573 (10Dwisehaupt) [23:04:59] 06Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for jschoenig - https://phabricator.wikimedia.org/T364396#9882574 (10Dwisehaupt) 05Open→03Resolved Met with jschoenig to set up the ssl cert. Verified access is working to civi and superset.