[00:00:10] custom country [00:02:43] (03PS1) 10Eileen: Follow on to Contribution insert switch - re-throw the exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034604 [00:07:17] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: New Segmentation Framework not quite usable yet - https://phabricator.wikimedia.org/T363527#9819423 (10Eileenmcnaughton) This is the same issue as the non-donor one - https://phabricator.wikimedia.org/T363959#9819404 - when we 'Acoustic-merge' a contact (... [00:08:45] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Deep Lapsed records are being labeled as Ultra Lapsed - https://phabricator.wikimedia.org/T365523#9819427 (10Eileenmcnaughton) a:03Eileenmcnaughton [00:18:50] (03PS1) 10Eileen: Move wmf_civicrm_contribution_message_import to DonationQueueConsumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034627 (https://phabricator.wikimedia.org/T365415) [00:28:33] (03CR) 10Cstone: [C:03+2] Do not map 'country' to a custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034603 (owner: 10Eileen) [00:29:17] (03PS1) 10Eileen: Remove extraneous initialize [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034628 (https://phabricator.wikimedia.org/T365415) [00:29:17] (03PS1) 10Eileen: Move minor helper to only caller [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034629 (https://phabricator.wikimedia.org/T365415) [00:29:19] (03PS1) 10Eileen: Remove calls to legacy timer function [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034630 (https://phabricator.wikimedia.org/T365415) [00:29:31] (03CR) 10Eileen: [V:03+2] Do not map 'country' to a custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034603 (owner: 10Eileen) [00:30:25] cstone: https://gerrit.wikimedia.org/r/1034627 moves the function to where you thought it already was :-) (now it has no other callers) [00:30:42] oh hooo niceee [00:30:50] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034631 [00:31:00] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034631 (owner: 10Eileen) [00:33:27] !log civicrm upgraded from c77df721 to f1c24cb7 [00:33:27] - let's try again [00:33:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:34:40] cstone: I requeued Prof Dr without the Dr [00:36:35] haha nice thanks [00:37:37] (03CR) 10Eileen: [C:03+2] "No obvious change - which is probably good" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034596 (owner: 10Ejegg) [00:40:35] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021, 13Patch-For-Review: Convert exchange rates module to an extension - https://phabricator.wikimedia.org/T270666#9819450 (10Eileenmcnaughton) This is almost done - just a couple of tidy ups in... [00:41:10] (03PS5) 10Ejegg: Delete old exchange_rates module [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034523 (https://phabricator.wikimedia.org/T270666) [00:49:54] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034523 (https://phabricator.wikimedia.org/T270666) (owner: 10Ejegg) [01:02:14] cstone: the re-deploy seems better [01:03:22] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Update RML donors to have status of non-donor in CiviCRM - https://phabricator.wikimedia.org/T365534#9819482 (10KHaggard) Ah I see. I think it would be worth asking Sheetal and Erica to weigh in on it. I don't think Sheetal has a phab handle yet so taggi... [01:09:15] (03Merged) 10jenkins-bot: Update theisland theme [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034596 (owner: 10Ejegg) [01:10:20] (03Merged) 10jenkins-bot: Delete old exchange_rates module [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034523 (https://phabricator.wikimedia.org/T270666) (owner: 10Ejegg) [01:11:03] ah eileen looks like nvalid value (donations) submitted for custom field 11:Prospect Type - Prospect.Type Source: array ( ) [01:11:29] cstone: ok will check - are you OK to +2 that move one? Just worried about conflicts [01:11:44] ah okay ill look at it [01:12:42] i dont know why linux machine decides wifi doesnt exist for like 30 seconds after it wakes up [01:14:31] (03CR) 10Cstone: [C:03+2] Move wmf_civicrm_contribution_message_import to DonationQueueConsumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034627 (https://phabricator.wikimedia.org/T365415) (owner: 10Eileen) [01:15:06] ah eileen that failmail chunk was from an audit [01:15:38] cstone: ok - I should have a patch in a mo [01:16:45] yeah no rush on it we can rerun it whenever that one was the last one to run automattically [01:23:26] (03PS1) 10Eileen: Do not map type to a custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034638 [01:23:28] cstone: ^^ [01:38:24] (03Merged) 10jenkins-bot: Move wmf_civicrm_contribution_message_import to DonationQueueConsumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034627 (https://phabricator.wikimedia.org/T365415) (owner: 10Eileen) [01:45:19] eileen: looks like type goes along with txn_type from the audit so type=recurring txn_type=subscr_cancel [01:45:41] cstone: ok - we should beef out that description huh? [01:45:54] any thoughts on what it should say? [01:46:20] i think type = queue [01:47:05] and i think only frup one is using it that way [01:47:11] since it does more than the others [01:49:42] (03PS2) 10Eileen: Do not map type to a custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034638 [01:49:52] cstone: does that capture it? [01:50:01] yep that makes sense! [01:53:01] 14Fundraising Sprint: didAnyoneTryThis(), 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Paypal: Revisit old PayPal bugs, delete code if no longer needed - https://phabricator.wikimedia.org/T355448#9819521 (10Eileenmcnaughton) grep -r "Updating legacy" * r... [01:53:17] cstone: if you have mental space I want to throw another one at you.... [01:53:31] sure [01:53:52] im trying to put edge case stuff in the queues locally to see if there are any other one off fields [01:54:59] oh cool - I can wait until you have your braing back out of that - should we deploy that type one & re-queue some in the meantime? [01:55:31] (03CR) 10Cstone: [C:03+2] Do not map type to a custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034638 (owner: 10Eileen) [01:55:51] oh yeah nothing I just tried failed eileen hah [01:56:01] oh cool [01:56:05] so my other one .... [01:56:17] https://phabricator.wikimedia.org/T355448 [01:56:50] so in 2018 paypal changed it's subscriber IDs & we have been handling the change ever since & there is a reason why we are still handling it 6 years on .... [01:57:35] because when look up recurrings we look up by civicrm-contribution_recur.trxn_id but when we 'fix' the legacy paypal ones we update civicrm_contribution_recur.processor_id [01:58:04] so, not matter how many times we do that we will never find the recurring the 'normal' way the following month & will re-fix it [01:59:23] is there a reason we never did you option 1 on the phab? [01:59:42] probably - or at least there probably was in 2018.... [01:59:57] it seems the most straightforward coming to this with no context [02:00:10] I don't even know if we use the same fields now - maybe ejegg does if he is about [02:00:38] cause in theory it would fix most of them on their own in a month [02:01:07] yeah - I imagine in 2018 it was expected to be short term & when it didn't resolve we just kept waiting [02:01:58] so we have the old one in trxn_id & the new one in processor_id on the 'fixed' records [02:02:06] oh hah, option 1 sounds like the right fix [02:02:22] thanks ejegg - it's an easy one to patch up [02:03:15] so we think the 'S-' style is just meaningless now? or should we stick that in processor_id? [02:03:51] sure, I guess it would be nice to keep it around someplace in case we need to correlate with older records [02:04:07] we could try looking things up in the console to see what shows up [02:04:37] do you have a CID handy that is associated with one of those? [02:04:48] 4995599 [02:05:09] thanks! [02:11:49] eileen: In the console, that's got the I- prefixed ID everywhere I can see [02:12:03] ejegg: so bye bye 'S'.... [02:12:26] yeah, I think that's OK [02:12:35] worst case, we can dredge it up from the logs [02:14:50] (03PS1) 10Eileen: Save the 'new' (hello 2018) subscriber_id to trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034639 (https://phabricator.wikimedia.org/T355448) [02:14:52] (03Merged) 10jenkins-bot: Do not map type to a custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034638 (owner: 10Eileen) [02:15:39] ok I think ^^ will do it [02:16:16] damn there is a very insistent chicken on the doorstep [02:16:48] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034640 [02:16:57] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034640 (owner: 10Eileen) [02:17:44] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034640 (owner: 10Eileen) [02:21:41] !log civicrm upgraded from f1c24cb7 to c9d64b68 [02:21:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:21:47] can requue now [02:35:17] (03CR) 10CI reject: [V:04-1] Save the 'new' (hello 2018) subscriber_id to trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034639 (https://phabricator.wikimedia.org/T355448) (owner: 10Eileen) [02:41:47] (03PS2) 10Eileen: Save the 'new' (hello 2018) subscriber_id to trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034639 (https://phabricator.wikimedia.org/T355448) [02:50:48] cstone: I requeued a few earlier messages cos I thought some of them might have been deadlocks in disguise - hence the last 2 fails are them re-failing [02:51:39] ooh back now eileen i can rerun the audit too [02:52:16] :-) [02:52:24] So how would one figure out what happened here.... https://civicrm.wikimedia.org/civicrm/damaged/edit?id=637037 [02:54:56] ah that might be one of the ideal ones [02:57:15] yeah eileen so that one adyen turned on autorescue fully which included ideal/sepa but we werent expecting that on our side so we never saved the autorescue reference [02:57:53] so what does that mean? - could we rescue the rescue reference? or are we thinking those will age out of relevance? [02:58:15] we could I think hack them back in lemme see [02:58:47] oh weird its after a chargeback too [03:00:43] ah i see okay that one we might have had the reference but there was a chargeback inbetween theres a task https://phabricator.wikimedia.org/T364860 [03:01:21] we also never actually tested autorescue with ideal since it was turned on by accident (and since turned back off) [03:01:26] ah right - yeah that's odd - maybe I should stop poking around in them :-) [03:01:52] yeah that one is like adyen aciddentaly turned it on, lead to untested payments, then things are happening because they were never tested [03:03:05] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Add Auto Rescue for SEPA Direct Debit - https://phabricator.wikimedia.org/T362575#9819565 (10Cstone) Edge case here to check on with a chargeback {T364860} [03:07:41] ok just reran that audit [03:45:34] cool [04:12:39] (03PS2) 10Eileen: Remove no-longer-used property [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034579 (https://phabricator.wikimedia.org/T363959) [04:12:39] (03PS3) 10Eileen: Remove old WMFDataManagement.updateWMFDonor api [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034580 (https://phabricator.wikimedia.org/T363959) [04:12:39] (03PS1) 10Eileen: Fix WMFDonor.update to work on non-donors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034643 (https://phabricator.wikimedia.org/T363959) [04:15:50] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fail Mail (civi1002) run-job: Get recipient data from Silverpop failed with code 1 - https://phabricator.wikimedia.org/T365544 (10Eileenmcnaughton) 03NEW [04:16:13] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: We lost some stats moving the donations queue consumer form drush - https://phabricator.wikimedia.org/T362206#9819583 (10Eileenmcnaughton) Just noting that I think this is DONE now - @cstone ? [04:38:19] 14Fundraising Sprint: didAnyoneTryThis(), 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Year end update for donor segments & statuses - https://phabricator.wikimedia.org/T344216#9819633 (10Eileenmcnaughton) After some digging on this I think that now that Joseph... [04:42:55] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Analytics request for 5 years of donor segment & status fields - https://phabricator.wikimedia.org/T344215#9819652 (10Eileenmcnaughton) @JMando - per conversation 1) you are no longer wanting us to add these fields but... [04:43:21] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Civi-Dedupe: Unnecessary deduple conflicts on language - https://phabricator.wikimedia.org/T298169#9819654 (10Eileenmcnaughton) a:03Eileenmcnaughton [04:47:48] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Update RML donors to have status of non-donor in CiviCRM - https://phabricator.wikimedia.org/T365534#9819681 (10MSuijkerbuijk_WMF) @Eileenmcnaughton @KHaggard @ERoden-WMF Hi all, jumping in with a suggestion in case it helps: in the past we've only targ... [06:57:43] (03PS1) 10Eileen: Fix call to create when it should update in newly deployed code [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034800 [07:00:48] ^^ should address the fail mail we just had [07:21:58] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1034813 (owner: 10L10n-bot) [07:42:07] (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1034813 (owner: 10L10n-bot) [11:14:28] (03CR) 10Damilare Adedoyin: [C:03+2] Fix call to create when it should update in newly deployed code [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034800 (owner: 10Eileen) [11:14:42] thanks for the fix eileen [11:15:03] I'd deploy that as soon as it passes to quiet down the failmails [11:34:20] (03Merged) 10jenkins-bot: Fix call to create when it should update in newly deployed code [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034800 (owner: 10Eileen) [11:35:46] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034891 [11:39:33] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034891 (owner: 10Damilare Adedoyin) [11:42:08] Ahh we broke all the PayPal recurs we can pause the job damilare too until one of us is more awake [11:42:50] Er pause the queue [11:47:04] (03CR) 10Damilare Adedoyin: "Recheck" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034891 (owner: 10Damilare Adedoyin) [11:49:24] (03CR) 10Damilare Adedoyin: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034891 (owner: 10Damilare Adedoyin) [11:50:16] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034891 (owner: 10Damilare Adedoyin) [11:53:44] no worries cstone, I'm on it now [12:07:06] 06Fundraising-Backlog: Issue with using Address Proximity in a Searchkit - https://phabricator.wikimedia.org/T360205#9820761 (10NNichols) Yes the idea is that anyone on a team could enter a zip code based on where they are visiting and get a list back that matches this criteria [12:09:41] ok looks like that eileen's patch did help quiet it down, but it isn't writing the right values in the db [12:09:56] for failure count and failure retry date [13:14:43] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q4 maintenance window - https://phabricator.wikimedia.org/T359885#9820982 (10Jgreen) [13:17:00] (03PS1) 10Damilare Adedoyin: Ensure failure count is updated correctly when recording failed payment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034914 [13:21:46] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q4 maintenance window - https://phabricator.wikimedia.org/T359885#9821043 (10Jgreen) [13:27:47] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_41) - 10https://gerrit.wikimedia.org/r/1034684 (owner: 10L10n-bot) [13:28:03] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q4 maintenance window - https://phabricator.wikimedia.org/T359885#9821071 (10Jgreen) [13:30:55] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q4 maintenance window - https://phabricator.wikimedia.org/T359885#9821083 (10Jgreen) [13:33:44] oh man [13:33:54] noisy night [13:35:22] damilare: so did it stop writing anything there? [13:36:00] yea they are all in the damaged queue for now [13:36:25] (03CR) 10CI reject: [V:04-1] Ensure failure count is updated correctly when recording failed payment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034914 (owner: 10Damilare Adedoyin) [13:36:37] want to get that patch up before requeueing https://gerrit.wikimedia.org/r/1034914 so we can update the failure count correctly [13:37:51] hmm, I think for PayPal we want to use the failure count they send us, without adding anything [13:38:38] oh I see, are they not sending us the retry count in the IPN? [13:39:15] (03PS2) 10Damilare Adedoyin: Ensure failure count is updated correctly when recording failed payment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034914 [13:40:55] ok, i guess it's not there in the one I'm looking at now anyway [13:41:14] no I didn't see any in the messages I checked either [13:41:34] I thought it was a count we maintain on our end [13:41:38] they do send us a next_payment_date [13:41:49] yep, I guess we do maintain it on our end for PayPal too [13:42:05] dunno where I got the notion that they kept track of that for us too [13:42:23] so let's see, are we mapping next_payment_date from the IPN to the retry date at least? [13:42:54] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 13Patch-For-Review: Non-donor status missing records in our segmentation framework - https://phabricator.wikimedia.org/T363959#9821186 (10ERoden-WMF) Thanks @Eileenmcnaughton I don't think the numbering had any significance, particularly for the status... [13:43:13] I don't see next_payment_date being mapped to anything in the ipn job [13:43:51] Oh I see, it's in the SubscriptionMessage class [13:43:55] OK, so that seems right [13:44:09] oh wait [13:44:27] no, we only map 'retry_at' and 'failure_retry_date' [13:44:40] so maybe we should also map 'next_payment_date' [13:44:45] right I see it too [13:45:08] is next_payment_date same as failure_retry_date? [13:50:14] (03PS3) 10Damilare Adedoyin: Ensure failure count is updated correctly when recording failed payment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034914 [13:58:38] damilare: I think maybe just for payment_failure type emails [13:59:38] damilare: how will that patch work when we need to set failure_count back to zero? [14:00:06] perhaps we should add one in the getRecurringFailCount ? [14:00:16] right [14:00:25] that makes sense [14:00:44] and then explicitly set it to zero when the type is not subscr_failed [14:01:02] back out in normalize [14:03:08] ok [14:03:40] in the IPN subscription message, we might have to add a new case [14:03:51] the txn_type for the message I saw is: recurring_payment_failed [14:04:33] (03PS1) 10Ejegg: Map next_payment_date to failure_retry_date [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034923 [14:05:29] but the case is expecting 'subscr_failed' [14:05:44] unless we did a mapping earlier, which I didn't see [14:06:27] oh hmm [14:06:36] could it be in that var_map bit? [14:10:44] I think that has to do with the key "txn_type" not the value [14:13:33] (03CR) 10CI reject: [V:04-1] Ensure failure count is updated correctly when recording failed payment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034914 (owner: 10Damilare Adedoyin) [14:15:25] actually ejegg I don't think we should be resetting the count to 0 in the importSubscriptionPaymentFailed function [14:15:47] we'd probably want to do that in a payment import [14:16:02] might be better to have the normalize function return the current state as is in the db [14:17:24] (03CR) 10Pcoombe: [C:04-1] "Testing this locally (using the fundraising-dev Docker environment) I always get the new donate skin when loading Special:LandingPage, eve" [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1020408 (https://phabricator.wikimedia.org/T257862) (owner: 10Jdlrobson) [14:17:30] oh right [14:20:53] (03PS4) 10Damilare Adedoyin: Ensure failure count is updated correctly when recording failed payment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034914 [14:22:08] (03PS2) 10Pcoombe: POC: Potential styles [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1020409 (https://phabricator.wikimedia.org/T257862) (owner: 10Jdlrobson) [14:22:08] (03CR) 10Pcoombe: "Thanks for this. Donate wiki also hosts some pages which are quite different from the donation forms e.g. https://donate.wikimedia.org/wik" [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1020409 (https://phabricator.wikimedia.org/T257862) (owner: 10Jdlrobson) [14:22:51] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q4 maintenance window - https://phabricator.wikimedia.org/T359885#9821540 (10Jgreen) [14:47:31] (03PS2) 10Ejegg: Map next_payment_date to failure_retry_date [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034923 [14:47:47] ok Damilare, I think ^^^ should catch the correct message types [14:47:58] looking at your patch again [14:47:59] (03CR) 10CI reject: [V:04-1] Map next_payment_date to failure_retry_date [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034923 (owner: 10Ejegg) [14:48:11] oops [14:48:51] (03PS3) 10Ejegg: Map next_payment_date to failure_retry_date [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034923 [14:55:36] (03CR) 10Damilare Adedoyin: "Thanks for adding this, looks good to me" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034923 (owner: 10Ejegg) [14:56:07] (03CR) 10Damilare Adedoyin: [C:03+2] Map next_payment_date to failure_retry_date [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034923 (owner: 10Ejegg) [14:56:38] (03Merged) 10jenkins-bot: Map next_payment_date to failure_retry_date [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034923 (owner: 10Ejegg) [15:02:29] (03CR) 10Ejegg: [C:03+2] Ensure failure count is updated correctly when recording failed payment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034914 (owner: 10Damilare Adedoyin) [15:11:38] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034959 [15:12:04] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034959 (owner: 10Ejegg) [15:13:40] let's see if there are minor lib updates to ship to the IPN listener [15:17:19] (03PS1) 10Ejegg: Update libraries [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034964 [15:17:33] Those seem safe enough [15:26:17] (03Merged) 10jenkins-bot: Ensure failure count is updated correctly when recording failed payment [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034914 (owner: 10Damilare Adedoyin) [15:26:18] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034959 (owner: 10Ejegg) [15:26:51] would do the civicrm deploy also [15:28:43] thanks damilare [15:29:54] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034967 [15:31:04] (03CR) 10Damilare Adedoyin: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034967 (owner: 10Damilare Adedoyin) [15:31:59] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1034967 (owner: 10Damilare Adedoyin) [15:34:12] !log civicrm upgraded from 8c5fee40 to b0a3965a [15:34:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:34:33] now to test with one of the damaged trxns [15:41:29] ok that worked but we should probably remove that date from the failure_retry_date [15:41:48] the date value seems to be the start date [15:52:53] (03PS1) 10Damilare Adedoyin: Don't add failure retry date if not specified [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034974 [16:06:21] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Add activies when recurring donation starts and leaves autorescue flow - https://phabricator.wikimedia.org/T361256#9822226 (10Cstone) a:03Cstone [16:14:16] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Not capturing transactions in ZAR - https://phabricator.wikimedia.org/T364792#9822257 (10EMartin) @damilare, can you confirm that all transactions that should have been captured are now captured? Also, what is the status of the audit files that should be... [16:32:19] (03CR) 10Ejegg: [C:03+2] Update libraries [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034964 (owner: 10Ejegg) [16:32:53] (03Merged) 10jenkins-bot: Update libraries [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1034964 (owner: 10Ejegg) [16:35:11] (03PS1) 10Ejegg: Update libraries [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1034983 [16:36:38] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 [16:36:40] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 (owner: 10Ejegg) [16:36:52] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 (owner: 10Ejegg) [16:47:32] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Not capturing transactions in ZAR - https://phabricator.wikimedia.org/T364792#9822500 (10Damilare) Hi @EMartin we were able to capture donor payments that were stuck in authorization within the date range 9th of May to 14th of May, 2024. Due to @MBeat33 r... [16:47:57] (03CR) 10Ejegg: [V:03+2 C:03+2] Update libraries [wikimedia/fundraising/SmashPig/vendor] - 10https://gerrit.wikimedia.org/r/1034983 (owner: 10Ejegg) [16:49:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Not capturing transactions in ZAR - https://phabricator.wikimedia.org/T364792#9822563 (10EMartin) Thanks @Damilare I will chase them for an update. [16:49:49] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 (owner: 10Ejegg) [16:50:02] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 (owner: 10Ejegg) [16:50:06] (03CR) 10Ejegg: [V:03+2 C:03+2] "recheck" [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 (owner: 10Ejegg) [16:50:20] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 (owner: 10Ejegg) [16:54:14] (03PS2) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 [16:54:26] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 (owner: 10Ejegg) [16:54:49] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1034984 (owner: 10Ejegg) [16:54:49] PROBLEM - Host frdb1005 is DOWN: PING CRITICAL - Packet loss = 100% [16:56:11] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [16:56:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [16:57:00] (03PS2) 10Damilare Adedoyin: Update recur status to failed when recurring is failing and use appropriate date for retry date if available. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034974 [16:57:11] PROBLEM - check_mysql on frdb1003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_mysql [16:57:23] ^^^ these are expected and should clear shortly now that frdb1005 is back up. [16:58:23] !log standalone SmashPig upgraded from a9c5ee43 to edf573bb [16:58:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:58:41] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Not capturing transactions in ZAR - https://phabricator.wikimedia.org/T364792#9822658 (10Damilare) Thank you. [16:59:51] RECOVERY - Host frdb1005 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [17:01:11] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 514688 Threads: 4 Questions: 23685168 Slow queries: 236 Opens: 1526 Open tables: 1137 Queries per second avg: 46.018 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [17:01:13] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 12442 Threads: 4 Questions: 341702 Slow queries: 14162 Opens: 1594 Open tables: 1587 Queries per second avg: 27.463 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [17:02:11] RECOVERY - check_mysql on frdb1003 is OK: Uptime: 9872 Threads: 33 Questions: 338113 Slow queries: 71 Opens: 1520 Open tables: 1393 Queries per second avg: 34.249 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_mysql [17:04:57] PROBLEM - Host frqueue1004 is DOWN: PING CRITICAL - Packet loss = 100% [17:07:11] RECOVERY - Host frqueue1004 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [17:09:36] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q4 maintenance window - https://phabricator.wikimedia.org/T359885#9822723 (10Jgreen) [17:19:11] 06Fundraising-Backlog, 10fundraising-tech-ops, 10Wikimedia-Fundraising-CiviCRM: for next maintenance window - Remove disabled custom fields - https://phabricator.wikimedia.org/T356443#9822746 (10Dwisehaupt) 05Open→03Resolved Deleted the rows: ` MariaDB [civicrm]> select id,name,label from civicrm_cus... [17:19:14] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q4 maintenance window - https://phabricator.wikimedia.org/T359885#9822750 (10Dwisehaupt) [17:34:38] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021: Convert banner history from a drupal table to a column in contribution_tracking - https://phabricator.wikimedia.org/T365090#9822805 (10Dwisehaupt) Tried the first query but looks like the `ci... [17:46:34] 06Fundraising-Backlog: Consider moving our mail to a subdomain - https://phabricator.wikimedia.org/T365637 (10spatton) 03NEW [17:58:14] 06Fundraising-Backlog: Recurring Donor (segment_id 400) records have conflicting status Deep Lapsed (status_id 60) - https://phabricator.wikimedia.org/T365639 (10KHaggard) 03NEW [18:00:36] 06Fundraising-Backlog: Recurring Donor (segment_id 400) records have conflicting status Deep Lapsed (status_id 60) - https://phabricator.wikimedia.org/T365639#9822881 (10KHaggard) [18:03:04] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q4 maintenance window - https://phabricator.wikimedia.org/T359885#9822892 (10Dwisehaupt) [18:14:58] 06Fundraising-Backlog: Recurring Donor (segment_id 400) records have conflicting status Deep Lapsed (status_id 60) - https://phabricator.wikimedia.org/T365639#9822917 (10KHaggard) Also here's a screenshot in case it helps: {F54146208} [18:42:55] looks like the quickest way to get the snooze in would be to use the CRM_Utils_Hook::summary [18:43:11] but it's only able to put the info above or below the summary [18:58:41] (03PS7) 10Jdlrobson: Introduce the Donate skin [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1020408 (https://phabricator.wikimedia.org/T257862) [18:59:10] (03CR) 10Jdlrobson: "Fixed. I used a string rather than boolean (face palm)" [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1020408 (https://phabricator.wikimedia.org/T257862) (owner: 10Jdlrobson) [18:59:49] (03CR) 10Jdlrobson: "You likely know what the better defaults should be - do you want to amend this patchset as you see fit?" [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1020409 (https://phabricator.wikimedia.org/T257862) (owner: 10Jdlrobson) [19:06:00] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021: Convert banner history from a drupal table to a column in contribution_tracking - https://phabricator.wikimedia.org/T365090#9823011 (10Eileenmcnaughton) @Dwisehaupt are the first query was to... [19:07:52] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021: Convert banner history from a drupal table to a column in contribution_tracking - https://phabricator.wikimedia.org/T365090#9823014 (10Eileenmcnaughton) it should have been ALTER TABLE ci... [19:11:48] (03CR) 10Ejegg: [C:04-1] "Needs a different status" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034974 (owner: 10Damilare Adedoyin) [19:12:14] dwisehaupt: did you see ^^ - I also need to do another db update based on feedback I just got [19:12:20] but it might not be crazy [19:12:23] yeah, just saw that. [19:12:28] but will need a trigger reload [19:12:34] which the civi upgrade will do [19:12:44] so i ran the column update. was instant. [19:13:05] do you want me to run the second query or hold off on the other feedback/info? [19:31:04] (03PS1) 10Eileen: Fix WMFDonor statuses so that we can use MIN() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035026 (https://phabricator.wikimedia.org/T363959) [19:33:10] (03PS3) 10Damilare Adedoyin: Update recur status to failed when recurring is failing and use appropriate date for retry date if available. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034974 [19:36:45] (03PS1) 10Damilare Adedoyin: Throw failmail on custom filters error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) [19:37:40] (03CR) 10Damilare Adedoyin: "Thanks ejegg for the CR, made the recommended change." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034974 (owner: 10Damilare Adedoyin) [19:37:48] dwisehaupt: I think the second query is ok but we need to stop the queues [19:38:34] (03CR) 10CI reject: [V:04-1] Throw failmail on custom filters error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) (owner: 10Damilare Adedoyin) [19:40:35] (03CR) 10Damilare Adedoyin: "Recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) (owner: 10Damilare Adedoyin) [19:41:23] ok. i can do that whenever. [19:45:08] are any of the queues running through coworker at this time? [19:49:36] 06Fundraising-Backlog, 07Epic, 10MW-1.43-notes (1.43.0-wmf.4; 2024-05-07): Rename utm_ URL parameters through donatewiki and paymentswiki - https://phabricator.wikimedia.org/T351325#9823117 (10Pcoombe) [19:50:39] 06Fundraising-Backlog, 07Epic, 10MW-1.43-notes (1.43.0-wmf.4; 2024-05-07): Rename utm_ URL parameters through donatewiki and paymentswiki - https://phabricator.wikimedia.org/T351325#9823130 (10Pcoombe) Banner code is updated, made some test donations and they look good! Next step is donatewiki which will be... [19:52:48] dwisehaupt: nothing that would be impacted - only low volume snooze that would not matter [19:52:49] (03CR) 10Ejegg: "As long as we're touching this, let's improve the 'nonsense' logging" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) (owner: 10Damilare Adedoyin) [19:54:24] ok cool, then i'll leave coworker up for now. [19:54:43] (03PS2) 10Damilare Adedoyin: Throw failmail on custom filters error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) [19:55:13] (03PS3) 10Damilare Adedoyin: Throw failmail on custom filters error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) [19:55:14] (03CR) 10Ejegg: [C:03+2] Throw failmail on custom filters error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) (owner: 10Damilare Adedoyin) [19:57:26] (03CR) 10CI reject: [V:04-1] Throw failmail on custom filters error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) (owner: 10Damilare Adedoyin) [19:57:53] waiting for the current queue consumers to wrap up. [20:01:13] ok. gonna start the second query now. [20:08:39] (03CR) 10Ejegg: [C:03+2] Fix WMFDonor statuses so that we can use MIN() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035026 (https://phabricator.wikimedia.org/T363959) (owner: 10Eileen) [20:10:48] ejegg: so if dwisehaupt is now doing the banner history update what is the order - with the patches I did the new field will be populated but your follow ons seem to require co-ordination over 2 repos? [20:11:10] so this needs to go first? https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1032889/1/gateway_common/DonationData.php [20:12:01] eileen: yes, we need to start adding it to the c_t messages [20:12:18] cstone or damilare could one of you look at that DonationInterface patch? [20:12:33] we just want to add that banner history log ID to the contribution tracking messages [20:14:01] so this one - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1032117 - we need to deploy before we turn the queues back on - banner history is disabled on prod but not locally already [20:15:26] (03PS3) 10Eileen: Decommission Banner History module, add column to contribution_tracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032117 (https://phabricator.wikimedia.org/T270672) [20:16:38] sure [20:17:20] (03CR) 10Damilare Adedoyin: [C:03+2] Add banner history log ID directly to c_t message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1032889 (https://phabricator.wikimedia.org/T270672) (owner: 10Ejegg) [20:17:53] (03CR) 10CI reject: [V:04-1] Decommission Banner History module, add column to contribution_tracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032117 (https://phabricator.wikimedia.org/T270672) (owner: 10Eileen) [20:19:59] (03PS4) 10Damilare Adedoyin: Throw failmail on custom filters error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1035027 (https://phabricator.wikimedia.org/T365469) [20:21:17] (03PS4) 10Eileen: Decommission Banner History module, add column to contribution_tracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032117 (https://phabricator.wikimedia.org/T270672) [20:21:55] ejegg: since dwisehaupt is running the queries I resolved the merge conflict such that the upgrade won't actually run on prod - but is mostly there for a 'record' [20:23:58] ok, cool [20:24:12] thanks for the CR damilare [20:24:34] I'll deploy that DonationInterface patch [20:27:07] np ejegg, please can you C+2 this again (finally got it to pass CI): https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1035027 and help review this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1034974 when you get the change [20:27:50] (03PS1) 10Eileen: Re-key donor statuses [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1035039 [20:28:34] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog: Deep Lapsed records are being labeled as Ultra Lapsed - https://phabricator.wikimedia.org/T365523#9823296 (10Eileenmcnaughton) 05Open→03Resolved [20:29:01] (03Merged) 10jenkins-bot: Fix WMFDonor statuses so that we can use MIN() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035026 (https://phabricator.wikimedia.org/T363959) (owner: 10Eileen) [20:29:52] 06Fundraising-Backlog: Civi / Adyen mismatch re: recurring retries - https://phabricator.wikimedia.org/T365519#9823312 (10MBeat33) One more from today: cid=63845570 VXQMX5TH3SPTS5X3 (refused at Adyen) cid=1993497 VXQMX5TH3SPTS5X3 (refused at Adyen) 112142885.28 is common to both CIDs [20:35:33] (03Merged) 10jenkins-bot: Add banner history log ID directly to c_t message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1032889 (https://phabricator.wikimedia.org/T270672) (owner: 10Ejegg) [20:38:57] Query OK, 60257643 rows affected (36 min 48.125 sec) [20:39:02] ok. it finished up. [20:39:43] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1035043 [20:39:45] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1035043 (owner: 10Ejegg) [20:40:59] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1035043 (owner: 10Ejegg) [20:41:23] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021: Convert banner history from a drupal table to a column in contribution_tracking - https://phabricator.wikimedia.org/T365090#9823356 (10Dwisehaupt) Ok, both queries have been run. Here's the o... [20:42:21] so eileen you'd still want to run the updated banner history queue consumer after I deploy that, to get anything that's been queued up in the meantime [20:42:48] eileen: et al: let me know when you want the queue consumers turned back on. [20:43:03] (03CR) 10CI reject: [V:04-1] Decommission Banner History module, add column to contribution_tracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032117 (https://phabricator.wikimedia.org/T270672) (owner: 10Eileen) [20:43:04] ejegg: ok - we we would deploy my one that starts to work with the new field & then turn the job on & then after 'a bit' merge the rest of yours? [20:44:18] yep yep [20:44:28] !log payments-wiki upgraded from 5b86bd09 to d871e439 [20:44:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:45:02] that's a weird fail: testRecurringInitialSchemeTxnId [20:45:08] arg - it passes locally [20:46:04] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032117 (https://phabricator.wikimedia.org/T270672) (owner: 10Eileen) [20:46:11] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2677 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [20:46:11] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2680 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [20:46:40] doing recheck in case it magically works [20:47:27] ^^ those are expected as replication rolls through. [20:47:32] (03PS11) 10Eileen: 5.74 rc, stock" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1027983 (https://phabricator.wikimedia.org/T364339) [20:47:39] i'll ack them in icinga. [20:47:47] cool [20:48:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2801 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [21:03:17] ok, I think there should be at least one c_t message in the queue with a banner history log id in it [21:06:13] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 27142 Threads: 13 Questions: 621772 Slow queries: 35451 Opens: 1711 Open tables: 1632 Queries per second avg: 22.908 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [21:06:32] (03CR) 10CI reject: [V:04-1] 5.74 rc, stock" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1027983 (https://phabricator.wikimedia.org/T364339) (owner: 10Eileen) [21:10:20] (03PS2) 10Eileen: Follow on to Contribution insert switch - re-throw the exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034604 [21:10:20] (03PS1) 10Eileen: Clean up confusion where donor segment is non donor but status is not [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035047 (https://phabricator.wikimedia.org/T363959) [21:11:21] (03PS2) 10Eileen: Clean up confusion where donor segment is non donor but status is not [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035047 (https://phabricator.wikimedia.org/T363959) [21:12:37] ejegg: I think maybe we should do ^^ clean up too while the queues are off - it is not that slow but would prob cause deadlocks [21:13:15] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 528264 Threads: 4 Questions: 23915790 Slow queries: 241 Opens: 1584 Open tables: 1123 Queries per second avg: 45.272 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [21:13:27] woohoo - recheck worked - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1032117 [21:17:28] nice [21:18:27] (03CR) 10Ejegg: [C:03+2] Decommission Banner History module, add column to contribution_tracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032117 (https://phabricator.wikimedia.org/T270672) (owner: 10Eileen) [21:19:00] (03CR) 10Ejegg: [C:03+2] "Let's make sure we run the c_t queue consumer before we turn this banner history consumer back on, so the updates always get a row to upda" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032117 (https://phabricator.wikimedia.org/T270672) (owner: 10Eileen) [21:19:27] ok, i need to pick up my kid. I'll look at the segment cleanup patches as soon as I can [21:22:53] (03PS12) 10Eileen: 5.74 rc, stock" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1027983 (https://phabricator.wikimedia.org/T364339) [21:26:11] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 530588 Threads: 4 Questions: 23936831 Slow queries: 236 Opens: 1592 Open tables: 1129 Queries per second avg: 45.113 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [21:38:33] (03Merged) 10jenkins-bot: Decommission Banner History module, add column to contribution_tracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032117 (https://phabricator.wikimedia.org/T270672) (owner: 10Eileen) [21:44:26] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1035051 [21:44:45] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1035051 (owner: 10Eileen) [21:45:27] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1035051 (owner: 10Eileen) [21:45:43] (03CR) 10CI reject: [V:04-1] 5.74 rc, stock" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1027983 (https://phabricator.wikimedia.org/T364339) (owner: 10Eileen) [21:52:16] (03PS2) 10Eileen: Re-key donor statuses [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1035039 [21:56:49] !log civicrm upgraded from b0a3965a to 84c36324 [21:56:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:59:17] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021: Convert banner history from a drupal table to a column in contribution_tracking - https://phabricator.wikimedia.org/T365090#9823592 (10Dwisehaupt) [21:59:43] 03Fundraising Sprint: justWork(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fr-drupal-upgrade-2021: Convert banner history from a drupal table to a column in contribution_tracking - https://phabricator.wikimedia.org/T365090#9823594 (10Dwisehaupt) Updated the description to have the correct... [22:02:30] I like the way all the jobs have slow start now :-) [22:02:58] I just deployed & ran contribution tracking & banner history - I suspect we can deploy the other part now [22:03:28] (03CR) 10Eileen: [C:03+2] Use banner_history_log_id from c_t queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032893 (https://phabricator.wikimedia.org/T270672) (owner: 10Ejegg) [22:03:47] although we should disable the job first [22:04:29] (03CR) 10Eileen: [C:03+1] "yay - but we should disable the job before we deploy this" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1032895 (https://phabricator.wikimedia.org/T270672) (owner: 10Ejegg) [22:15:46] great! [22:16:36] (03PS3) 10Eileen: Follow on to Contribution insert switch - re-throw the exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034604 [22:16:37] (03CR) 10Ejegg: [C:03+2] Follow on to Contribution insert switch - re-throw the exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034604 (owner: 10Eileen) [22:16:42] (03PS3) 10Eileen: Clean up confusion where donor segment is non donor but status is not [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035047 (https://phabricator.wikimedia.org/T363959) [22:16:44] (03CR) 10Ejegg: [C:03+2] Clean up confusion where donor segment is non donor but status is not [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035047 (https://phabricator.wikimedia.org/T363959) (owner: 10Eileen) [22:22:40] (03CR) 10Ejegg: "Should the default be the number or the text?" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1035039 (owner: 10Eileen) [22:23:01] oh yeah - the text [22:24:30] I have to take Jack to an apptment - will try to get back on line from there - the 5.74 isn't passing & I can see why I just need to push in a fix [22:28:19] (03PS3) 10Eileen: Re-key donor statuses [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1035039 [22:28:31] (03CR) 10Eileen: "yep!" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1035039 (owner: 10Eileen) [22:29:16] (03CR) 10CI reject: [V:04-1] Re-key donor statuses [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1035039 (owner: 10Eileen) [22:29:32] (03PS1) 10Eileen: Temp load options [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035053 [22:40:58] (03Merged) 10jenkins-bot: Follow on to Contribution insert switch - re-throw the exception [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1034604 (owner: 10Eileen) [22:40:59] (03Merged) 10jenkins-bot: Clean up confusion where donor segment is non donor but status is not [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1035047 (https://phabricator.wikimedia.org/T363959) (owner: 10Eileen) [22:54:54] I can do the trigger commit [22:56:26] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1035059 [22:56:29] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1035059 (owner: 10Ejegg) [22:57:16] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1035059 (owner: 10Ejegg) [23:07:44] let me know when it's all ready for me to do the trigger update.