[00:05:47] wow, drupal db dropped from 109G to 6G by dropping those tables. [00:06:00] good bit of data duplication there. :) [00:11:44] yeah quick pull in save 100GB of disk space into our sprint so we can tick it off [00:13:38] :) [00:17:13] I think I see a new retrospective OKR :) [00:19:01] other than the usage drop, all the other host and db metrics looked aok after the work. [00:33:00] cstone: do you mean I can include that text in the description of the phab? 👍🏽 [00:36:06] cool [00:52:46] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog: Test Adyen Auto-Rescue - https://phabricator.wikimedia.org/T357458#9603848 (10Cstone) second round of testing looks good, we turned it on today to start fully with the 2024-03-06 charge [00:54:42] anilk: feel free to edit the phab if you want I was just putting placeholder text [00:58:30] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.21; 2024-03-05), 07Spike: PayTM In India: What do we do - https://phabricator.wikimedia.org/T358625#9603854 (10AKanji-WMF) [01:09:05] thanks anilk ! [01:16:08] 06Fundraising-Backlog: American Express needs to be accounted for differently in Civi - https://phabricator.wikimedia.org/T318250#9603905 (10Cstone) For looking at searchkit vs superset, @RKumar_WMF what superset are you using to compare to? That way we can match the conditions in searchkit to what superset is... [01:47:34] (03PS1) 10Ejegg: Alphabetize some email form things [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008980 (https://phabricator.wikimedia.org/T356370) [02:13:06] (03PS2) 10Ejegg: WIP new variant for recurring upgrade landing pages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008974 (https://phabricator.wikimedia.org/T356370) [02:13:08] (03PS1) 10Ejegg: Move $recurringOptions calculation out of EmailForm [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008982 (https://phabricator.wikimedia.org/T356370) [05:31:21] doing the trigger update now. [05:33:49] taking longer than expected since it's waiting for a table on a long running stats query. [05:48:09] they are all complete, new trigger diff matches, all dbs back replicating. just a few failmails from the queries backing up but all with jobs that run regularly and don't need any fixes. [06:10:57] cool [07:21:50] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1009168 (owner: 10L10n-bot) [07:44:04] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.21; 2024-03-05): Add ACH payment type for US donation processing - https://phabricator.wikimedia.org/T324517#9604729 (10RKumar_WMF) Can you please provide the recurring link too @AnnWF ? [10:26:23] (03PS2) 10Raimond Spekking: Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1009168 (owner: 10L10n-bot) [10:26:39] (03CR) 10Raimond Spekking: [C: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1009168 (owner: 10L10n-bot) [10:33:49] fail mail with contribution recur [10:36:06] ok they are all paypal, so not related to the AutoRescue feature [10:38:53] looks like its failing to pull up the recur record with the subscr_id [10:40:04] but I was able to do so using searchkit [10:40:10] checking code [11:31:14] ok something weird is going, the status of most of this failing recurring is empty with ID 0. [11:32:32] not sure what caused that but from the logs table, looks like there's a pattern for cancelled recurring to status 0 and happened between Jan and Feb. [11:34:44] looks like its unique to paypal [11:39:17] so ignoring the history for now and focusing on the current failmail, this line is the trigger for the failmail: https://github.com/wikimedia/wikimedia-fundraising-crm/blob/fcbf6b4839dbc8987495f11742985bd35cf1a68b/drupal/sites/default/civicrm/extensions/wmf-civicrm/Civi/WMFQueueMessage/RecurDonationMessage.php#L70 [11:43:42] looking through the conditions for that line, it looks like it also makes sense for a status id of 0 to return false for the isRecurringFound function call that checks the value of the status_id [11:44:05] here: return (bool) $this->getContributionRecurValue('contribution_status_id'); [11:44:16] https://github.com/wikimedia/wikimedia-fundraising-crm/blob/fcbf6b4839dbc8987495f11742985bd35cf1a68b/drupal/sites/default/civicrm/extensions/wmf-civicrm/Civi/WMFQueueMessage/RecurDonationMessage.php#L81C28-L81C53 [11:45:44] however, there are some recurs with cancelled status (id not 0) that seem to be failing also. I'm not sure why that is, I'd write a test for that and debug through [11:46:56] however I'm about 70% sure, that line L81 and status id of 0 are the cause of the current failmail [12:23:05] hi damilare [12:23:15] just saw the failmails [12:23:20] how can I help [12:25:36] hey Jack, I'm just writing a test to debug and confirm my findings. Seems somewhere in the new code, we are changing paypal cancelled status id to 0 which leads to the error. [12:26:03] I can push up the test I'm using to debug, so you also can do so on your end [12:28:17] just looking at the code you mentioned erlier [12:28:21] earlier* [12:34:15] the contribution_status_id=0 looks iffy [12:34:50] ok, so this appears to be the reason for the 0 status_id [12:34:52] https://github.com/wikimedia/wikimedia-fundraising-crm/blob/fcbf6b4839dbc8987495f11742985bd35cf1a68b/drupal/sites/default/civicrm/extensions/wmf-civicrm/Civi/WMFHelper/ContributionRecur.php#L75 [12:36:35] ok we're accidentally assigning a string to the int [12:36:59] CiviCRM does some mapping of statuses at times [12:37:05] is that code outside of that [12:37:23] In Progress = 5 right? [12:37:52] MariaDB [civicrm]> SELECT ov.value AS contribution_status_id, ov.name AS contribution_status_name [12:37:54] -> FROM civicrm.civicrm_option_value ov [12:37:56] -> INNER JOIN civicrm.civicrm_option_group og ON (og.id = ov.option_group_id) [12:37:58] -> WHERE og.name = 'contribution_recur_status'; [12:38:00] +------------------------+--------------------------+ [12:38:02] | contribution_status_id | contribution_status_name | [12:38:04] +------------------------+--------------------------+ [12:38:06] | 1 | Completed | [12:38:08] | 2 | Pending | [12:38:10] | 3 | Cancelled | [12:38:12] | 6 | Overdue | [12:38:14] | 4 | Failed | [12:38:16] | 5 | In Progress | [12:38:18] | 14 | Processing | [12:38:20] | 15 | Failing | [12:38:22] +------------------------+--------------------------+ [12:39:15] damilare: how did you determine that was resulting in the 0 [12:40:25] yh firstly, the logs for the recur records before they were turned to 0 shows that they were in a cancelled status [12:40:27] 3 [12:41:45] and then a modification was made on the records around feb (most likely a payment) that led to the change to 0 [12:42:29] now in this test I wrote, I created a recur record, cancelled it, and added a payment to it. Debugging through entered into this conditional [12:43:50] https://github.com/wikimedia/wikimedia-fundraising-crm/blob/fcbf6b4839dbc8987495f11742985bd35cf1a68b/drupal/sites/default/civicrm/extensions/wmf-civicrm/Civi/WMFQueue/RecurringQueueConsumer.php#L137 [12:44:11] and after it passes through the update, the status changes to 0. [12:44:55] I changed the key/value to this "'contribution_status_id:name' => 'In Progress'," and the test pass through and made the right change [12:45:20] ahhh nice [12:47:50] hmm ok do you wanna push that up with your test and I can review [12:47:58] I guess we also need to fix the affected batch [12:48:02] (03PS1) 10Damilare Adedoyin: Fix failmails due to invalid recur status id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009246 [12:48:51] yea, we can change the status of the affected batch to "In progress" and send them back to the queue from the damaged queue [12:49:24] I think there were 5000+ affected records [12:49:41] yep 5750 [12:50:35] all paypal/paypal_ec [12:51:13] we can run an update on all of them also. [12:56:38] (03CR) 10Jgleeson: [C: 03+2] "Works great. Thanks for the test also!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009246 (owner: 10Damilare Adedoyin) [13:00:38] thanks for the CR jgleeson! I'm just going to push a follow patch to make a small edit to the test to show that the recur was indeed in the cancelled status before the payment message was sent [13:00:49] sure [13:00:52] I should also create a phab for this for visibility [13:05:50] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Failmail caused by paypal recurrings in invalid status id - https://phabricator.wikimedia.org/T359336 (10Damilare) [13:09:32] (03PS1) 10Damilare Adedoyin: Verify record is cancelled in RecurQueueTest [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009248 (https://phabricator.wikimedia.org/T359336) [13:15:59] jgleeson: I'm updating all the empty status to "In progress" on searchkit [13:17:48] damila [13:17:56] oops [13:18:21] yep [13:20:33] I was looking at the api explorer before as it felt a bit safer than raw SQL but sure if you think searchkit makes sense go with that [13:21:23] yh, it's pretty much the same. Done [13:21:41] I was impressed by this fancy cli civi stntax `cv api4 ContributionRecur.update +v contribution_status_id=5 +w 'contribution_status_id = 0'` [13:22:21] that's neat [13:22:26] syntax* [13:22:58] +w is where, but that +v is easy to not remember [13:23:13] values [13:23:21] ahhh [13:23:37] I was thinking "verbosity" [13:23:42] now that makes sense [13:25:53] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1009254 [13:35:38] (03CR) 10Damilare Adedoyin: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1009254 (owner: 10Damilare Adedoyin) [13:36:16] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1009254 (owner: 10Damilare Adedoyin) [13:41:49] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: Recent donations not saved after merging of two CIDs - https://phabricator.wikimedia.org/T326583#9606474 (10jgleeson) 05Open→03Resolved a:03jgleeson thanks @SHust! I'll close this ticket and create a new one for the new job men... [13:44:22] ok resent the items back to the queue after the deployment [13:44:29] let's see if they get sent back [13:50:48] nice [14:08:45] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Failmail caused by paypal recurrings in invalid status id - https://phabricator.wikimedia.org/T359336#9606676 (10Damilare) Original patch here: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1009246/1 [14:09:54] (03CR) 10Damilare Adedoyin: [C: 03+2] "Self +2'ing as its only adding 1 line and some comments to an existing test which wouldn't make it into production." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009248 (https://phabricator.wikimedia.org/T359336) (owner: 10Damilare Adedoyin) [14:19:38] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Failmail caused by paypal recurrings in invalid status id - https://phabricator.wikimedia.org/T359336#9606846 (10Damilare) a:03Damilare [14:28:20] (03Merged) 10jenkins-bot: Verify record is cancelled in RecurQueueTest [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009248 (https://phabricator.wikimedia.org/T359336) (owner: 10Damilare Adedoyin) [14:42:38] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: Recent donations not saved after merging of two CIDs - https://phabricator.wikimedia.org/T326583#9607203 (10jgleeson) 05Resolved→03In progress [14:47:09] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Adyen: Handle issuer timeouts via Adyen - https://phabricator.wikimedia.org/T331317#9607259 (10jgleeson) a:03jgleeson [15:22:41] thanks damilare and jgleeson|skrun for dealing with the chaos! [15:33:41] hey fr-tech: Dylan has some kind of dance performance at school this morning, depending on how long it runs I might not be back for standup. [15:34:49] 06Fundraising-Backlog: Research Task - CiviCRM Migration - https://phabricator.wikimedia.org/T359224#9607565 (10AKanji-WMF) a:03Eileenmcnaughton [15:41:56] np cstone [15:42:42] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences: Suppression list logic - sanity check - https://phabricator.wikimedia.org/T325125#9607584 (10AKanji-WMF) [16:34:21] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10FR-email-preferences, 13Patch-For-Review: Create variant version of Recurring Upgrade landing page - https://phabricator.wikimedia.org/T356370#9607800 (10Ejegg) @ehughes, one question on how the form is supposed to work. There is a bit of t... [16:35:19] 06Fundraising-Backlog: Enable 90 day snooze: Comms Preferences Centre update - https://phabricator.wikimedia.org/T358878#9607808 (10AKanji-WMF) Flagging that the functionality we are working towards is to enable supporters to also select comms preferences in the form: https://docs.google.com/presentation/d/11s... [16:36:15] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10FR-email-preferences, 13Patch-For-Review: Create variant version of Recurring Upgrade landing page - https://phabricator.wikimedia.org/T356370#9607814 (10Ejegg) @ehughes I'm also going to tweak the supplied CSS a bit to make the submit butt... [16:41:21] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Adyen: Handle issuer timeouts via Adyen - https://phabricator.wikimedia.org/T331317#9607864 (10jgleeson) Our timeout is already 60 seconds for Adyen, and this will be retired 3 times. I'm now curious whether that means the donor will have to sit th... [16:51:02] cstone: thanks for fixing the acls on that amex task [16:51:30] hah i am not entirely sure the difference of the two but rakhi and evelyn cant see the nda [16:54:51] 06Fundraising-Backlog: Add a job to check that deleted contact donations are not lost. - https://phabricator.wikimedia.org/T359422 (10jgleeson) [16:58:47] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Adyen: Better handle terse Adyen idempotency result - https://phabricator.wikimedia.org/T331319#9607951 (10jgleeson) a:03jgleeson [17:05:34] cstone: they should/could technically be added to WMF-NDA but the FR acl is sufficient so whatever [17:06:21] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog: SEPA direct debit as mainstream payment method in EU campaigns - https://phabricator.wikimedia.org/T324520#9607976 (10AKanji-WMF) [17:11:01] (03PS1) 10Damilare Adedoyin: Cause failmail when queue task fails and gets aborted. Also set logs for single failures of task run. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009307 (https://phabricator.wikimedia.org/T358259) [17:11:19] (03PS2) 10Damilare Adedoyin: Cause failmail when queue task fails and gets aborted. Also set logs for single failures of task run. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009307 (https://phabricator.wikimedia.org/T358259) [17:14:09] (03PS3) 10Ejegg: New variant for recurring upgrade landing pages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008974 (https://phabricator.wikimedia.org/T356370) [17:22:39] (03CR) 10Wfan: [C: 03+2] Alphabetize some email form things [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008980 (https://phabricator.wikimedia.org/T356370) (owner: 10Ejegg) [17:36:03] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10FR-email-preferences, 13Patch-For-Review: Create variant version of Recurring Upgrade landing page - https://phabricator.wikimedia.org/T356370#9608163 (10ehughes) @Ejegg Thanks for this, I would defer to @MSuijkerbuijk_WMF and/or @spatton f... [18:39:45] 14Fundraising Sprint Highway to Hello World, 14Fundraising Sprint Istanbool (not Constantidouble), 06Fundraising-Backlog, 10fundraising-tech-ops: CT-to-Civi-Phase 2 - drop the drupal contribution_tracking table - https://phabricator.wikimedia.org/T334542#9608360 (10Dwisehaupt) 05Open→03Resolved a:03Dw... [18:39:47] 14Fundraising Sprint Istanbool (not Constantidouble), 06Fundraising-Backlog, 07Epic: Scope contribution tracking - https://phabricator.wikimedia.org/T182030#9608364 (10Dwisehaupt) [18:41:42] 14Fundraising Sprint Highway to Hello World, 14Fundraising Sprint Istanbool (not Constantidouble), 06Fundraising-Backlog, 10fundraising-tech-ops: CT-to-Civi-Phase 2 Disable the module & delete the code - https://phabricator.wikimedia.org/T334543#9608384 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt... [18:41:44] 14Fundraising Sprint Istanbool (not Constantidouble), 06Fundraising-Backlog, 07Epic: Scope contribution tracking - https://phabricator.wikimedia.org/T182030#9608389 (10Dwisehaupt) [18:47:56] damilare: I'm going to steal T356479 from you in Phab if you don't mind [18:47:58] T356479: Show total amount once donor chooses amount - upgrading landing page - https://phabricator.wikimedia.org/T356479 [18:48:35] thanks for the review, wfan! I think the follow-on should be pretty quick too: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1008982 [18:48:45] yeah sure no p ejegg, you mentioned it during stand up [18:49:50] :) [18:50:51] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for sbaruchel - https://phabricator.wikimedia.org/T355901#9608398 (10Dwisehaupt) Certificate renewed and sent via email. Password sent via SMS. [18:50:55] (oh neat, the archived projects show up as grey) [18:57:43] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10FR-email-preferences: Show total amount once donor chooses amount - upgrading landing page - https://phabricator.wikimedia.org/T356479#9608408 (10Ejegg) a:05Damilare→03Ejegg [18:59:57] (03PS2) 10Ejegg: Show working in value of TTL [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1006949 (owner: 10Esanders) [19:00:05] (03CR) 10Ejegg: [C: 03+2] Show working in value of TTL [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1006949 (owner: 10Esanders) [19:01:15] 06Fundraising-Backlog: 2024 Sprint E - Priorities - https://phabricator.wikimedia.org/T359434 (10AKanji-WMF) 03NEW [19:08:08] (03PS3) 10Ejegg: Cause failmail when queue task fails and gets aborted. Also set logs for single failures of task run. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009307 (https://phabricator.wikimedia.org/T358259) (owner: 10Damilare Adedoyin) [19:08:22] (03CR) 10Ejegg: [C: 03+2] "Looks good!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009307 (https://phabricator.wikimedia.org/T358259) (owner: 10Damilare Adedoyin) [19:09:28] thanks for the CR ejegg [19:09:47] 06Fundraising-Backlog: 2024 Sprint E - Priorities - https://phabricator.wikimedia.org/T359434#9608457 (10AKanji-WMF) [19:09:58] yw! [19:10:41] damilare: I was just looking at those failmails with 'Contribution recur ID passed in but could not be loaded' and the damaged UI link gives me an error: https://civicrm.wikimedia.org/civicrm/damaged/edit?id=617866 [19:10:58] Any idea on that UI error or the reason for those failmails? [19:11:00] ohh that's probably because I've resent all of them [19:11:24] ohh yh that was the errors from this morning I spoke about on standup [19:11:35] ah ok, there's just no 'message X not found' handling yet [19:12:11] it was caused because some contribution_status_id were set to 0 when paypal reactivated a cancelled recurring and a payment came through [19:12:47] so this https://github.com/wikimedia/wikimedia-fundraising-crm/blob/fcbf6b4839dbc8987495f11742985bd35cf1a68b/drupal/sites/default/civicrm/extensions/wmf-civicrm/Civi/WMFQueueMessage/RecurDonationMessage.php#L81 [19:12:57] returns a false for the status id value check [19:13:19] here's the patch that fixed it: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1009246 [19:13:35] yep, we should add a "Not found" UI to the damaged table [19:13:45] I'd create a phab for that [19:15:06] ah, thanks for the explanation! [19:15:17] 06Fundraising-Backlog: 2024 Sprint E - Priorities - https://phabricator.wikimedia.org/T359434#9608491 (10AKanji-WMF) [19:16:19] np [19:16:19] 03Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Ensure Damaged UI shows "Not Found" when a message no longer exist - https://phabricator.wikimedia.org/T359437 (10Damilare) 03NEW [19:24:08] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.21; 2024-03-05): ACH recurring - https://phabricator.wikimedia.org/T359441 (10AnnWF) 03NEW [19:25:05] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.21; 2024-03-05): Add ACH payment type for US donation processing - https://phabricator.wikimedia.org/T324517#9608568 (10AnnWF) >>! In T324517#9604729, @RKumar_WMF wrote: > Can you please provide the recurring link to... [19:26:40] (03Merged) 10jenkins-bot: Cause failmail when queue task fails and gets aborted. Also set logs for single failures of task run. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009307 (https://phabricator.wikimedia.org/T358259) (owner: 10Damilare Adedoyin) [19:28:56] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1009320 [19:30:19] (03Merged) 10jenkins-bot: Move $recurringOptions calculation out of EmailForm [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008982 (https://phabricator.wikimedia.org/T356370) (owner: 10Ejegg) [19:30:22] (03Merged) 10jenkins-bot: Update reference to newly namespaced MessageGroupStates::class [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/990793 (https://phabricator.wikimedia.org/T340724) (owner: 10Wangombe) [19:30:25] (03Merged) 10jenkins-bot: Show working in value of TTL [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1006949 (owner: 10Esanders) [19:32:29] (03CR) 10Damilare Adedoyin: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1009320 (owner: 10Damilare Adedoyin) [19:33:21] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1009320 (owner: 10Damilare Adedoyin) [19:55:48] 06Fundraising-Backlog: Add comms preferences footer to all Civi-generated emails - https://phabricator.wikimedia.org/T359445 (10AKanji-WMF) 03NEW [20:41:49] dwisehaupt or Jeff_Green can one of you help us understand some stuff in tech talk? [20:41:54] https://meet.google.com/odz-juym-nog [20:42:03] sure [20:50:59] (03PS1) 10Ejegg: Fix queue fail log [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009351 [20:56:39] !log changed wmf_cli logger to point to stderr instead of stdout [20:56:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:00:15] PROBLEM - check_puppetrun on civi1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[coworker] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [21:05:15] RECOVERY - check_puppetrun on civi1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [21:07:11] PROBLEM - check_puppetrun on frdev1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[coworker] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [21:10:38] ejegg: is that fredge one ok to merge now? [21:12:15] RECOVERY - check_puppetrun on frdev1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [21:15:15] PROBLEM - check_puppetrun on civi1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[coworker] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [21:17:09] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:20:15] RECOVERY - check_puppetrun on civi1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [21:22:09] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:26:06] let me look once more eileen, i think so [21:27:09] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:31:39] 14Fundraising Spring - anybodyKnowWhatThisDoes(), 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 13Patch-For-Review: Correcting Gift Source Coding on Benevity import - https://phabricator.wikimedia.org/T354911#9609343 (10Eileenmcnaughton) I just spoke to @NNichols & it seems OK to only ma... [21:32:09] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:32:40] (03PS1) 10Wfan: Enable Recurring for ACH [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1009355 (https://phabricator.wikimedia.org/T359441) [21:33:25] (03PS1) 10Wfan: Add ACH recurring test link [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1008922 (https://phabricator.wikimedia.org/T359441) [21:33:37] (03CR) 10CI reject: [V: 04-1] Enable Recurring for ACH [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1009355 (https://phabricator.wikimedia.org/T359441) (owner: 10Wfan) [21:34:47] (03PS1) 10Wfan: Enable ACH recurring [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008923 (https://phabricator.wikimedia.org/T359441) [21:35:43] (03PS2) 10Wfan: Enable Recurring for ACH [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1009355 (https://phabricator.wikimedia.org/T359441) [21:35:49] (03CR) 10CI reject: [V: 04-1] Enable ACH recurring [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008923 (https://phabricator.wikimedia.org/T359441) (owner: 10Wfan) [21:37:09] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:37:15] (03PS2) 10Wfan: Enable ACH recurring [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1008923 (https://phabricator.wikimedia.org/T359441) [21:37:34] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog: Incorrect Gift Source - https://phabricator.wikimedia.org/T356894#9609364 (10Eileenmcnaughton) [21:38:06] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog: Incorrect Gift Source - https://phabricator.wikimedia.org/T356894#9609365 (10Eileenmcnaughton) 05Open→03Resolved Fixed this on a call with Nora [21:38:47] cstone and ejegg, ACH recurring is ready to test, wonder if everything looks good, could we deploy that tomorrow for Rakhi to test? [21:40:14] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.21; 2024-03-05), 13Patch-For-Review: ACH recurring - https://phabricator.wikimedia.org/T359441#9609373 (10AnnWF) once deploy, the test link could be: https://payments.wikimedia.org/index.php?title=Special:AdyenChec... [21:41:59] (03CR) 10Damilare Adedoyin: [C: 03+2] Fix queue fail log [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009351 (owner: 10Ejegg) [21:42:09] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:47:07] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:52:09] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:53:01] ACKNOWLEDGEMENT - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker Dwisehaupt checking our errors after restart. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [21:54:17] 06Fundraising-Backlog: 2024 Sprint E - Priorities - https://phabricator.wikimedia.org/T359434#9609414 (10AKanji-WMF) [21:56:16] 03Fundraising Sprint: didAnyoneTryThis(), 06Fundraising-Backlog: Reattempt failed Ingencio transactions from Oct, Nov, Dec 2023 - https://phabricator.wikimedia.org/T353677#9609420 (10AKanji-WMF) Hi @EMartin - can you please provide more criteria as to who we need to email to let them know we're going to restar... [21:59:49] (03Merged) 10jenkins-bot: Fix queue fail log [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1009351 (owner: 10Ejegg) [22:06:25] (03PS5) 10Ejegg: Move data wranging for fredge to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1007451 (https://phabricator.wikimedia.org/T357469) (owner: 10Eileen) [22:06:29] (03CR) 10Ejegg: [C: 03+2] Move data wranging for fredge to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1007451 (https://phabricator.wikimedia.org/T357469) (owner: 10Eileen) [22:23:32] (03Merged) 10jenkins-bot: Move data wranging for fredge to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1007451 (https://phabricator.wikimedia.org/T357469) (owner: 10Eileen) [22:52:09] RECOVERY - check_coworker on frdev1002 is OK: PROCS OK: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [23:02:09] PROBLEM - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [23:06:48] ACKNOWLEDGEMENT - check_coworker on frdev1002 is CRITICAL: PROCS CRITICAL: 0 processes with args php /srv/coworker/bin/coworker Dwisehaupt acking again after the process attempted a restart cleanly. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [23:17:09] RECOVERY - check_coworker on frdev1002 is OK: PROCS OK: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_coworker [23:53:48] 10fundraising-tech-ops: Fundraising access request for jcolvin - https://phabricator.wikimedia.org/T357457#9609673 (10Dwisehaupt) [23:54:07] 10fundraising-tech-ops: Fundraising access request for jcolvin - https://phabricator.wikimedia.org/T357457#9609674 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt [23:54:51] 10fundraising-tech-ops: Fundraising access request for iackerman - https://phabricator.wikimedia.org/T357456#9609678 (10Dwisehaupt) [23:55:06] 10fundraising-tech-ops: Fundraising access request for iackerman - https://phabricator.wikimedia.org/T357456#9609679 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt [23:55:37] 10fundraising-tech-ops: Fundraising access request for jroberson - https://phabricator.wikimedia.org/T357817#9609683 (10Dwisehaupt) [23:55:59] 10fundraising-tech-ops: Fundraising access request for jroberson - https://phabricator.wikimedia.org/T357817#9609684 (10Dwisehaupt) 05Open→03Resolved [23:58:29] 10fundraising-tech-ops: Fundraising access request for lhowland - https://phabricator.wikimedia.org/T359496 (10Dwisehaupt) 03NEW [23:59:04] 10fundraising-tech-ops: Fundraising access request for lhowland - https://phabricator.wikimedia.org/T359496#9609703 (10Dwisehaupt) 05Open→03Resolved