[00:35:07] PROBLEM - check_load on franio2003 is CRITICAL: LOAD CRITICAL - total load average: 14.54, 53.43, 39.66 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [00:40:07] PROBLEM - check_load on franio2003 is CRITICAL: LOAD CRITICAL - total load average: 4.65, 22.77, 30.13 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [00:45:07] PROBLEM - check_load on franio2003 is CRITICAL: LOAD CRITICAL - total load average: 3.99, 9.73, 22.40 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [00:50:05] PROBLEM - check_load on franio2003 is CRITICAL: LOAD CRITICAL - total load average: 0.06, 3.81, 16.36 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [01:05:07] RECOVERY - check_load on franio2003 is OK: LOAD OK - total load average: 6.17, 5.80, 9.35 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=franio2003&service=check_load [11:04:16] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Add text to CentralNotice admin UI when user has admin but no 2FA - https://phabricator.wikimedia.org/T394321#11789381 (10Johannnes89) I believe this task can be declined due to {T418580}? It's no longer possible to hold CN admin permissions without... [15:01:32] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Add text to CentralNotice admin UI when user has admin but no 2FA - https://phabricator.wikimedia.org/T394321#11789952 (10Ejegg) 05Open→03Declined Yep, sounds good @Johannnes89 [15:13:05] 06Fundraising-Backlog: Gravy recurring CLP failing - https://phabricator.wikimedia.org/T422389 (10Cstone) 03NEW [15:14:23] 06Fundraising-Backlog: Gravy recurring CLP failing - https://phabricator.wikimedia.org/T422389#11790021 (10Cstone) [15:16:21] 06Fundraising-Backlog: Gravy recurring CLP failing - https://phabricator.wikimedia.org/T422389#11790027 (10Cstone) [15:17:57] 06Fundraising-Backlog: Gravy recurring CLP failing - https://phabricator.wikimedia.org/T422389#11790077 (10Cstone) [15:19:21] 06Fundraising-Backlog: Bre-B Payment method in Colombia - https://phabricator.wikimedia.org/T422391 (10RKumar_WMF) 03NEW [15:28:16] (03PS5) 10Wfan: Trigger failmail if a validation error not categorized [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1262356 (https://phabricator.wikimedia.org/T421155) [15:29:06] (03CR) 10Ejegg: [C:03+2] Trigger failmail if a validation error not categorized [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1262356 (https://phabricator.wikimedia.org/T421155) (owner: 10Wfan) [15:29:31] (03PS2) 10Wfan: Render failmail for all validation errors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264861 (https://phabricator.wikimedia.org/T421155) [15:29:36] (03CR) 10Ejegg: [C:03+2] Render failmail for all validation errors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264861 (https://phabricator.wikimedia.org/T421155) (owner: 10Wfan) [15:30:05] oh I should rebase one of those over the other [15:30:26] (03PS3) 10Wfan: Render failmail for all validation errors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264861 (https://phabricator.wikimedia.org/T421155) [15:30:43] (03CR) 10Ejegg: [C:03+2] Render failmail for all validation errors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264861 (https://phabricator.wikimedia.org/T421155) (owner: 10Wfan) [15:34:25] ???? Did SmashPig get set as a private repo in github? [15:34:40] I'm unable to composer update with the regular ssh credentials [15:34:48] it's asking for an access token [15:34:51] seems sketchy [15:35:41] or is it a different repo? [15:35:56] I can clone it elsewhere with ssh [15:36:25] oh lollll [15:36:33] i forgot, I had cloned locally [15:37:06] 'remote' url was just pointing to the main smashpig src folder [15:37:10] ok, false alarm [15:37:53] (03PS1) 10Ejegg: Update SmashPig to v1.2.3.22 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268233 [15:37:57] (03CR) 10Ejegg: [C:03+2] Update SmashPig to v1.2.3.22 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268233 (owner: 10Ejegg) [15:41:26] (03PS1) 10Ejegg: Update SmashPig, plus minor lib updates [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1268234 [15:41:43] (03CR) 10Ejegg: [C:03+2] Update SmashPig, plus minor lib updates [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1268234 (owner: 10Ejegg) [15:42:31] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1268235 [15:42:37] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1268235 (owner: 10Ejegg) [15:57:04] 06Fundraising-Backlog, 06FR-donorrelations: Gravy recurring CLP failing - https://phabricator.wikimedia.org/T422389#11790192 (10MBeat33) [15:59:18] (03Merged) 10jenkins-bot: Trigger failmail if a validation error not categorized [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1262356 (https://phabricator.wikimedia.org/T421155) (owner: 10Wfan) [15:59:20] (03Merged) 10jenkins-bot: Render failmail for all validation errors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1264861 (https://phabricator.wikimedia.org/T421155) (owner: 10Wfan) [15:59:22] (03CR) 10CI reject: [V:04-1] Update SmashPig to v1.2.3.22 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268233 (owner: 10Ejegg) [16:00:30] (03CR) 10Ejegg: [V:03+2 C:03+2] "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268233 (owner: 10Ejegg) [16:03:09] !log civicrm upgraded from 4ee11209 to 9104e70b [16:03:10] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [16:04:04] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add a campaign to email activities for double opt in emails - https://phabricator.wikimedia.org/T422395 (10Lars) 03NEW [16:10:04] (03Merged) 10jenkins-bot: Update SmashPig, plus minor lib updates [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1268234 (owner: 10Ejegg) [16:10:37] (03CR) 10Wfan: [C:03+1] "LGTM, any reason still wip?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267948 (https://phabricator.wikimedia.org/T389865) (owner: 10Ejegg) [16:15:43] (03CR) 10Ejegg: "Just needs a smoke test!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267948 (https://phabricator.wikimedia.org/T389865) (owner: 10Ejegg) [16:15:51] (03PS1) 10Wfan: Update SmashPig to v1.2.3.22 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1268241 [16:23:09] (03CR) 10Wfan: [C:03+2] Update SmashPig to v1.2.3.22 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1268241 (owner: 10Wfan) [16:24:18] hmm, we have an inconsistency in the trxn_id and invoice_id recorder on gravy recurrings [16:24:59] some have the full RECURRING GRAVY 2901274b-cf1f-4b71-b552-4bcc5aa07279 trxn_id plus an invoice_id, and some have the bare gateway_txn_id and no invoice_id [16:37:41] (03Merged) 10jenkins-bot: Update SmashPig to v1.2.3.22 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1268241 (owner: 10Wfan) [16:53:21] !log disabled recurring donations charge job while diagnosing gr4vy routing errors [16:53:22] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [17:01:23] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Employee matching emails are not being sent - https://phabricator.wikimedia.org/T421187#11790495 (10Lars) Verified that this is working as intended. [17:05:08] (03CR) 10Lars SG: "See comment" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267894 (https://phabricator.wikimedia.org/T422051) (owner: 10Wfan) [17:05:56] (03CR) 10Lars SG: "See comment" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267894 (https://phabricator.wikimedia.org/T422051) (owner: 10Wfan) [17:06:51] u r right! Update it~ [17:22:43] (03PS3) 10Wfan: Only demote the primary if email has location type [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267894 (https://phabricator.wikimedia.org/T422051) [17:23:51] (03PS1) 10Wfan: Add more unit test coverage for verify email queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) [17:27:24] (03PS1) 10Wfan: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1268250 [17:27:39] (03CR) 10Wfan: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1268250 (owner: 10Wfan) [17:27:43] (03CR) 10Wfan: [V:03+2 C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1268250 (owner: 10Wfan) [17:52:43] (03CR) 10CI reject: [V:04-1] Add more unit test coverage for verify email queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) (owner: 10Wfan) [18:17:39] (03PS1) 10Wfan: Update vendor composer.lock for smashpig version 1.2.3.22 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268261 [18:17:45] (03CR) 10Zuul test: "Merge Failed." [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268261 (owner: 10Wfan) [18:18:05] 06Fundraising-Backlog, 07payments-orchestration: general gravy validation error as bad_request - https://phabricator.wikimedia.org/T421958#11790797 (10Ejegg) Looks like some of these were caused by misconfigured flow rules in the gr4vy console. [18:18:14] (03CR) 10CI reject: [V:04-1] Update vendor composer.lock for smashpig version 1.2.3.22 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268261 (owner: 10Wfan) [18:25:11] (03PS2) 10Wfan: Update vendor composer.lock for smashpig version 1.2.3.22 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268261 [18:25:23] (03CR) 10Zuul test: "Merge Failed." [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268261 (owner: 10Wfan) [18:25:54] (03CR) 10CI reject: [V:04-1] Update vendor composer.lock for smashpig version 1.2.3.22 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268261 (owner: 10Wfan) [18:26:10] 06Fundraising-Backlog, 07payments-orchestration: Always route Gravy recurring transactions through their original connection - https://phabricator.wikimedia.org/T422416 (10Ejegg) 03NEW [18:27:48] (03PS3) 10Wfan: Update vendor composer.lock for smashpig version 1.2.3.22 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268261 [18:30:52] 06Fundraising-Backlog, 07payments-orchestration: Always route Gravy recurring transactions through their original connection - https://phabricator.wikimedia.org/T422416#11790858 (10Ejegg) Looks like we would do this by adding the payment_service_id to the createPayment call: https://docs.gr4vy.com/reference/tr... [18:36:44] (03CR) 10Lars SG: "Works like this for me" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267894 (https://phabricator.wikimedia.org/T422051) (owner: 10Wfan) [19:19:40] (03CR) 10Wfan: [C:03+2] Update vendor composer.lock for smashpig version 1.2.3.22 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268261 (owner: 10Wfan) [19:20:46] (03CR) 10Wfan: Only demote the primary if email has location type (033 comments) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267894 (https://phabricator.wikimedia.org/T422051) (owner: 10Wfan) [19:20:51] (03PS4) 10Wfan: Only demote the primary if email has location type [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267894 (https://phabricator.wikimedia.org/T422051) [19:26:35] (03CR) 10Lars SG: [C:03+2] Only demote the primary if email has location type [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267894 (https://phabricator.wikimedia.org/T422051) (owner: 10Wfan) [19:29:54] (03PS1) 10Ejegg: Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268269 [19:30:01] (03CR) 10Ejegg: [C:03+2] Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268269 (owner: 10Ejegg) [19:30:58] (03CR) 10Lars SG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) (owner: 10Wfan) [19:33:09] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Smashpig, 10Recurring-Donations: Ensure validation errors in recurring charges send failmail - https://phabricator.wikimedia.org/T421155#11791007 (10Ejegg) This is deployed, and we certainly are getting some failmail! After deployment the patch p... [19:37:15] (03PS2) 10Ejegg: Add setting for arbitrary charge retry cadence [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267104 (https://phabricator.wikimedia.org/T413905) [19:39:06] (03Merged) 10jenkins-bot: Merge branch 'REL1_43' into fundraising/REL1_43 [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1268269 (owner: 10Ejegg) [20:32:09] !log payments-wiki upgraded from 80cda166 to 1ad85e6c [20:32:10] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [20:33:32] !log donorwiki upgraded from c2d03117 to 1ad85e6c [20:33:33] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [20:42:17] !log re-enabled recurring donation charge job [20:42:18] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [20:47:42] (03PS6) 10Ejegg: Refactor SmashPig::doRefund to work with API4 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267431 (https://phabricator.wikimedia.org/T421277) [20:47:42] (03PS8) 10Ejegg: Form to submit refunds to processors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1262277 (https://phabricator.wikimedia.org/T421277) [21:12:32] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: API4 Request Authorization failed: UserJob::get on cv flush - https://phabricator.wikimedia.org/T422425 (10Lars) 03NEW [21:12:54] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: API4 Request Authorization failed: UserJob::get on cv flush - https://phabricator.wikimedia.org/T422425#11791351 (10Lars) [[ https://github.com/civicrm/civicrm-core/pull/35332 | Upstream PR ]] [21:49:37] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for emartin - https://phabricator.wikimedia.org/T416547#11791423 (10Dwisehaupt) certificate renewed and sent via email. password sent via SMS. [21:53:09] (03CR) 10Lars SG: "See comment, I think it's a little confusing with a mix of 0 and 1-indexing." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1267104 (https://phabricator.wikimedia.org/T413905) (owner: 10Ejegg) [21:53:29] (03PS1) 10Ejegg: Add contribution status to context for civicrm_links hook [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268279 (https://phabricator.wikimedia.org/T421277) [22:01:53] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: API4 Request Authorization failed: UserJob::get on cv flush - https://phabricator.wikimedia.org/T422425#11791447 (10Lars) a:03Lars [22:10:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Call import hook from batch import - https://phabricator.wikimedia.org/T421978#11791488 (10Lars) Requires {T422221} [22:45:05] (03PS2) 10Wfan: Add more unit test coverage for verify email queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) [22:45:26] (03PS3) 10Wfan: Add more unit test coverage for verify email queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) [22:48:08] (03PS4) 10Wfan: Add more unit test coverage for verify email queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) [22:48:17] (03PS5) 10Wfan: Add more unit test coverage for verify email queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) [23:01:40] (03PS6) 10Wfan: Add more unit test coverage for verify email queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) [23:23:01] (03CR) 10CI reject: [V:04-1] Add more unit test coverage for verify email queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1268249 (https://phabricator.wikimedia.org/T422051) (owner: 10Wfan)