[00:25:52] (03PS1) 10Eileen: Fix batch_merge job [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1180688 [00:27:50] (03PS2) 10Eileen: Fix batch_merge job [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1180688 [00:30:24] (03PS3) 10Eileen: Fix batch_merge job [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1180688 [00:48:36] (03PS1) 10Ejegg: Pass 'checksum' through DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180690 (https://phabricator.wikimedia.org/T402462) [00:51:01] (03CR) 10CI reject: [V:04-1] Pass 'checksum' through DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180690 (https://phabricator.wikimedia.org/T402462) (owner: 10Ejegg) [01:03:10] (03CR) 10Ejegg: [C:03+2] "Code looks good, seems safe to try out!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1180688 (owner: 10Eileen) [01:03:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-dlocal, 10MediaWiki-extensions-DonationInterface: DLocal CO donation form failing with no error message - https://phabricator.wikimedia.org/T400818#11105004 (10AnnWF) Update from Cristiano Betta https://wikimedia.slack.com/archives/C070F1DVBRN/p1... [01:04:18] (03PS2) 10Ejegg: Pass 'checksum' through DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180690 (https://phabricator.wikimedia.org/T402462) [01:07:23] (03PS1) 10Wfan: INVALID_MESSAGE Invalid data in e-mail preferences message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180692 (https://phabricator.wikimedia.org/T402200) [01:07:54] (03PS2) 10Wfan: INVALID_MESSAGE Invalid data in e-mail preferences message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180692 (https://phabricator.wikimedia.org/T402200) [01:08:44] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 13Patch-For-Review: dLocal missing tax ID + status mismatch errors - https://phabricator.wikimedia.org/T401428#11105010 (10AnnWF) a:05jgleeson→03AnnWF [01:11:50] (03CR) 10CI reject: [V:04-1] INVALID_MESSAGE Invalid data in e-mail preferences message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180692 (https://phabricator.wikimedia.org/T402200) (owner: 10Wfan) [01:14:34] (03PS3) 10Wfan: INVALID_MESSAGE Invalid data in e-mail preferences message [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180692 (https://phabricator.wikimedia.org/T402200) [01:20:28] (03Merged) 10jenkins-bot: Fix batch_merge job [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1180688 (owner: 10Eileen) [01:23:16] !log payments-wiki upgraded from 1235f11f to 0944453c [01:23:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:31:52] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07payments-orchestration: For PayPal via Gravy, show appropriate text when donor cancels - https://phabricator.wikimedia.org/T402472 (10Ejegg) 03NEW [01:39:27] (03PS3) 10Ejegg: Pagination for donation list tables [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) [01:41:28] !log payments-wiki rolled back from 0944453c to 1235f11f [01:41:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:07:18] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1180694 [02:08:08] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1180694 (owner: 10Eileen) [02:08:54] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1180694 (owner: 10Eileen) [02:10:44] !log civicrm upgraded from 279b3993 to c3b729c0 [02:10:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:01:28] (03PS1) 10Eileen: Declare more message fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1180698 [03:43:29] (03CR) 10CI reject: [V:04-1] Declare more message fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1180698 (owner: 10Eileen) [04:53:14] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 38.34, 55.69, 33.63 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [04:55:18] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 27.25, 47.30, 32.70 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [05:18:12] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 2.84, 2.65, 9.11 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [05:20:16] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 1.81, 2.23, 8.22 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [06:28:51] 06Fundraising-Backlog, 06FR-donorrelations: Snooze status flapping - https://phabricator.wikimedia.org/T402464#11105333 (10Eileenmcnaughton) So observations on https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=65029231 As of now Thu Aug 21 06:24:15 AM UTC 2025 I can see he IS still snoozed - this... [06:32:41] !log config revision changed from 45c6fa38 to 3378426a [06:32:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:46:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2283 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [06:51:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2583 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [06:56:14] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 651933 Threads: 11 Questions: 58437334 Slow queries: 834601 Opens: 8448 Open tables: 2482 Queries per second avg: 89.637 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [08:30:48] (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180754 (owner: 10L10n-bot) [13:10:29] (03CR) 10Jgleeson: [C:04-1] "This is still not working for me. The redirect now kicks in, but I'm still getting errors preventing a successful transaction. I used the " [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180153 (https://phabricator.wikimedia.org/T392065) (owner: 10Ejegg) [13:17:15] (03CR) 10Damilare Adedoyin: "Thanks ejegg, this looks good to me just had a suggestion in the comments. It would be good to have some unit tests to this to support the" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) (owner: 10Ejegg) [13:40:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 78.92, 65.00, 35.33 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:43:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 59.38, 61.31, 38.56 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:45:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 11.81, 39.88, 33.36 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [13:48:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 3.33, 29.24, 31.58 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [13:53:49] (03PS44) 10Jgleeson: Add Gravy payment error threshold monitoring & alerting system. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167604 (https://phabricator.wikimedia.org/T387411) [14:03:00] (03PS45) 10Jgleeson: Add Gravy payment error threshold monitoring & alerting system. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167604 (https://phabricator.wikimedia.org/T387411) [14:03:00] (03PS1) 10Jgleeson: Move redis connection settings outside data-store namespace [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180871 (https://phabricator.wikimedia.org/T387411) [14:10:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 1.99, 2.71, 8.69 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [14:13:13] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 3.69, 3.27, 8.73 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [14:26:15] PROBLEM - check_procs on frdb1006 is CRITICAL: PROCS CRITICAL: 4131 processes https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_procs [14:30:15] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: investigate why an Adyen / Gravy recurring donation failed with 'invalid date' - https://phabricator.wikimedia.org/T402345#11106751 (10jgleeson) a:03jgleeson [14:31:15] PROBLEM - check_procs on frdb1006 is CRITICAL: PROCS CRITICAL: 4222 processes https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_procs [14:39:09] (03CR) 10Ejegg: "See comment on the Phab (" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180153 (https://phabricator.wikimedia.org/T392065) (owner: 10Ejegg) [14:41:15] RECOVERY - check_procs on frdb1006 is OK: PROCS OK: 1057 processes https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_procs [14:44:51] ejegg: so what do you wanna with that patch. Merge is not working or wait for the fix [14:54:28] 10fundraising-tech-ops: Fundraising access request for Stacy Burnett - https://phabricator.wikimedia.org/T402061#11106873 (10Jgreen) [14:55:07] 10fundraising-tech-ops: Fundraising access request for Lars Sander-Green - https://phabricator.wikimedia.org/T402062#11106874 (10Jgreen) [15:07:56] (03PS46) 10Jgleeson: Add Gravy payment error threshold monitoring & alerting system. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167604 (https://phabricator.wikimedia.org/T387411) [15:25:21] 10fundraising-tech-ops: Fundraising access request for Lars Sander-Green - https://phabricator.wikimedia.org/T402062#11107065 (10Lars) [15:26:04] 10fundraising-tech-ops: Fundraising access request for Lars Sander-Green - https://phabricator.wikimedia.org/T402062#11107072 (10Lars) [15:43:48] 10fundraising-tech-ops: Fundraising access request for Lars Sander-Green - https://phabricator.wikimedia.org/T402062#11107246 (10Lars) Public key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAdjKB/Y0ltDDhRVxHSzcEuXuB6LqIZ2cc+YJGmEfqt1 lsandergreen@wikimedia.org [15:49:21] (03PS1) 10Jgleeson: Refactor main Redis config structure and standardize queue configurations [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1180899 [15:49:21] (03PS1) 10Jgleeson: Refactor other Redis config structure [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1180900 [15:52:18] ejegg: those two patches ^^ bring our fun-dev smashpig/config inline with the library config as discussed last night. I did test out removing the queue and it seems to work ok although I did see one comment with might suggest otherwise here https://github.com/wikimedia/wikimedia-fundraising-SmashPig/blob/b0aa75f6ad227e190a1b713b24fcec1f09ebd8fe/config/main.yaml#L96 [15:52:42] jgleeson: I figured we could add the createPayment functionality to SmashPig for starters, but if you want to wait till they fix the return val I'm fine with that. It's a little patch, not likely to go stale with waiting [15:52:51] ok, let me look [15:53:31] and this patch is a smol tidy up https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1180871/1 [15:53:34] (i've been playing with vue tests, but probably I need to debug those api.php errors from last night's deploy...) [15:53:34] 10fundraising-tech-ops: Fundraising access request for Lars Sander-Green - https://phabricator.wikimedia.org/T402062#11107329 (10Jgreen) [15:54:13] 10fundraising-tech-ops: Fundraising access request for Lars Sander-Green - https://phabricator.wikimedia.org/T402062#11107331 (10Jgreen) [15:56:15] ejegg: re:sepa. ok, cool, we can merge our side. No one will be using it until we announce it, so it should be ok. Also prevents us from tracking it as a fr-tech responsibility in the ever-growing gravy meeting spreadsheet. [15:57:04] (03CR) 10Jgleeson: [C:03+2] "LGTM. As discussed, we can merge this as-is as the redirect is happening as expected. We can then wait for Gr4vy to fix their integration " [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180153 (https://phabricator.wikimedia.org/T392065) (owner: 10Ejegg) [15:57:45] (03Merged) 10jenkins-bot: SEPA for Gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180153 (https://phabricator.wikimedia.org/T392065) (owner: 10Ejegg) [16:00:45] that spreadsheet... :P [16:01:33] (03PS2) 10Jgleeson: Move redis connection settings outside data-store namespace [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180871 (https://phabricator.wikimedia.org/T387411) [16:01:39] (03CR) 10Ejegg: [C:03+2] Move redis connection settings outside data-store namespace [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180871 (https://phabricator.wikimedia.org/T387411) (owner: 10Jgleeson) [16:01:54] :) [16:02:10] (03Merged) 10jenkins-bot: Move redis connection settings outside data-store namespace [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180871 (https://phabricator.wikimedia.org/T387411) (owner: 10Jgleeson) [16:02:23] (03PS47) 10Jgleeson: Add Gravy payment error threshold monitoring & alerting system. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167604 (https://phabricator.wikimedia.org/T387411) [16:02:56] (03CR) 10Wfan: [C:03+2] Set banner hide cookies to Secure / SameSite=None [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1180165 (https://phabricator.wikimedia.org/T255366) (owner: 10Ejegg) [16:05:20] (03PS4) 10Ejegg: Pagination for donation list tables [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) [16:07:58] (03PS5) 10Ejegg: Pagination for donation list tables [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) [16:11:14] (03CR) 10CI reject: [V:04-1] Pagination for donation list tables [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) (owner: 10Ejegg) [16:15:51] (03Merged) 10jenkins-bot: Set banner hide cookies to Secure / SameSite=None [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1180165 (https://phabricator.wikimedia.org/T255366) (owner: 10Ejegg) [16:18:15] (03PS6) 10Ejegg: Pagination for donation list tables [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) [16:26:04] (03CR) 10Ejegg: "Thanks for the CR, Damilare! I've added tests and simplified numPages." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) (owner: 10Ejegg) [16:31:33] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: process_unsubscribes log activity failed due to contact_id not exit - https://phabricator.wikimedia.org/T401967#11107605 (10Ejegg) Hmm, should we also check for the contact being merged to another contact when the original ID is gone? [16:33:29] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: process_unsubscribes log activity failed due to contact_id not exit - https://phabricator.wikimedia.org/T401967#11107638 (10AnnWF) Good point will add a patch, thanks [16:41:36] 06Fundraising-Backlog, 06FR-donorrelations: Snooze status flapping - https://phabricator.wikimedia.org/T402464#11107667 (10SHust) Thank you so much, @Eileenmcnaughton! [16:41:57] (03CR) 10Ejegg: [V:03+2 C:03+2] Refactor main Redis config structure and standardize queue configurations [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1180899 (owner: 10Jgleeson) [16:42:07] (03CR) 10Ejegg: [V:03+2 C:03+2] Refactor other Redis config structure [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1180900 (owner: 10Jgleeson) [17:05:18] (03CR) 10Damilare Adedoyin: [C:03+2] "Thanks for all the work on this ejegg. Tested and all looks good, thanks for the additional tests to add to the coverage also." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) (owner: 10Ejegg) [17:08:40] (03Merged) 10jenkins-bot: Pagination for donation list tables [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180596 (https://phabricator.wikimedia.org/T400934) (owner: 10Ejegg) [17:09:18] thanks damilare ! [17:09:29] np [17:18:14] 06Fundraising-Backlog: Fundraising Sprint Q - Priorities - https://phabricator.wikimedia.org/T402571 (10AKanji-WMF) 03NEW [17:18:27] 06Fundraising-Backlog: Fundraising Sprint Q - Priorities - https://phabricator.wikimedia.org/T402571#11107879 (10AKanji-WMF) [17:20:13] Well I'm totally stumped about those api.php errors [17:20:24] gonna get lunch and think some more [17:25:00] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: process_unsubscribes log activity failed due to contact_id not exit - https://phabricator.wikimedia.org/T401967#11107909 (10AnnWF) [17:29:24] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM: Annual Recurring Statuses implementation in Civi (Audience Framework update) - https://phabricator.wikimedia.org/T377438#11107929 (10AKanji-WMF) For before big English - line 4 at least "New recurring related fields > add a L... [17:33:28] (03PS1) 10Wfan: do not force fiscal number if not ar br for dlocal [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180915 (https://phabricator.wikimedia.org/T401428) [17:34:02] (03CR) 10CI reject: [V:04-1] do not force fiscal number if not ar br for dlocal [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180915 (https://phabricator.wikimedia.org/T401428) (owner: 10Wfan) [17:53:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 13Patch-For-Review: URGENT: Merge Behavior Changed — Causing RML Emails to Be Sent in Error! - https://phabricator.wikimedia.org/T401752#11108062 (10SHust) Thanks for the update, @Eileenmcnaughton. Please let me know if there's an... [17:58:00] (03PS2) 10Wfan: do not force fiscal number if not ar br for dlocal [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180915 (https://phabricator.wikimedia.org/T401428) [17:58:28] (03CR) 10CI reject: [V:04-1] do not force fiscal number if not ar br for dlocal [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180915 (https://phabricator.wikimedia.org/T401428) (owner: 10Wfan) [18:02:23] (03PS1) 10Wfan: Add queue for set-primary-email [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1180920 (https://phabricator.wikimedia.org/T396523) [18:05:01] so let's see, was it anything like the same issue when Damilare deployed before? [18:05:48] ejegg do you mean that provider error? [18:06:05] the problems with api.php [18:06:40] that was just yesterday, right? [18:07:05] yh I'm unable to reproduce it locally and that showed up on all the api request [18:07:27] yep it was only yesterday [18:08:48] (03PS1) 10Wfan: Add config defaults for set-primary-email queue [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180922 (https://phabricator.wikimedia.org/T396523) [18:08:49] oh could it be the adyen_submit_payment_api_request used by the apps team [18:09:11] from the web logs Jeff_Green just got me, it didn't seem to be [18:09:24] they mostly had referrers of the GravyGateway [18:09:54] ok, but the error in the log does indeed look similar [18:10:08] e.g. Configuration key 'payment-provider/google/class' does not exist [18:10:39] let me just make sure I did deploy the right SHA [18:10:53] and not just re-deploy the same version [18:10:57] 03Fundraising Sprint: Pirate's Booty, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10FR-email-preferences, 13Patch-For-Review: trigger email verification email when dpc queue have email update - https://phabricator.wikimedia.org/T396523#11108120 (10AnnWF) [18:12:02] really looks like it... [18:12:08] (03PS3) 10Wfan: do not force fiscal number if not ar br for dlocal [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180915 (https://phabricator.wikimedia.org/T401428) [18:12:41] (03CR) 10CI reject: [V:04-1] do not force fiscal number if not ar br for dlocal [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180915 (https://phabricator.wikimedia.org/T401428) (owner: 10Wfan) [18:17:08] so could they be submitting something wrong for the gateway? [18:17:34] actually, we should be able to hardcode gateway in the gateway-specific API subclasses [18:19:22] ejegg: I dont seem to have access to the logs, what time did they start coming in [18:22:01] the app endpoint should be stuck hard to adyen unless we changed that? [18:22:42] but if its not hardcoded is it getting switched to gravy automatically? [18:23:01] hmm for that google error, I can only see errors around the time of my deployment in frlog [18:24:51] ok the apple errors are shown at about the time of your deployment [18:25:33] yh we don't have that api set for gravy yet [18:26:44] (03PS1) 10Ejegg: Hardcode $gateway in gateway-specific API classes [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180926 [18:26:55] just grasping at straws, but ^^^ might help [18:28:05] damilare: you can still see the errors in /var/log/remote/payments.error from 01:29 - 01:39 [18:28:12] definitely not all the api.php calls [18:28:29] should we hardcode it in the apple pay one since it can only be adyen right now? [18:28:46] and then Jeff_Green pulled the full web logs for that interval from a couple of servers, and put them in frpm's /tmp dir [18:29:00] cstone: oh, is gateway a parameter there? [18:29:18] i need to look at it again [18:29:26] I oh i see, yeah, I think so [18:29:48] ah but it's already hardcoded [18:29:52] /home/elliott/src/docker/fundraising-dev/src/payments/extensions/DonationInterface/adyen_gateway/adyen_apple.api.php [18:29:55] ugh [18:30:01] not $this->gateway = 'adyen'; [18:30:24] used to needing ctrl-shift-c to copy from terminals, but in the IDE it copies the file path [18:30:41] oh, meeting time! [18:34:20] 10fundraising-tech-ops: Fundraising access request for Stacy Burnett - https://phabricator.wikimedia.org/T402061#11108246 (10Jgreen) [18:39:38] 10fundraising-tech-ops: Fundraising access request for Stacy Burnett - https://phabricator.wikimedia.org/T402061#11108296 (10Jgreen) [18:50:29] (03PS4) 10Wfan: Add is contribution has paypal in epc to determin if show email edit in epc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178090 (https://phabricator.wikimedia.org/T401006) [18:51:11] (03PS5) 10Wfan: Add is contribution has paypal in epc to determin if show email edit in epc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178090 (https://phabricator.wikimedia.org/T401006) [19:08:00] fr-tech would somebody mind sanity-checking this one so I can try seeing if it lets us deploy payments-wiki? https://gerrit.wikimedia.org/r/1180926 [19:08:22] looking ejegg [19:09:16] (03CR) 10Damilare Adedoyin: [C:03+2] Hardcode $gateway in gateway-specific API classes [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180926 (owner: 10Ejegg) [19:09:36] thanks damilare [19:10:01] ejegg: would it be possible to add a smashpig update to the deploy? [19:10:40] it's for the venmo update that was merged yesterday [19:11:01] oh totally [19:11:07] will do [19:11:30] does the SmashPig version need tagging? [19:11:44] yh I'm working on that now [19:11:50] k, cool [19:11:59] oh that'll pick up the SEPA patch too, right? [19:12:18] (03Merged) 10jenkins-bot: Hardcode $gateway in gateway-specific API classes [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180926 (owner: 10Ejegg) [19:15:48] ok, I see .12 on packagist. I'll add the composer.json update [19:16:12] (03PS1) 10Damilare Adedoyin: Update smashpig to v0.9.7.12 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180940 [19:16:29] ah, beat me to it [19:16:49] oops sorry, I'd hand over at this point [19:17:01] (03CR) 10Ejegg: [C:03+2] Update smashpig to v0.9.7.12 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180940 (owner: 10Damilare Adedoyin) [19:24:05] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180941 [19:24:08] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180941 (owner: 10Ejegg) [19:29:24] (03PS1) 10Ejegg: Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1180943 [19:29:27] (03CR) 10Ejegg: [C:03+2] Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1180943 (owner: 10Ejegg) [19:32:32] (03Merged) 10jenkins-bot: Update smashpig to v0.9.7.12 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180940 (owner: 10Damilare Adedoyin) [19:32:33] (03CR) 10CI reject: [V:04-1] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180941 (owner: 10Ejegg) [19:32:43] hrm? [19:33:01] ohhh node still runs its test on deployment [19:33:08] ok then, I guess I'll leave them [19:34:39] (03PS2) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180941 [19:34:42] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180941 (owner: 10Ejegg) [19:38:20] (03PS6) 10Wfan: Add is contribution has paypal in epc to determin if show email edit in epc [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1178090 (https://phabricator.wikimedia.org/T401006) [19:39:03] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180941 (owner: 10Ejegg) [19:39:41] 06Fundraising-Backlog: Pre-renewal notice to Active Annual recurring donors - https://phabricator.wikimedia.org/T393432#11108465 (10AKanji-WMF) Thanks @ppenloglou I will add this to the next sprint - FR-Tech will need to build the logic for this triggered mail. [19:43:03] (03CR) 10CI reject: [V:04-1] Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1180943 (owner: 10Ejegg) [19:47:40] (03PS2) 10Ejegg: Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1180943 [19:47:43] (03CR) 10Ejegg: Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1180943 (owner: 10Ejegg) [19:47:51] PROBLEM - check_memory on frdb2003 is CRITICAL: CRIT Memory 99% used. Largest process: mariadbd (1792394) = 81.2% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_memory [19:50:54] 06Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Tech-Analytics: Metabase Access Request - Steph Toyofuku - https://phabricator.wikimedia.org/T402587 (10AKanji-WMF) 03NEW [19:52:27] PROBLEM - check_memory on frdb2003 is CRITICAL: CRIT Memory 99% used. Largest process: mariadbd (1792394) = 80.6% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_memory [19:53:00] !log payments-wiki upgraded from 1235f11f to 49bef1cf [19:53:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:53:27] argh, immediately getting api.php errors [19:53:44] ok, rolling back again [19:54:06] is it just the apple endpoint or all the apis? [19:54:08] hmm paypal didn't give the eror [19:54:56] !log payments-wiki rolled back from 49bef1cf to 1235f11f [19:54:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:55:15] ahh too late, I was about to test the apple [19:55:35] oops [19:56:03] cstone it seems to be just the gravy api action [19:57:09] RECOVERY - check_memory on frdb2003 is OK: OK Memory 21% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_memory [19:58:25] ejegg the gravy paypal went while it was on [20:00:06] (03Merged) 10jenkins-bot: Update composer.lock for new SmashPig [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1180943 (owner: 10Ejegg) [20:05:01] we could deploy the branch on payment staging and monitor there [20:05:10] some payment methods like apple pay may not work [20:05:22] but I wonder if cc would? [20:11:22] ejegg strange, I was able to put in a transaction on gravy cc in staging with that commit [20:11:42] right, it's certainly not all the payment attempts failing [20:12:01] I guess we could log more [20:12:53] damilare: I did see this additional error in the logs: TypeError: Cannot assign null to property SmashPig\PaymentProviders\Gravy\PaymentProvider::$cacheParameters of type array [20:13:01] seems to be a cc one [20:13:49] checking [20:15:41] (03PS1) 10Damilare Adedoyin: Instantiate cache parameter with array [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180962 [20:18:03] weird, it usually lets you leave it w/o a default, as long as you only assign it non-null things [20:18:21] wouldn't we have had problems in tests if that missing default was the issue, damilare ? [20:18:52] I think its because I specified a type of array but wasn't assigned a value [20:19:18] might have passed as a warning in test [20:19:28] hmm, let me double check locally [20:20:26] my env makes warnings pretty noisy in tests, and no problem on them now [20:21:05] we also have protected PaymentProviderValidator $validator [20:21:12] (with no ? for nullable) [20:21:17] and it's not complaining [20:21:59] same for $requestMapper and $responseMapper [20:22:13] so I think it must somehow be getting null from config [20:22:24] oh, is there an override in production? [20:23:10] I don't think so [20:24:27] right, none of those payment-provider keys are being overridden [20:24:40] ugh, what the heck is happening here? [20:32:10] so if we can't figure this out right now, what should we do in order to be able to deploy payments-wiki? [20:33:09] given we know it could be from that getCurrencies method, we can apply that safety check [20:33:31] in this patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1180582 [20:34:44] hmm, I guess [20:35:28] I wonder if we could log more info [20:35:50] what else would help us understand? [20:36:28] yep we should, the request parameters without the pii, I think we log that already but we could log it earlier [20:36:49] oh right, if we log here we will at least associate it with the ct_id [20:36:57] so that patch is probably good as is [20:37:00] at least for a start [20:37:09] want to restore it? [20:37:15] sure [20:37:19] (03Restored) 10Damilare Adedoyin: Ensure the SmashPig PaymentProvider is gotten safely in getCurrencies. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180582 (owner: 10Damilare Adedoyin) [20:37:25] (03PS8) 10Damilare Adedoyin: Ensure the SmashPig PaymentProvider is gotten safely in getCurrencies. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180582 [20:38:13] (03CR) 10Ejegg: [C:03+2] Ensure the SmashPig PaymentProvider is gotten safely in getCurrencies. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180582 (owner: 10Damilare Adedoyin) [20:40:39] (03Merged) 10jenkins-bot: Ensure the SmashPig PaymentProvider is gotten safely in getCurrencies. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180582 (owner: 10Damilare Adedoyin) [20:42:46] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180971 [20:46:51] ejegg: to be on the safe side can we also deploy the type error fix on smashpig [20:47:28] damilare: but all the other props on there are non-nullable and aren't complaining of a lack of default... [20:50:43] (03CR) 10Damilare Adedoyin: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180971 (owner: 10Damilare Adedoyin) [20:50:53] right and the tests should have failed too [20:51:44] so puzzling [20:52:34] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180971 (owner: 10Damilare Adedoyin) [20:52:59] ok that has got in [20:53:09] Thanks! I'll deploy donorwiki too just to get that pagination out [20:53:18] cool cool [20:54:09] !log donorwiki upgraded from 5dcb98fd to cb76e2b7 [20:54:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:55:15] !log payments-wiki upgraded from 1235f11f to cb76e2b7 [20:55:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:58:01] well, no more exceptions in payments.error, but we are logging that warning dozens of times a minute [20:58:30] but at least we have them linked with the ct_id [20:58:34] so let's take a peek [20:58:40] ahh I knew that'd spam [20:58:56] cc/apple-pay/gpay worked in my tests though [21:03:14] so I can see some of those transactions logged in gravy, so they're definitely getting donations through [21:03:45] searching by external_identifier=ct_id on the gravy dashboard [21:07:57] shall we log the whole api request? [21:08:12] seems like it's a lot of repeat ct_ids there [21:10:48] sure lets do that, I'm seeing some object call missing the payment_method like payment-provider//class [21:11:03] but the payment method is specified in the url [21:14:21] right, it's like some totally broken API calls are being made [21:19:20] (03PS1) 10Ejegg: log post values on error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180976 [21:19:40] damilare: ^^^ [21:19:51] checking [21:20:04] oh let me check the client-side err log [21:22:15] nothing i can see [21:22:39] (03CR) 10Damilare Adedoyin: [C:03+2] log post values on error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180976 (owner: 10Ejegg) [21:23:13] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [21:25:12] (03Merged) 10jenkins-bot: log post values on error [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180976 (owner: 10Ejegg) [21:28:12] RECOVERY - check_mysql on frdb1007 is OK: Uptime: 170180 Threads: 4 Questions: 150867856 Slow queries: 34 Opens: 3851 Open tables: 2025 Queries per second avg: 886.519 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [21:29:28] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180980 [21:31:02] (03CR) 10Damilare Adedoyin: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180980 (owner: 10Damilare Adedoyin) [21:37:18] PROBLEM - Host franio2001 is DOWN: PING CRITICAL - Packet loss = 100% [21:38:10] RECOVERY - Host franio2001 is UP: PING OK - Packet loss = 0%, RTA = 30.36 ms [21:42:13] (03PS4) 10Wfan: do not force fiscal number if not ar br for dlocal [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1180915 (https://phabricator.wikimedia.org/T401428) [21:50:03] man, zuul takes the longest when you're most anxious to see it done [21:50:14] yep, that last merge is taking a bit [21:50:15] damilare: I can take it from here - it's so late in your time zone! [21:50:28] thanks! [21:50:44] (03CR) 10Ejegg: [V:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180980 (owner: 10Damilare Adedoyin) [21:52:11] !log payments-wiki upgraded from cb76e2b7 to 207b4d6a [21:52:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:53:19] oho! [21:53:26] "action":"employerSearch" [21:53:35] "action":"di_recurring_convert" [21:54:17] so let's see, can we avoid the whole validation there? [21:54:38] employerSearch shouldn't need any gateway at all [21:54:42] should it? [21:55:03] it shouldnt [21:55:56] ah damn, we're instantiating a gateway just for the log prefix! [21:55:58] boooo [21:57:24] I'm going to make that not a thign [21:57:43] it makes sense until theres no gateway [21:57:43] we'rea already getting the session info [22:02:59] ok, employer search was easy enough to change [22:03:06] but recurring conversion might be trickier [22:04:33] oh maybe in that case it's just a matter of avoiding the check when there's no payment method [22:08:33] (03PS1) 10Ejegg: Employer Search API: don't instantiate gateway [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180988 [22:08:48] cstone: ^^^ should fix most of those [22:12:23] is it supposed to be logging on search? or when does that log [22:12:34] really just to log errors [22:12:43] ok thought so [22:12:55] you know, we could move initLogger into the setError function [22:12:57] (03CR) 10Cstone: [C:03+2] Employer Search API: don't instantiate gateway [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180988 (owner: 10Ejegg) [22:13:10] cool cool, thanks! [22:13:39] (03PS1) 10Ejegg: Don't get PaymentProvider when no method [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180989 [22:13:55] ok, and that should clear up the ones for di_recurring_convert [22:14:24] (they all had the payment-provider//class error string, indicating blank method) [22:15:11] (03Merged) 10jenkins-bot: Employer Search API: don't instantiate gateway [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180988 (owner: 10Ejegg) [22:19:15] (03CR) 10Cstone: [C:03+2] Don't get PaymentProvider when no method [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180989 (owner: 10Ejegg) [22:19:37] thanks again cstone [22:19:47] np thanks for doing all the investigating [22:21:18] (03Merged) 10jenkins-bot: Don't get PaymentProvider when no method [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1180989 (owner: 10Ejegg) [22:31:12] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180995 [22:32:34] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180995 (owner: 10Ejegg) [22:32:46] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1180995 (owner: 10Ejegg) [22:33:33] !log payments-wiki upgraded from 207b4d6a to 37616266 [22:33:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:37:55] arr [22:37:58] still a few more [22:38:06] action: logPaymentsFormError [22:40:32] hmmmm all we do with the adapter there is get the session data - but we could just read that directly from session, I think? [22:47:23] (03PS4) 10Wfan: WIP email verification [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1178912 (https://phabricator.wikimedia.org/T396523) [22:59:23] oh hah, I think the logRecurUpgradeFormError is discarding everything w/o logging: https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/includes/Api/ApiClientErrorBase.php$19 [23:00:53] ohh we do use the donor key there! [23:53:02] 03Fundraising Sprint: Pirate's Booty, 06Fundraising-Backlog, 07fr-current-sprint, 10MW-1.45-notes (1.45.0-wmf.13; 2025-08-05), 13Patch-For-Review: Do not show email update for donor with recurring active paypal - https://phabricator.wikimedia.org/T401006#11109261 (10AnnWF)