[00:33:29] (03PS4) 10Eileen: Remove slow check from the upgrade [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769106 (https://phabricator.wikimedia.org/T303363) [00:33:31] (03PS7) 10Eileen: Re-add settings_location.php [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/763346 (https://phabricator.wikimedia.org/T303363) [00:33:33] (03PS5) 10Eileen: Re-add WMF code to support our templates [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/763403 (https://phabricator.wikimedia.org/T303363) [00:33:35] (03PS3) 10Eileen: Add created & modified dates to relationhsip view [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769145 (https://phabricator.wikimedia.org/T294487) [00:33:37] (03PS3) 10Eileen: SearchKit - Allow searching by range [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769531 [00:33:39] (03PS3) 10Eileen: Afform - Refactor toward multiple search displays on an afform [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769532 [00:33:41] (03PS1) 10Eileen: Just catch up [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769796 [00:33:49] ejegg|afk: I just added a patch to just catch the exception - seems like it doesn't hurt to skip that [00:45:39] (03CR) 10jerkins-bot: [V: 04-1] Re-add settings_location.php [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/763346 (https://phabricator.wikimedia.org/T303363) (owner: 10Eileen) [00:46:46] looking eileen [00:49:25] (03CR) 10jerkins-bot: [V: 04-1] Remove slow check from the upgrade [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769106 (https://phabricator.wikimedia.org/T303363) (owner: 10Eileen) [00:50:02] (03CR) 10jerkins-bot: [V: 04-1] Just catch up [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769796 (owner: 10Eileen) [00:53:06] ejegg: I also logged it https://lab.civicrm.org/dev/core/-/issues/3112 [00:53:32] oh, did you manage to replicate locally? [00:53:52] I was wondering if there was some weird metadata cache I had that survived the rebuild or something [00:54:30] just doing another rebuild to get out of the partially upgraded state, and I'll apply your latest patch [00:56:03] ejegg: yeah I replicated it [00:56:42] eileen: we still on for the upgrade? [00:56:56] i have a diff ready to pause the p-c jobs when we are ready. [00:57:01] dwisehaupt: I think so - just need ejegg to check if the latest patch is OK for him [00:57:07] cool cool. [00:57:15] not rushing at all. just checking. [01:06:37] ok, got through the upgrade this time! [01:09:21] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Stock civicrm 5.48-alpha [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/762972 (https://phabricator.wikimedia.org/T303363) (owner: 10Eileen) [01:09:31] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Just catch up [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769796 (owner: 10Eileen) [01:09:40] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Remove slow check from the upgrade [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769106 (https://phabricator.wikimedia.org/T303363) (owner: 10Eileen) [01:09:47] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Re-add settings_location.php [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/763346 (https://phabricator.wikimedia.org/T303363) (owner: 10Eileen) [01:09:57] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Re-add WMF code to support our templates [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/763403 (https://phabricator.wikimedia.org/T303363) (owner: 10Eileen) [01:10:12] yay [01:11:34] dwisehaupt: I'll start merging through - if you want to turn jobs off [01:11:48] ok. i'll turn them off [01:12:12] (03CR) 10Ejegg: [C: 03+2] "Looks good" [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769145 (https://phabricator.wikimedia.org/T294487) (owner: 10Eileen) [01:13:01] (03PS1) 10Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769806 [01:13:16] (03CR) 10Eileen: [C: 03+2] CiviCRM submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769806 (owner: 10Eileen) [01:13:24] eileen: that Queue patch is WIP - did you need that merged today too? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm/+/768849 [01:13:31] ejegg: nah it's fine [01:13:34] k [01:14:08] if we want to include the search kit ones David call play with them - or I can just put those on staging [01:18:55] dang I forgot how slow the merge is [01:20:17] ok. after doing the process in the right order the jobs on civi1001 are disabled. :) [01:23:18] :-) [01:23:32] cool - just waiting for https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/769806 [01:24:41] 10Fundraising Sprint Discworld reformatted as ntfs, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Request to make $1k donation bot for END gifts only - https://phabricator.wikimedia.org/T301982 (10Ejegg) @RLewis OK, it should be possible to create an Endowment-only large donation notification now.... [01:28:49] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/769812 [01:29:00] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/769812 (owner: 10Eileen) [01:29:16] ok that is getting there! [01:30:16] dwisehaupt: ok - ready to roll [01:30:27] jobs are already off so I guess I can push out [01:31:12] yeah. go for it. [01:31:44] !log civicrm changed from 4cb2bdbc to 8501c38c [01:31:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:33:20] upgrade ran ok - I'm gonna regenerate the triggers now [01:33:52] sounds good. just let me know when i can pull them to apply them. [01:38:05] (03PS1) 10Eileen: Update CiviCRM triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769814 (https://phabricator.wikimedia.org/T303363) [01:40:09] ejegg: or dwisehaupt this is the triggers update for +2 https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/769814 [01:41:01] looking [01:42:40] (03CR) 10Dwisehaupt: [C: 03+2] "Looks good." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769814 (https://phabricator.wikimedia.org/T303363) (owner: 10Eileen) [01:44:02] looks like the verification run is rolling. [01:45:05] oh hey, new civicrm_oauth tables? [01:45:52] ejegg: yeah - not sure if that means the extension is now always enabled [01:46:26] I can't see the change to the prospect table which is odd [01:46:51] hmm looks like it was already there [01:52:51] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/769815 [01:52:59] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/769815 (owner: 10Eileen) [01:56:12] !log civicrm revision changed from 8501c38c to 9fb68b24 [01:56:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:56:38] dwisehaupt: the triggers file is on live now - drupal/sites/all/modules/wmf_civicrm/scripts/triggers.mysql [01:58:26] coolthanks. rolling them. [02:01:36] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769145 (https://phabricator.wikimedia.org/T294487) (owner: 10Eileen) [02:07:09] ok. triggers look good. [02:07:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [02:07:21] reviewing the diff of them on the cli is not fun. [02:07:32] ^^^ is known during the update. i'll ack it. [02:08:07] ACKNOWLEDGEMENT - check_mysql on frdb1002 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) Dwisehaupt trigger updates https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [02:08:31] eileen: should be all clear to test/verify [02:08:57] dwisehaupt: cool - I'll do a slow start [02:10:02] dwisehaupt: yep - looks fine [02:11:09] cool. i'll start the replication and start putting everything back in place. [02:12:17] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 1226583 Threads: 11 Questions: 50463959 Slow queries: 612 Opens: 152614091 Flush tables: 1 Open tables: 200 Queries per second avg: 41.141 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [02:17:45] i wish there was an rsync_blaster option for just localsettings. [02:18:11] ok. we should be back with jobs in place. [02:20:28] cool [02:23:40] (03Merged) 10jenkins-bot: Add created & modified dates to relationhsip view [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/769145 (https://phabricator.wikimedia.org/T294487) (owner: 10Eileen) [02:30:21] (03PS1) 10Eileen: Submodule CiviCRM update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769816 [02:30:33] (03CR) 10Eileen: [C: 03+2] Submodule CiviCRM update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769816 (owner: 10Eileen) [02:31:14] ok. i need to head out. i'll check back in a bit just in case, but have a good one. [02:36:48] (03PS9) 10Ejegg: Add google pay js [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/768752 (https://phabricator.wikimedia.org/T300686) (owner: 10Wfan) [02:37:46] (03CR) 10Ejegg: [C: 03+2] "Thanks, this looks great and works in local testing!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/768752 (https://phabricator.wikimedia.org/T300686) (owner: 10Wfan) [02:39:43] (03Merged) 10jenkins-bot: Add google pay js [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/768752 (https://phabricator.wikimedia.org/T300686) (owner: 10Wfan) [02:42:51] thanks [02:46:35] (03Merged) 10jenkins-bot: Submodule CiviCRM update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769816 (owner: 10Eileen) [02:49:00] (03PS1) 10Ejegg: Stop falling back to server-side GeoIP lookup [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/769817 (https://phabricator.wikimedia.org/T301485) [02:50:25] (03CR) 10jerkins-bot: [V: 04-1] Stop falling back to server-side GeoIP lookup [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/769817 (https://phabricator.wikimedia.org/T301485) (owner: 10Ejegg) [02:50:41] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/769818 [02:50:53] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/769818 (owner: 10Eileen) [02:54:48] !log revision changed from 9fb68b24 to 252269c8 [02:54:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:56:50] 10Fundraising Sprint Cheese String Theory, 10Fundraising Sprint Discworld reformatted as ntfs, 10Fundraising-Backlog, 10donate.wikimedia.org, and 2 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T301485 (1... [02:57:30] (03Abandoned) 10Ejegg: Stop falling back to server-side GeoIP lookup [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/769817 (https://phabricator.wikimedia.org/T301485) (owner: 10Ejegg) [03:31:14] (03PS1) 10Ejegg: Remove drupal OAuth and dependencies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769821 (https://phabricator.wikimedia.org/T298973) [03:32:25] 10Fundraising Sprint Discworld reformatted as ntfs, 10Fundraising-Backlog, 10Fr-drupal-upgrade-2021, 10Patch-For-Review: disable civicrm oauth extension and remove module from crm/drupal repository - https://phabricator.wikimedia.org/T298973 (10Ejegg) [03:45:26] (03PS1) 10Ejegg: WIP Ping Braintree GraphQL API [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/769822 (https://phabricator.wikimedia.org/T302946) [03:45:28] (03PS1) 10Ejegg: WIP createClientToken for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/769823 (https://phabricator.wikimedia.org/T302948) [03:45:52] (03CR) 10jerkins-bot: [V: 04-1] WIP Ping Braintree GraphQL API [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/769822 (https://phabricator.wikimedia.org/T302946) (owner: 10Ejegg) [03:45:54] (03CR) 10jerkins-bot: [V: 04-1] WIP createClientToken for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/769823 (https://phabricator.wikimedia.org/T302948) (owner: 10Ejegg) [03:55:04] (03PS2) 10Ejegg: WIP Ping Braintree GraphQL API [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/769822 (https://phabricator.wikimedia.org/T302946) [03:55:06] (03PS2) 10Ejegg: WIP createClientToken for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/769823 (https://phabricator.wikimedia.org/T302948) [07:41:38] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/769868 (owner: 10L10n-bot) [08:16:43] (03CR) 10Raimond Spekking: [C: 03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/769868 (owner: 10L10n-bot) [13:32:44] 10Fundraising Sprint Discworld reformatted as ntfs, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Request to make $1k donation bot for END gifts only - https://phabricator.wikimedia.org/T301982 (10RLewis) @Ejegg thanks, I think it would work but I'm having trouble trying to the Endowment financial... [15:57:21] wfan: cstone could you add me and damilare to the google pay test group pls [15:57:51] I clicked "join" but it still tells me "You don't have permission to access this content" [15:58:01] thanks in advance! [15:59:12] actually scrap that. we both just got access [15:59:20] unless you did it! :) [16:00:13] There is a link you can add yourself, let me find the link:) [16:01:33] https://groups.google.com/forum/#!forum/googlepay-test-mode-stub-data [16:23:19] 10Fundraising-Backlog, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, and 3 others: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned - https://phabricator.wikimedia.org/T282131 (10phuedx) [16:23:45] hey wfan thanks we got access :) [16:24:13] quick question, is there a Google Pay civicrm patch to add GPay as a payment method? [16:32:29] Not yet as I know now [16:35:57] ah cool no worries. we tried importing the donation into civicrm and got an error around missing a payment method [16:37:26] wfan: do you want us to add a ticket for that? [16:37:33] lemme see if there already is one [16:37:38] I think we should? Haha thanks! [16:41:42] 10Fundraising-Backlog, 10FR-Adyen, 10Epic: Add GPay to CivICRM - https://phabricator.wikimedia.org/T303623 (10jgleeson) [16:42:52] 10Fundraising-Backlog, 10FR-Adyen, 10Epic: Add GPay to Adyen Audit Processor list - https://phabricator.wikimedia.org/T303624 (10jgleeson) [16:43:20] wfan: damilare and I just added couple of tickets to the epic copying what we did for Apple Pay [16:43:52] btw we testing your patches both in donation interface and smash-pig and they worked great for us! [16:44:13] Haha sounds good and thanks! 😀 [16:45:39] wfan: cstone did you find any test cards that return a failure result? [16:45:50] to see how it behaves in that scenario [16:46:59] I think adyen test all the google pay test card with their same card 4111, so not yet touched that error card, but we can create one according to their email about 3ds [16:47:49] ah ok [16:47:52] thanks [16:51:38] looks like we have had some sporadic timeouts with adyen overnight. [16:52:00] yeah a bunch dwisehaupt [16:52:36] looks like they started around 5am UTC [16:52:55] are they all apple pay? [16:53:33] the few i spot checked were. i'll look more in a sec. [16:53:51] still doing the morning mail cleanup. [16:53:54] yeah I think they are. I wonder if apple pay was having issues behind the scenes [16:56:41] most were. of the 32 since last night 27 are apple pay. [16:57:37] 1 timed out that wasn't apple pay and 4 failed with Payment details [16:57:40] are not supported [17:01:35] (03CR) 10AndyRussG: Support full_name for all messages, not just ChecksFile (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/769098 (https://phabricator.wikimedia.org/T300690) (owner: 10Ejegg) [17:31:48] /topic Errors or problems donating? Email problemsdonating@wikimedia.org. Feedback welcome on our meta talk page: http://bit.ly/1y9WUuL. Channel logs: https://wm-bot.wmflabs.org/libera_logs/%23wikimedia-fundraising/ [17:32:13] woohoo! [17:32:37] sweet. [17:32:38] thanks. [19:05:50] so i've dug in a tiny bit on the logs for the applepay adyen failures. just looking at payments1007 i can see a mix of successes and failures. [19:06:43] ipset and fw rules look right and match dns when i check. [19:07:18] it's possible that they are either having issues themselves or are popping hosts in and out of rotation with dns more frequently than we can update to get them. [19:07:35] the latter seems less likely. [19:10:16] probably worth reaching out to them at some point to see if they can provide any insight. [20:21:21] dwisehaupt: it's also possible for us to increase our timeouts - 7 seconds may be a bit optimistic for a payment to fully process in some cases [20:21:44] So that would be in the smashpig/local-settings/adyen folder of the LocalSettings repo [20:22:51] under the curl: key, add a timeout: 15 [20:23:39] true. that may work also. [20:25:06] i'll just try that now [20:28:20] OK, that's updated [20:29:20] oh hmm, "4 failed with Payment details are not supported" [20:29:43] That's an error from our own code, because we thought the Apple Pay flow would never involve a redirect [20:29:56] but it seems like maybe sometimes it does [20:30:03] We should file a bug for that one [20:31:00] coolthanks. there haven't been any of those failures in over an hour, but that doesn't mean they are gone. :) [20:31:37] oh. i stand corrected. i was in the wrong mail folder. :) [20:36:25] 10Fundraising Sprint Discworld reformatted as ntfs, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Request to make $1k donation bot for END gifts only - https://phabricator.wikimedia.org/T301982 (10Ejegg) @RLewis I think I just created it successfully. I clicked on the first option and shift-clicked... [22:59:08] (03CR) 10Jforrester: [C: 03+2] Namespace extension [extensions/LandingCheck] - 10https://gerrit.wikimedia.org/r/768323 (owner: 10Reedy) [23:14:28] (03Merged) 10jenkins-bot: Namespace extension [extensions/LandingCheck] - 10https://gerrit.wikimedia.org/r/768323 (owner: 10Reedy)