[00:01:11] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 2687497 Threads: 10 Questions: 155577417 Slow queries: 3163983 Opens: 7109 Open tables: 1817 Queries per second avg: 57.889 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [00:33:28] 10fundraising-tech-ops: Deactivate fundraising accounts for khaggard - https://phabricator.wikimedia.org/T372843 (10Dwisehaupt) 03NEW [00:34:49] 10fundraising-tech-ops: Deactivate fundraising accounts for khaggard - https://phabricator.wikimedia.org/T372843#10076017 (10Dwisehaupt) [00:43:01] 10fundraising-tech-ops: Deactivate fundraising accounts for khaggard - https://phabricator.wikimedia.org/T372843#10076021 (10Dwisehaupt) [00:45:28] 10fundraising-tech-ops: Deactivate fundraising accounts for khaggard - https://phabricator.wikimedia.org/T372843#10076022 (10Dwisehaupt) [01:07:24] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2425Q1 maintenance window - https://phabricator.wikimedia.org/T337582#10076075 (10Dwisehaupt) [09:37:17] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10076710 (10zeljkofilipin) [09:37:39] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10076712 (10zeljkofilipin) The job is passing again, can this task be resolved? [09:46:24] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10076784 (10zeljkofilipin) 05In progress→03Open a:05zeljkofilipin→03None [12:52:00] (03PS13) 10Damilare Adedoyin: Handle Report execution webhook message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059932 (https://phabricator.wikimedia.org/T370797) [12:52:49] (03CR) 10CI reject: [V:04-1] Handle Report execution webhook message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059932 (https://phabricator.wikimedia.org/T370797) (owner: 10Damilare Adedoyin) [12:56:12] (03PS1) 10Damilare Adedoyin: Implement DownloadReportJob execute function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1064020 [12:57:14] (03CR) 10CI reject: [V:04-1] Implement DownloadReportJob execute function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1064020 (owner: 10Damilare Adedoyin) [12:57:57] (03PS14) 10Damilare Adedoyin: Handle Report execution webhook message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059932 (https://phabricator.wikimedia.org/T370797) [12:59:01] (03PS2) 10Damilare Adedoyin: Implement DownloadReportJob execute function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1064020 [12:59:58] (03CR) 10CI reject: [V:04-1] Implement DownloadReportJob execute function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1064020 (owner: 10Damilare Adedoyin) [13:00:34] (03PS3) 10Damilare Adedoyin: Implement DownloadReportJob execute function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1064020 [13:01:45] (03PS4) 10Damilare Adedoyin: Implement DownloadReportJob execute function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1064020 [13:04:31] (03PS11) 10Damilare Adedoyin: Setup 3DS Flow [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1060129 (https://phabricator.wikimedia.org/T367786) [13:41:24] hi fr-tech! [13:44:09] (03CR) 10Damilare Adedoyin: [C:04-1] "Thanks for working on this and the test jgleeson, tried testing with a sandbox export and got an error which I explained inline." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1062119 (owner: 10Jgleeson) [13:47:34] howdy ejegg [13:47:45] thanks for the review damilare. will check it out now [13:47:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10077539 (10Ejegg) 05Open→03Resolved a:03Ejegg Hah! Nice catch, @Pcoombe. [13:55:07] np [14:00:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10077632 (10krobinson) Thanks @EMartin - I wonder if that is the same issue as ours seems to impact more than just those banks, and began... [14:06:30] (03PS5) 10Jgleeson: Add Gravy Audit parser [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1062119 [14:08:38] (03CR) 10Jgleeson: Add Gravy Audit parser (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1062119 (owner: 10Jgleeson) [14:14:35] (03CR) 10Ejegg: "Thanks for the tip - this is definitely easier to understand looking at the individual patches. Test coverage looks good, smoke tests OK " [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1058727 (https://phabricator.wikimedia.org/T363965) (owner: 10Eileen) [14:18:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10077726 (10Ejegg) >>! In T372571#10075275, @EMartin wrote: > @AMJohnson Adyen is saying they cannot troubleshoot and think the matter is... [14:22:43] (03PS1) 10Ejegg: Comment out upgrade fn run on prod & breaking dev [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1064040 [14:23:07] i'mma just self-merge that one ^^ [14:23:19] (03CR) 10Ejegg: [C:03+2] Comment out upgrade fn run on prod & breaking dev [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1064040 (owner: 10Ejegg) [14:34:14] pcoombe: I wonder if we can add a dns-prefetch link on donatewiki so that browsers look up payments-wiki dns early? [14:42:40] i'll make a phab for that [14:43:09] (03Merged) 10jenkins-bot: Comment out upgrade fn run on prod & breaking dev [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1064040 (owner: 10Ejegg) [14:44:06] ah maybe in special:LandingPage [14:47:04] 06Fundraising-Backlog: Prefetch payments DNS on donatewiki forms - https://phabricator.wikimedia.org/T372903 (10Ejegg) 03NEW [14:48:09] (03CR) 10Damilare Adedoyin: [C:03+2] "Done" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1062119 (owner: 10Jgleeson) [14:52:51] thanks damilare ! [14:53:18] np [14:55:13] (03PS1) 10Ejegg: Prefetch payments DNS on donatewiki form [extensions/FundraiserLandingPage] - 10https://gerrit.wikimedia.org/r/1064049 (https://phabricator.wikimedia.org/T372903) [14:56:05] fr-tech ^^^ might also help NL donors, if the problem is in the handoff from donatewiki to payments-wiki [14:56:37] it should generally speed up that handoff for all users coming in through donatewiki [15:01:31] (03Merged) 10jenkins-bot: Add Gravy Audit parser [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1062119 (owner: 10Jgleeson) [15:02:11] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10077954 (10Ejegg) >>! In T372571#10077632, @krobinson wrote: > It feels like this is happening in the handoff from donate.wiki to paymen... [15:02:28] (03PS15) 10Damilare Adedoyin: Handle Report execution webhook message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059932 (https://phabricator.wikimedia.org/T370797) [15:02:41] (03PS5) 10Damilare Adedoyin: Implement DownloadReportJob execute function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1064020 [15:03:24] ahhh silly me i didn't see the screenshots [15:04:30] so if they can't load paymentswiki at all, we need to check on the deliverability of IPv4-only payments-wiki in NL [15:06:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10077970 (10krobinson) @Ejegg , @AMJohnson is cleaning up the error triage sheet and will send you a new tab shortly with just the Dutch... [15:08:26] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10077982 (10Ejegg) So the big difference with payments-wiki is that we only serve it over IPv4, not IPv6 - we might need to revisit that... [15:56:15] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2425Q1 maintenance window - https://phabricator.wikimedia.org/T337582#10078220 (10Dwisehaupt) [16:56:23] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10078577 (10AMJohnson) As @krobinson noted we've separated out NL tickets and they are on their own tab in Error Triage here: https://doc... [18:16:21] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Imports to CiviCRM - make it possible to specify the default for a field without having to have a column - https://phabricator.wikimedia.org/T342717#10078879 (10MDemosWMF) @Eil... [18:28:43] (03CR) 10Cstone: [C:03+2] "this looks good, thanks!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063191 (https://phabricator.wikimedia.org/T372559) (owner: 10Damilare Adedoyin) [18:29:11] thanks cstone [18:29:12] (03Merged) 10jenkins-bot: Ensure Gravy recurring card payments can be charged via Civi SmashPig recurring charge job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063191 (https://phabricator.wikimedia.org/T372559) (owner: 10Damilare Adedoyin) [18:29:45] i also tried charging more than the token damilare and that worked too (although not part of the ticket hah but if we ever do the recurring upgrade) [18:30:05] awesome! [18:46:58] (03PS1) 10Damilare Adedoyin: Update smashpig to v0.8.11.2 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1064068 [18:54:28] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10079012 (10Cstone) @AMJohnson has any of these donors gotten back to you that they were successfully able to donate if they tried again/... [18:59:07] (03PS12) 10Damilare Adedoyin: Gravy ACH (Trustly) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063208 (https://phabricator.wikimedia.org/T371810) [18:59:21] (03PS12) 10Damilare Adedoyin: Setup 3DS Flow [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1060129 (https://phabricator.wikimedia.org/T367786) [19:00:29] 06Fundraising-Backlog: Batch data entry feature request: error notification + show completed batch - https://phabricator.wikimedia.org/T371540#10079019 (10MDemosWMF) After talking with @Eileenmcnaughton it may not be doable to have an output file available like the import mapping does, but we would still like to... [19:00:40] (03PS38) 10Damilare Adedoyin: Setup listener for transaction event message. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1056531 (https://phabricator.wikimedia.org/T370797) [19:00:40] (03PS19) 10Damilare Adedoyin: ProcessCaptureRequestJob for authorization messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1057937 (https://phabricator.wikimedia.org/T370797) [19:00:41] (03PS13) 10Damilare Adedoyin: Record capture job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1058580 (https://phabricator.wikimedia.org/T370797) [19:00:42] (03PS15) 10Damilare Adedoyin: Setup Refunds method and maintenance script [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059078 (https://phabricator.wikimedia.org/T367808) [19:00:43] (03PS10) 10Damilare Adedoyin: Refund job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059356 (https://phabricator.wikimedia.org/T370797) [19:00:44] (03PS16) 10Damilare Adedoyin: Handle Report execution webhook message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059932 (https://phabricator.wikimedia.org/T370797) [19:00:56] (03PS6) 10Damilare Adedoyin: Implement DownloadReportJob execute function [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1064020 [19:09:03] (03PS1) 10Umherirrender: Add missing documentation to class properties [extensions/LandingCheck] - 10https://gerrit.wikimedia.org/r/1064072 [19:12:29] (03CR) 10Damilare Adedoyin: [C:03+2] Update smashpig to v0.8.11.2 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1064068 (owner: 10Damilare Adedoyin) [19:27:17] 10fundraising-tech-ops: Install and configure new host frlog2002 - https://phabricator.wikimedia.org/T372933#10079087 (10Dwisehaupt) [19:32:48] (03Merged) 10jenkins-bot: Update smashpig to v0.8.11.2 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1064068 (owner: 10Damilare Adedoyin) [19:39:18] (03PS1) 10Damilare Adedoyin: Update smashpig to v0.8.11.2 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1064082 [19:40:12] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1064083 [19:45:15] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Paypal: Really fix PayPal fluctuating subscr_id problem - https://phabricator.wikimedia.org/T371845#10079128 (10Ejegg) Looks like there are just 2 subscriptions getting these issues - maybe just update the data to... [20:13:34] (03CR) 10Damilare Adedoyin: [C:03+2] Update smashpig to v0.8.11.2 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1064082 (owner: 10Damilare Adedoyin) [20:18:20] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Imports to CiviCRM - make it possible to specify the default for a field without having to have a column - https://phabricator.wikimedia.org/T342717#10079234 (10Eileenmcnaug... [20:18:33] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Imports to CiviCRM - make it possible to specify the default for a field without having to have a column - https://phabricator.wikimedia.org/T342717#10079236 (10Eileenmcnaug... [20:33:24] (03Merged) 10jenkins-bot: Update smashpig to v0.8.11.2 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1064082 (owner: 10Damilare Adedoyin) [20:39:33] (03PS13) 10Damilare Adedoyin: Setup 3DS flow [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) [20:39:45] (03CR) 10Damilare Adedoyin: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1064083 (owner: 10Damilare Adedoyin) [20:40:34] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1064083 (owner: 10Damilare Adedoyin) [22:37:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10079610 (10EMartin) Here is the latest update from Adyen on this matter: Jake Leininger 1:16 PM (2 hours ago) to me, Adyen, rkumar@wik... [23:07:51] 10fundraising-tech-ops: Install and configure new host frlog2002 - https://phabricator.wikimedia.org/T372933#10079776 (10Dwisehaupt) [23:09:01] 10fundraising-tech-ops: Install and configure new host frlog2002 - https://phabricator.wikimedia.org/T372933#10079777 (10Dwisehaupt) Backing up and recloning the data from frdata2001 since there was a misconfiguration of the /srv/ partition that had it way short on disk space. I grew the VG and LV and rebuilt th... [23:15:20] (03PS1) 10Eileen: Add last_name,first_name index [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1064130 (https://phabricator.wikimedia.org/T353971) [23:15:56] dwisehaupt: so && is the other thing I want to do during maintenance - I'll need someone to plus 2 but it creates a combined index on last_name, first_name [23:16:17] I think it maybe takes 10 mins from memory when I did it on staging [23:26:33] that looks good to me, but i'm probably not a good one to judge php code. :) [23:46:00] ejegg: or cstone could one of you look at ^^ [23:55:08] just sent $kid to the store to get rice vinegar so i'm clear for a while.