[00:04:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [33.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [00:10:56] (03CR) 10Wfan: [C:03+2] Ensure low-confidence names do not overwrite existing high confidence names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324798 (https://phabricator.wikimedia.org/T434715) (owner: 10Lars SG) [00:17:11] (03PS1) 10Eileen: Fix DAFPay payment method when it is in web_location_url [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326953 (https://phabricator.wikimedia.org/T434317) [00:19:46] (03CR) 10Cstone: [C:03+2] Fix DAFPay payment method when it is in web_location_url [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326953 (https://phabricator.wikimedia.org/T434317) (owner: 10Eileen) [00:20:28] (03Merged) 10jenkins-bot: Fix DAFPay payment method when it is in web_location_url [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326953 (https://phabricator.wikimedia.org/T434317) (owner: 10Eileen) [00:22:12] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1326954 [00:22:26] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1326954 (owner: 10Eileen) [00:25:01] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1326954 (owner: 10Eileen) [00:25:43] !log SmashPig upgraded from 89909858 to 0e3e59b4 [00:25:44] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [00:27:29] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing, 13Patch-For-Review: Chariot data issue - DAFpay payment method - https://phabricator.wikimedia.org/T434317#12229249 (10Eileenmcnaughton) @MDemosWMF these should be fixed for new donations going forwards [00:29:02] (03Merged) 10jenkins-bot: Ensure low-confidence names do not overwrite existing high confidence names [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324798 (https://phabricator.wikimedia.org/T434715) (owner: 10Lars SG) [00:37:20] (03PS3) 10Dmiranda: Updated email-preference-center radio buttons to render blank on load [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326938 (https://phabricator.wikimedia.org/T434711) [00:38:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [00:51:45] (03CR) 10Lars SG: [C:03+2] "Sorry, I think I confused things - I just meant that $() was unneeded! But it works as is, looks good, sorry for the lack of clarity there" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326938 (https://phabricator.wikimedia.org/T434711) (owner: 10Dmiranda) [00:56:55] (03CR) 10Dmiranda: "No problem at all, I appreciate the feedback and thanks for taking a look!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326938 (https://phabricator.wikimedia.org/T434711) (owner: 10Dmiranda) [00:58:10] (03PS1) 10Eileen: Update composer ominmail + minor updates [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326959 [00:58:54] (03Merged) 10jenkins-bot: Updated email-preference-center radio buttons to render blank on load [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326938 (https://phabricator.wikimedia.org/T434711) (owner: 10Dmiranda) [01:02:28] (03CR) 10Lars SG: [C:03+2] Update composer ominmail + minor updates [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326959 (owner: 10Eileen) [01:06:20] (03PS1) 10Eileen: Upgrading wikimedia/omnimail-silverpop (1.35 => 1.36) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326960 [01:06:58] (03CR) 10CI reject: [V:04-1] Upgrading wikimedia/omnimail-silverpop (1.35 => 1.36) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326960 (owner: 10Eileen) [01:07:02] (03PS1) 10Lars SG: Shared US phone number normalization helper [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326961 (https://phabricator.wikimedia.org/T423967) [01:08:26] (03PS2) 10Eileen: Upgrading wikimedia/omnimail-silverpop (1.35 => 1.36) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326960 [01:08:52] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326960 (owner: 10Eileen) [01:13:45] (03PS1) 10Eileen: Fix typo in last commit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326963 (https://phabricator.wikimedia.org/T434317) [01:15:09] (03CR) 10Lars SG: [C:03+2] Upgrading wikimedia/omnimail-silverpop (1.35 => 1.36) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326960 (owner: 10Eileen) [01:24:42] (03Merged) 10jenkins-bot: Update composer ominmail + minor updates [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326959 (owner: 10Eileen) [01:34:02] (03Merged) 10jenkins-bot: Upgrading wikimedia/omnimail-silverpop (1.35 => 1.36) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326960 (owner: 10Eileen) [02:14:14] (03PS1) 10Ejegg: Always keep original tracking fields [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326965 (https://phabricator.wikimedia.org/T434734) [02:18:06] eileen: ^^^ might be all it takes [02:18:32] let me make up a test [02:18:39] smoke test at least [02:18:41] (03CR) 10Eileen: [C:03+2] Always keep original tracking fields [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326965 (https://phabricator.wikimedia.org/T434734) (owner: 10Ejegg) [02:18:55] ejegg: opps I was too keen to really test [02:19:44] heh, well, I'll just test locally before deploying [02:19:50] :-) [02:20:47] (03Merged) 10jenkins-bot: Always keep original tracking fields [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326965 (https://phabricator.wikimedia.org/T434734) (owner: 10Ejegg) [02:21:53] ejegg: also - one minor one that will prevent further failmail on the job https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/1326963 [02:24:26] (03CR) 10Ejegg: [C:03+2] Fix typo in last commit [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326963 (https://phabricator.wikimedia.org/T434317) (owner: 10Eileen) [02:29:26] (03CR) 10Eileen: [V:03+2] "just putting the v2 back rather than pointless wait to deploy" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326963 (https://phabricator.wikimedia.org/T434317) (owner: 10Eileen) [02:30:25] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1326966 [02:30:37] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1326966 (owner: 10Eileen) [02:33:42] !log SmashPig upgraded from 0e3e59b4 to 4e9d9df0 [02:33:43] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [02:42:03] that wasn't quite the fix eileen, unfortunately, because of where we handle the utm_ / wmf_ switcharoo [02:42:19] I'm going to move that logic up to where we read it in from the querystring [02:42:21] oops - I got waaay too excited [02:42:55] I'd like to log whether we're still getting any old-style utm_ params on the qs, actually [03:01:28] (03PS1) 10Ejegg: Do wmf_ -> utm_ conversion right in harvest loop [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326969 (https://phabricator.wikimedia.org/T434734) [03:01:35] well, ^^^ will do it [03:03:02] (03CR) 10CI reject: [V:04-1] Do wmf_ -> utm_ conversion right in harvest loop [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326969 (https://phabricator.wikimedia.org/T434734) (owner: 10Ejegg) [03:03:42] oh what's that test doing? [03:06:30] bah, the ersatz ..cc value from the first go round is sticking [03:06:47] that's one value we should allow to be overwritten [03:07:09] i mean, we should just stop concatenating the payment method into the utm_source ideally [03:07:12] what a mess [03:11:24] well, we're also using a test-only override there. I'll just mock the whole request in that test [03:22:51] (03PS1) 10Eileen: Fix needs_attention batches getting stuck instead of re-verifying [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326970 [03:23:37] ^^ minor update to make managing problem batches a little easier - means that if they are fixed up they work [03:23:59] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07fr-combowiki: Limit number of fields ComboWiki reads from URL - https://phabricator.wikimedia.org/T435273 (10Ejegg) 03NEW [03:29:39] (03PS2) 10Ejegg: Do wmf_ -> utm_ conversion right in harvest loop [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326969 (https://phabricator.wikimedia.org/T434734) [03:36:41] eileen: can you explain how a batch that was successfully verified at one point could end up in needs_attention status? [03:39:39] total_verified means the batch total is verified against an external data point (e.g the header of the csv file), validated means the contributions in the batch adds up to that amount - needs_attention gets triggered when the transition from total_verfied to validated goes wrong - either for material or timing reasons (in the latter case because the settle queue is still running) [03:40:29] ejegg: do I need to merge your follow up given I merged the wrong one? [03:40:57] the initial one is basically a no-op without the follow-up [03:41:04] ok cool [03:45:31] hmm, i wonder if there's anything more we can do to avoid those timing snafus [03:45:43] like set a batch to 'settling' while the settle queue runs? [03:46:12] but the settle queue consumer has no idea when to set it back [03:46:32] and might be processing stuff from a bunch of different batches [03:48:45] ooh, php-cache/integration-tests finally supports psr/simple-cache 2.x and 3.x https://github.com/php-cache/integration-tests [03:50:06] eileen: do we treat wmf-civicrm as required for all our other extensions? It isn't literally, but do we assume it is? Context is I want to add a helper that will be used in wmf-civicrm and omnimail, so I'd add it to wmf-civicrm... [03:52:06] we can put tests back in addshore's psr cache wrapper! [03:52:33] larssandergreen: we've tried to make the dependency go the other way when possible [03:52:54] so on the off chance someone else can use omnimail they can install it without all our other stuff [03:53:13] ok, fair enough, I'll do it that way even if it looks a little weird [03:53:44] yeah, i ended up with a bit of message logic in the SMashPig extension for that very reason [03:56:16] bah, though I realize the whole extension is full of 'wmf' log channel references [03:56:25] and a few tests use wmf helpers :( [03:56:47] oh dang, I have to get to bed. [03:57:14] right, but I think I understand enough of that needs_attention patch to review [03:59:31] yeah larssandergreen omnimail is becoming increasingly not something that anyone else could use.... [04:00:12] so if it's too ugly it's ok to assume I think [04:00:31] it's bearable, just thinking in general [04:00:58] ejegg: re setting back to settling - there is a phab to basically not set to needs_attention if the queue is not empty... - I just don't know how to check that .... [04:02:46] (03CR) 10Ejegg: "Looks good, might deduplicate a tiny bit of code" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326970 (owner: 10Eileen) [04:03:32] oh hah, there is a peek() function that we probably should put on that interface [04:04:30] (03PS1) 10Lars SG: Add SMS consent source custom field and set it on Acoustic-sourced activities [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326971 (https://phabricator.wikimedia.org/T423967) [04:04:43] it is implemented by the Predis backend but not PDO [04:04:49] so tests would be unhappy [04:05:10] php-queue is basically another captive dependency so we can just add taht [04:07:31] right I really need to sleep [04:08:24] does that deduplication make sense eileen ? I can just C+2 as is if it seems odd to build up the API request that way [04:10:15] (03PS2) 10Eileen: Fix needs_attention batches getting stuck instead of re-verifying [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326970 [04:10:26] ejegg: I fixed it I think [04:11:43] oh nice, shared the select even [04:12:03] (03CR) 10Ejegg: [C:03+2] Fix needs_attention batches getting stuck instead of re-verifying [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326970 (owner: 10Eileen) [04:17:58] (03PS2) 10Lars SG: Add SMS consent source custom field and set it on Acoustic-sourced activities [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326971 (https://phabricator.wikimedia.org/T423967) [04:22:00] (03PS1) 10Lars SG: Add OptIn API Action to opt in phone numbers at Acoustic [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326972 (https://phabricator.wikimedia.org/T423967) [04:23:13] (03PS5) 10Lars SG: Shared US phone number normalization helper [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326961 (https://phabricator.wikimedia.org/T423967) [04:27:06] (03PS1) 10Eileen: Trustly: R03/unhandled-R-code Return leg is pure Trustly, not gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326973 (https://phabricator.wikimedia.org/T433921) [04:33:04] (03Merged) 10jenkins-bot: Fix needs_attention batches getting stuck instead of re-verifying [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326970 (owner: 10Eileen) [04:38:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [04:53:41] (03CR) 10Eileen: [C:03+2] Add SMS consent source custom field and set it on Acoustic-sourced activities [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326971 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [04:58:39] (03CR) 10Lars SG: [C:03+2] Trustly: R03/unhandled-R-code Return leg is pure Trustly, not gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326973 (https://phabricator.wikimedia.org/T433921) (owner: 10Eileen) [04:59:16] (03Merged) 10jenkins-bot: Trustly: R03/unhandled-R-code Return leg is pure Trustly, not gravy [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326973 (https://phabricator.wikimedia.org/T433921) (owner: 10Eileen) [05:06:16] (03PS1) 10Eileen: Upgrading wikimedia/smash-pig (v1.2.5.17 => v1.2.5.18) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326975 [05:09:58] (03PS1) 10Eileen: Update smashpig, silverpop [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1326976 [05:10:07] (03CR) 10Eileen: [C:03+2] Update smashpig, silverpop [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1326976 (owner: 10Eileen) [05:10:34] (03CR) 10Eileen: [C:03+2] Shared US phone number normalization helper [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326961 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [05:12:04] (03CR) 10Eileen: "Looks pretty sensible - this isn't called yet so this just adds the api - which would probably be further tested before being added to rea" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326972 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [05:12:16] (03CR) 10Eileen: [C:03+2] Add OptIn API Action to opt in phone numbers at Acoustic [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326972 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [05:12:41] (03CR) 10Eileen: [C:03+2] Upgrading wikimedia/smash-pig (v1.2.5.17 => v1.2.5.18) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326975 (owner: 10Eileen) [05:13:22] (03CR) 10Eileen: [C:03+2] Update is_eligible_for_donor_portal to new audience framework [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1324351 (https://phabricator.wikimedia.org/T430369) (owner: 10Lars SG) [05:13:46] (03CR) 10Eileen: [C:03+2] Update getDonorPortalUrl to new audience framework [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324352 (https://phabricator.wikimedia.org/T430369) (owner: 10Lars SG) [05:26:03] (03PS3) 10Lars SG: SMS opt-out contacts from Acoustic when modified there [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1325564 (https://phabricator.wikimedia.org/T423967) [05:28:40] (03PS1) 10Lars SG: Follow up: fix SMS opt in activity subject [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326978 (https://phabricator.wikimedia.org/T423967) [05:32:27] (03Merged) 10jenkins-bot: Update smashpig, silverpop [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1326976 (owner: 10Eileen) [05:32:28] (03Merged) 10jenkins-bot: Shared US phone number normalization helper [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326961 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [05:32:31] (03Merged) 10jenkins-bot: Add SMS consent source custom field and set it on Acoustic-sourced activities [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326971 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [05:34:54] (03Merged) 10jenkins-bot: Add OptIn API Action to opt in phone numbers at Acoustic [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326972 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [05:36:00] (03Merged) 10jenkins-bot: Upgrading wikimedia/smash-pig (v1.2.5.17 => v1.2.5.18) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326975 (owner: 10Eileen) [05:36:31] (03Merged) 10jenkins-bot: Update getDonorPortalUrl to new audience framework [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324352 (https://phabricator.wikimedia.org/T430369) (owner: 10Lars SG) [05:47:01] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1326980 [05:49:14] (03CR) 10CI reject: [V:04-1] SMS opt-out contacts from Acoustic when modified there [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1325564 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [05:49:50] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1326980 (owner: 10Eileen) [05:52:32] !log civicrm upgraded from 9049b363 to 4f83dccd [05:52:34] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [05:53:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [150.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [06:04:33] (03PS2) 10Lars SG: Follow up: fix SMS opt in activity subject [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326978 (https://phabricator.wikimedia.org/T423967) [06:04:53] (03CR) 10Eileen: [C:03+2] Follow up: fix SMS opt in activity subject [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326978 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [06:06:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [105.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [06:23:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [06:27:45] (03Merged) 10jenkins-bot: Follow up: fix SMS opt in activity subject [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1326978 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [07:05:05] (03PS1) 10Eileen: Add Base62 isValid helper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1326997 (https://phabricator.wikimedia.org/T433921) [07:11:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [60.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [07:43:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [08:26:11] FIRING: [26x] NodeDown: Node civi2002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [08:26:47] FIRING: HAProxyExporterDown: HAProxy on fransw2003.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [08:28:11] FIRING: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-eqiad and (null) (10.195.0.249) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [08:28:51] FIRING: [4x] SwitchCoreInterfaceDown: Switch core interface down - fasw1-f5a-codfw:et-0/0/47 (Core: pfw1-codfw:et-0/1/0) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [08:30:47] FIRING: HAProxyExporterDown: HAProxy on fransw2003.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [08:31:11] FIRING: [4x] NodeDown: Node fransw2003 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [08:31:11] FIRING: [26x] NodeDown: Node civi2002 has been down for long. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [08:32:57] FIRING: MariadbNoReplicas: No MariaDB replicas connected to : frdb1007.frack.eqiad.wmnet:9004 - https://alerts.wikimedia.org/?q=alertname%3DMariadbNoReplicas [08:35:47] FIRING: [6x] HAProxyExporterDown: HAProxy on fransc2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [08:36:10] FIRING: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:36:11] RESOLVED: [30x] NodeDown: Node civi2002 has been down for long. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [08:36:11] RESOLVED: [30x] NodeDown: Node civi2002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [08:36:47] RESOLVED: [6x] HAProxyExporterDown: HAProxy on fransc2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [08:37:57] RESOLVED: MariadbNoReplicas: No MariaDB replicas connected to : frdb1007.frack.eqiad.wmnet:9004 - https://alerts.wikimedia.org/?q=alertname%3DMariadbNoReplicas [08:38:17] RESOLVED: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-eqiad and (null) (10.195.0.249) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [08:38:51] RESOLVED: [4x] SwitchCoreInterfaceDown: Switch core interface down - fasw1-f5a-codfw:et-0/0/47 (Core: pfw1-codfw:et-0/1/0) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [08:40:47] RESOLVED: [6x] HAProxyExporterDown: HAProxy on fransc2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [08:51:10] RESOLVED: WidespreadPuppetFailure: Puppet has failed on over 5% of hosts - https://frmon.wikimedia.org/d/dwcgqww/puppet - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:51:43] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 06SRE: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12229923 (10cmooney) >>! In T434865#12228113, @cmooney wrote: > As discussed on irc I'll kick off the upgrade on pfw1-codfw at 09:00am UTC tomrorow (... [11:55:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: [TypeError] Civi\WMFQueueMessage\DonationModifyMessage::getContribution(): Return value must be of type array, null returned - https://phabricator.wikimedia.org/T434656#12230751 (10Laurabarluzzi) Other new logs from today: - /var/log/process-control/dona... [12:22:43] 06Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for tholland - https://phabricator.wikimedia.org/T435244#12230874 (10AKanji-WMF) [12:23:10] 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops, 06SRE: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12230878 (10cmooney) As per conversation with @Jgreen on irc we will do the eqiad pair at 09:00UTC tomorrow morning (Thurs 20th). [13:27:43] (03CR) 10Damilare Adedoyin: [C:03+2] Merge upstream in to our code [wikimedia/fundraising/crm/civiproxy] - 10https://gerrit.wikimedia.org/r/1321625 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [13:28:44] (03CR) 10Damilare Adedoyin: [C:03+2] Fix includes for rest4.php [wikimedia/fundraising/crm/civiproxy] - 10https://gerrit.wikimedia.org/r/1321631 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [13:29:40] (03CR) 10Damilare Adedoyin: [C:03+2] Add option to allow self-signed SSL cert (for development) [wikimedia/fundraising/crm/civiproxy] - 10https://gerrit.wikimedia.org/r/1321632 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [13:30:04] (03CR) 10Damilare Adedoyin: [C:03+2] Check in vendor and composer.lock [wikimedia/fundraising/crm/civiproxy] - 10https://gerrit.wikimedia.org/r/1321633 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [14:17:09] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fr-tech chores list - https://phabricator.wikimedia.org/T258527#12231502 (10Damilare) [14:23:24] (03PS4) 10Lars SG: SMS opt-out contacts from Acoustic when modified there [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1325564 (https://phabricator.wikimedia.org/T423967) [14:48:35] (03CR) 10CI reject: [V:04-1] SMS opt-out contacts from Acoustic when modified there [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1325564 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [15:03:32] (03PS5) 10Lars SG: SMS opt-out contacts from Acoustic when modified there [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1325564 (https://phabricator.wikimedia.org/T423967) [15:50:02] thanks damilare ! [15:56:36] 06Fundraising-Backlog, 06FR-donorrelations: dLocal: add product_name within the basket object - https://phabricator.wikimedia.org/T433851#12232460 (10SBurnett-WMF) @AKanji-WMF Response: This is what my friend says: Short answer: "Donation" works as a fallback, but it throws away the signal product_name is des... [16:18:55] (03CR) 10Wfan: [C:03+2] "testSetUtmValuesFromWmfValues is the unit test covers this looks good!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326969 (https://phabricator.wikimedia.org/T434734) (owner: 10Ejegg) [16:19:55] 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2627Q1 Analytics Updates - https://phabricator.wikimedia.org/T433888#12232607 (10AStein-WMF) [16:20:47] (03Merged) 10jenkins-bot: Do wmf_ -> utm_ conversion right in harvest loop [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326969 (https://phabricator.wikimedia.org/T434734) (owner: 10Ejegg) [16:25:07] 06Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for tholland - https://phabricator.wikimedia.org/T435244#12232626 (10AKanji-WMF) [16:37:49] (03CR) 10Lars SG: [C:03+2] Add leadgen API action [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1324417 (https://phabricator.wikimedia.org/T434150) (owner: 10Ejegg) [16:39:25] (03CR) 10Lars SG: [C:03+1] "This works for me and all looks reasonable, but would be good to have someone who knows a bit more about this stuff give it a quick once o" [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1324820 (https://phabricator.wikimedia.org/T434150) (owner: 10Ejegg) [16:46:14] thanks lars! [16:46:36] (03CR) 10Ejegg: [V:03+2] Merge upstream in to our code [wikimedia/fundraising/crm/civiproxy] - 10https://gerrit.wikimedia.org/r/1321625 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [16:46:54] (03CR) 10Ejegg: [V:03+2] Fix includes for rest4.php [wikimedia/fundraising/crm/civiproxy] - 10https://gerrit.wikimedia.org/r/1321631 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [16:47:02] (03CR) 10Ejegg: [V:03+2] Add option to allow self-signed SSL cert (for development) [wikimedia/fundraising/crm/civiproxy] - 10https://gerrit.wikimedia.org/r/1321632 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [16:47:13] (03CR) 10Ejegg: [V:03+2] Check in vendor and composer.lock [wikimedia/fundraising/crm/civiproxy] - 10https://gerrit.wikimedia.org/r/1321633 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [16:48:02] (03PS7) 10Ejegg: Update civiproxy to trixie / PHP8.4 [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1288927 (https://phabricator.wikimedia.org/T426078) [16:48:57] (03Merged) 10jenkins-bot: Add leadgen API action [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1324417 (https://phabricator.wikimedia.org/T434150) (owner: 10Ejegg) [16:49:56] (03PS6) 10Wfan: Add a Monthly Convert [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1326359 (https://phabricator.wikimedia.org/T433143) [17:00:09] (03PS2) 10Ejegg: Lead generation queue consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324962 (https://phabricator.wikimedia.org/T434150) [17:07:39] FIRING: SystemdService: Systemd service metabase.service stopped on fran1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [17:09:03] 06Fundraising-Backlog: Update fun-dev SSL cert - https://phabricator.wikimedia.org/T435374 (10Ejegg) 03NEW [17:12:39] RESOLVED: SystemdService: Systemd service metabase.service stopped on fran1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [17:21:34] FIRING: [2x] DiskSpace: Disk space WARNING for (instance frpm1002.frack.eqiad.wmnet:9100):/home - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:28:53] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 06FR-donorrelations, 03Civi RoadMap FY 26-27: PayPal flow doesn't allow employer field to be utilized - https://phabricator.wikimedia.org/T410318#12233218 (10AKanji-WMF) related {T433142} [17:30:01] 06Fundraising-Backlog, 07fr-payments-ACH, 07payments-orchestration: Check that Gravy's Trustly failure flow fixes are working as expected - https://phabricator.wikimedia.org/T435378 (10Ejegg) 03NEW [17:36:16] 06Fundraising-Backlog, 06FR-donorrelations, 07payments-orchestration: consider updating 'expiry date' to 'expiration date' on US donation form - https://phabricator.wikimedia.org/T414624#12233261 (10MBeat33) This is on WMF's end, per Gravy today https://docs.gr4vy.com/guides/payments/secure-fields/theming#u... [17:42:11] 06Fundraising-Backlog: Set up and test Gr4vy's 3DS - https://phabricator.wikimedia.org/T435380 (10RKumar_WMF) 03NEW [17:42:17] oh man we have two translation keys for that? [17:42:20] donate_interface-donor-expiration [17:42:25] donate_interface-donor-expiry-date [17:44:11] FIRING: [3x] NodeDown: Node fran1002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:44:47] FIRING: HAProxyExporterDown: HAProxy on fransw1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [17:48:57] FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : frdb2003.frack.codfw.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb2003.frack.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [17:49:11] RESOLVED: [3x] NodeDown: Node fran1002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:49:47] RESOLVED: HAProxyExporterDown: HAProxy on fransw1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [17:53:57] RESOLVED: MariadbReplicationRunning: MariaDB replica thread not running on : frdb2003.frack.codfw.wmnet:9004 - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb2003.frack.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning [17:56:41] FIRING: [4x] NodeDown: Node fran1002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:58:47] FIRING: HAProxyExporterDown: HAProxy on fransc1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [18:01:41] RESOLVED: [4x] NodeDown: Node fran1002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:03:47] RESOLVED: [2x] HAProxyExporterDown: HAProxy on fransc1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [18:06:48] 06Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for tholland - https://phabricator.wikimedia.org/T435244#12233401 (10Dwisehaupt) [18:08:18] 06Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for tholland - https://phabricator.wikimedia.org/T435244#12233407 (10Dwisehaupt) SSL client cert created and sent via email. Password sent via sms. Civicrm account created and tied to a new contact. Metabase account created and invite se... [18:09:15] 06Fundraising-Backlog, 10fundraising-tech-ops: CiviCRM access and certificate request for Maria Fernanda Umbarila Lara - https://phabricator.wikimedia.org/T433738#12233422 (10Dwisehaupt) [18:09:30] 06Fundraising-Backlog, 10fundraising-tech-ops: CiviCRM access and certificate request for Maria Fernanda Umbarila Lara - https://phabricator.wikimedia.org/T433738#12233423 (10Dwisehaupt) 05Open→03Resolved Verified account and access are working. [18:10:07] 06Fundraising-Backlog, 10fundraising-tech-ops, 06Fundraising-Tech-Roadmap: FR-Tech FY2627Q1 Analytics Updates - https://phabricator.wikimedia.org/T433888#12233429 (10AStein-WMF) 05Open→03In progress [18:10:46] 10fundraising-tech-ops: Fundraising access request for lmedley - https://phabricator.wikimedia.org/T432483#12233432 (10Dwisehaupt) [18:11:02] 10fundraising-tech-ops: Fundraising access request for lmedley - https://phabricator.wikimedia.org/T432483#12233433 (10Dwisehaupt) 05Open→03Resolved [18:17:49] 10fundraising-tech-ops: Switch over to new payments servers - https://phabricator.wikimedia.org/T424582#12233446 (10Dwisehaupt) [18:18:52] 10fundraising-tech-ops: Switch over to new payments servers - https://phabricator.wikimedia.org/T424582#12233447 (10Dwisehaupt) 05Open→03Resolved Pool swap happened 2026-08-18 and payments1005 payments1006 and payments1007 were fully pulled from payments config today. Closing. [18:19:47] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q1 maintenance window - https://phabricator.wikimedia.org/T419557#12233450 (10Dwisehaupt) [18:23:34] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Link and open tracking for CiviMail show as disabled when creating a new mailing, but they are not - https://phabricator.wikimedia.org/T435386 (10Lars) 03NEW [18:24:04] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Link and open tracking for CiviMail show as disabled when creating a new mailing, but they are not - https://phabricator.wikimedia.org/T435386#12233467 (10Lars) p:05Triage→03High [18:24:11] FIRING: NodeDown: Node frdb1006 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:25:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [14.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [18:26:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [12.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [18:26:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [12.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [18:26:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [12.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [18:29:11] RESOLVED: NodeDown: Node frdb1006 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:30:08] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 2 others: Engage <> Gr4vy for Direct Mail CC and ACH (moto) - https://phabricator.wikimedia.org/T417842#12233488 (10MDemosWMF) Gr4vy ACH update: They've made the changes to add the sec... [18:35:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [54.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [18:36:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [10.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [18:36:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [10.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [18:36:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [10.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [18:42:11] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-payments-general: GoFundMe integration - https://phabricator.wikimedia.org/T432890#12233574 (10MDemosWMF) @Lars @MSuijkerbuijk_WMF For those opting into email - what field in Civi should this align to? I see a general 'Opt-in' in the Communications... [18:44:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [30.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [18:44:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [30.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [18:44:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [30.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [18:49:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [30.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [18:49:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [30.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [18:49:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [30.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [18:49:51] (03PS1) 10Wfan: Add venmo to combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1327174 (https://phabricator.wikimedia.org/T432231) [18:51:09] 03Fundraising Sprint: Ottoman-o-war, 07Fundraising-ComboWiki-Dependency, 03Civi RoadMap FY 26-27, 07Epic, and 2 others: Add Gravy Venmo Vue Component - https://phabricator.wikimedia.org/T432231#12233609 (10AnnWF) [18:51:24] 03Fundraising Sprint: Ottoman-o-war, 07Fundraising-ComboWiki-Dependency, 03Civi RoadMap FY 26-27, 07Epic, and 2 others: Add Gravy Venmo Vue Component - https://phabricator.wikimedia.org/T432231#12233611 (10AnnWF) a:05Cstone→03AnnWF [18:55:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [31.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [18:58:38] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10fundraising-tech-ops, 03Civi RoadMap FY 26-27: FR-Tech FY2627Q1 maintenance window - https://phabricator.wikimedia.org/T419557#12233662 (10Dwisehaupt) [19:02:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [18.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [19:02:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [18.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [19:02:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [18.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [19:03:31] (03Abandoned) 10Wfan: WIP: Venmo on combowiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1311094 (https://phabricator.wikimedia.org/T432231) (owner: 10Cstone) [19:07:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [6.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [19:07:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [6.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [19:07:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [6.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [19:11:55] (03CR) 10Ejegg: [C:03+2] Update civiproxy to trixie / PHP8.4 [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1288927 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [19:12:19] (03CR) 10Ejegg: [V:03+2 C:03+2] Update civiproxy to trixie / PHP8.4 [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1288927 (https://phabricator.wikimedia.org/T426078) (owner: 10Ejegg) [19:12:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [34.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [19:17:18] (03PS4) 10Ejegg: Update fundraising-ml image to add Jupyter [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1321209 (https://phabricator.wikimedia.org/T434031) [19:20:34] 06Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for tholland - https://phabricator.wikimedia.org/T435244#12233792 (10Dwisehaupt) [19:21:21] 06Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for tholland - https://phabricator.wikimedia.org/T435244#12233793 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt Verified certificate is installed and access is working. [19:23:21] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-payments-general: GoFundMe integration - https://phabricator.wikimedia.org/T432890#12233811 (10Lars) @MDemosWMF Yes, Communications.optin is the main one, but we do unfortunately have others that we are trying to simplify, so let's check in before... [19:32:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [50.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [19:35:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail: INVALID_RECURRING from donations civi1002 Civi\WMFQueue\QueueConsumer - https://phabricator.wikimedia.org/T435394 (10Laurabarluzzi) 03NEW [19:41:00] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Develop & implement retention policy for email addresses overwritten when donor provides new email address on donation form - https://phabricator.wikimedia.org/T434832#12233875 (10SHust) Thanks again, @Lars. Please let me know if an... [19:42:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [11.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [19:42:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [11.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [19:42:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [11.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [19:47:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [19:47:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [19:47:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [19:51:20] (03PS5) 10Ejegg: Update fundraising-ml image to add Jupyter [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1321209 (https://phabricator.wikimedia.org/T434031) [19:51:37] (03CR) 10Ejegg: [V:03+2 C:03+2] Update fundraising-ml image to add Jupyter [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1321209 (https://phabricator.wikimedia.org/T434031) (owner: 10Ejegg) [20:02:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [41.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [20:02:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [41.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [20:02:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [41.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [20:05:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [58.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [20:07:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [13.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [20:07:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [13.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [20:07:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [13.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [20:10:33] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-payments-general: GoFundMe integration - https://phabricator.wikimedia.org/T432890#12233946 (10MDemosWMF) @Lars it looks like GoFundMe handles all TYLs at the time of the donation: "Both scenarios: since Wikimedia has claimed their Nonprofit Page,... [20:10:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [12.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [20:10:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [12.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [20:10:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [12.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [20:15:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [20:15:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [20:15:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [20:17:15] (03CR) 10Lars SG: [C:03+1] "I'm +1ing this as is and then I'll push up an amended commit with tests, opt in prefs, etc. So if you want to +2 after that, would be grea" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324962 (https://phabricator.wikimedia.org/T434150) (owner: 10Ejegg) [20:20:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [20:23:12] 06Fundraising-Backlog, 07FR-Fraud: New fraud model feature - https://phabricator.wikimedia.org/T435399 (10Ejegg) 03NEW [20:24:33] (03PS3) 10Lars SG: Lead generation queue consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1324962 (https://phabricator.wikimedia.org/T434150) (owner: 10Ejegg) [20:32:55] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, and 2 others: Send Leadgen emails to Civi then Acoustic - https://phabricator.wikimedia.org/T434150#12234076 (10Lars) [[ https://www.mediawiki.org/wiki/API:Action_API | MW API doc for CORS issu... [20:33:54] (03PS8) 10Ejegg: Setup for local lead generation testing [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1324820 (https://phabricator.wikimedia.org/T434150) [20:35:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [53.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [20:43:44] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, and 2 others: Send Leadgen emails to Civi then Acoustic - https://phabricator.wikimedia.org/T434150#12234091 (10Lars) @CaCogdill-WMF This one is getting close. Here are the details on how the f... [20:44:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [14.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [20:44:56] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-acoustic: Add & populate new columns to mailing_data_provider_table Content Id & Click Name - https://phabricator.wikimedia.org/T435402 (10Eileenmcnaughton) 03NEW [20:47:15] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Add endowment amount & Foundation amount to the batch data tables - https://phabricator.wikimedia.org/T435403 (10Eileenmcnaughton) 03NEW [20:47:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [8.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [20:47:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [8.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [20:47:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [8.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [20:48:32] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, and 2 others: Send Leadgen emails to Civi then Acoustic - https://phabricator.wikimedia.org/T434150#12234129 (10Lars) Also for deploy we'll need: [ ] a process control job. [20:57:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [20:57:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [20:57:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [5.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [21:01:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [51.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [21:02:01] (03PS1) 10Eileen: Make subject more useful for failmail [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327192 [21:03:01] (03CR) 10CI reject: [V:04-1] Make subject more useful for failmail [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327192 (owner: 10Eileen) [21:03:51] (03PS2) 10Eileen: Make subject more useful for failmail [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327192 [21:06:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [36.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [21:15:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 10Recurring-Donations: Trustly subsequent recurring fails with invalid_service_credentials - https://phabricator.wikimedia.org/T434933#12234219 (10MBeat33) We're seeing these fail at a low but steady rate: 8/19 - 2 donations so f... [21:19:19] (03PS1) 10Eileen: Improve failmail subject when removed from the queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327195 [21:21:34] FIRING: DiskSpace: Disk space WARNING for (instance frpm1002.frack.eqiad.wmnet:9100):/home - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:24:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [56.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [21:25:43] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Failmail ack system - https://phabricator.wikimedia.org/T435410 (10Eileenmcnaughton) 03NEW [21:29:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [80.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [21:40:23] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE, and 2 others: Add MGFE requested fields from audience framework refresh - https://phabricator.wikimedia.org/T430012#12234337 (10Lars) One update for deployment: ` CREATE TABLE IF NOT EXISTS `silverpop_mg... [21:54:55] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#12234439 (10Lars) Alter to get rid of indexes so we can run syn... [21:56:07] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#12234451 (10Lars) Drop once we've updated the triggers: ` ALTER... [22:14:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [22:19:07] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#12234585 (10Lars) The Acoustic queries to update data that will... [22:44:39] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Add endowment amount & Foundation amount to the batch data tables - https://phabricator.wikimedia.org/T435403#12234654 (10Eileenmcnaughton) Oh look - it turns out past me already thought of this - we have 'amount_journalled_to_endowment'... [23:02:20] (03PS1) 10Eileen: Add additional mailing-provider fields for analytics [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327211 (https://phabricator.wikimedia.org/T435402) [23:10:39] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [23:14:03] (03PS1) 10Lars SG: Redeploy after revert: Add app_install_id column to contribution_tracking [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1327213 (https://phabricator.wikimedia.org/T433883) [23:17:23] (03CR) 10Eileen: [C:03+2] SMS opt-out contacts from Acoustic when modified there [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1325564 (https://phabricator.wikimedia.org/T423967) (owner: 10Lars SG) [23:19:43] FIRING: RedisQueueSize: Redis Queue contribution_tracking is high: [2306] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [23:22:43] FIRING: RedisQueueSize: Redis Queue contribution_tracking is Critical: [2760] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [23:22:52] (03PS2) 10Lars SG: Redeploy after revert: Add app_install_id column to contribution_tracking [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1327213 (https://phabricator.wikimedia.org/T433883) [23:23:22] ^^^ acking that since we are doing maint. [23:23:47] (03CR) 10CI reject: [V:04-1] Redeploy after revert: Add app_install_id column to contribution_tracking [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1327213 (https://phabricator.wikimedia.org/T433883) (owner: 10Lars SG) [23:24:11] (03CR) 10CI reject: [V:04-1] Add additional mailing-provider fields for analytics [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327211 (https://phabricator.wikimedia.org/T435402) (owner: 10Eileen) [23:27:11] (03PS3) 10Lars SG: Redeploy after revert: Add app_install_id column to contribution_tracking [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1327213 (https://phabricator.wikimedia.org/T433883) [23:30:52] (03CR) 10Lars SG: [C:03+2] Redeploy after revert: Add app_install_id column to contribution_tracking [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1327213 (https://phabricator.wikimedia.org/T433883) (owner: 10Lars SG) [23:31:47] (03Merged) 10jenkins-bot: Redeploy after revert: Add app_install_id column to contribution_tracking [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1327213 (https://phabricator.wikimedia.org/T433883) (owner: 10Lars SG) [23:33:46] (03PS1) 10Lars SG: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1327216 [23:35:39] FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DSystemdService [23:36:30] !log civicrm upgraded from 4f83dccd to ba2bc5f4 [23:36:32] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [23:36:51] (03CR) 10Lars SG: [C:03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1327216 (owner: 10Lars SG) [23:38:34] (03Merged) 10jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1327216 (owner: 10Lars SG) [23:40:21] !log tools upgraded from 9f2c4da9 to 41162cd2 [23:40:22] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [23:42:32] (03PS2) 10Eileen: Add additional mailing-provider fields for analytics [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327211 (https://phabricator.wikimedia.org/T435402) [23:42:33] (03PS1) 10Eileen: Populated additional mailing fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327218 (https://phabricator.wikimedia.org/T435402) [23:43:04] 03Fundraising Sprint: Ottoman-o-war, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint: Remove un-needed separate endowment/foundation calculated fields from Civi & Acoustic - https://phabricator.wikimedia.org/T418885#12234800 (10Dwisehaupt) >>! In T418885#12117670, @Lars wrote: >... [23:47:59] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-acoustic, 13Patch-For-Review: Add & populate new columns to mailing_data_provider_table Content Id & Click Name - https://phabricator.wikimedia.org/T435402#12234820 (10Dwisehaupt) Alter complete: ` MariaDB [civicrm]> ALTER TABLE `civicrm_mailing_... [23:50:56] (03CR) 10Dmiranda: [C:03+2] "Thanks @emcnaughton@wikimedia.org!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1327218 (https://phabricator.wikimedia.org/T435402) (owner: 10Eileen)