[00:37:43] (03PS1) 10Eileen: Accept any folder in audit root for files path [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296701 (https://phabricator.wikimedia.org/T424083) [00:41:56] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Make it possible to configure the from name and email address for all transactional emails from Civi - https://phabricator.wikimedia.org/T428004#11979715 (10ppenloglou) We do this on some fundraising emails sent from Acoustic. ` From Name English: Li... [00:54:07] (03PS1) 10Eileen: Add DAF Pay payment instrument [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296702 (https://phabricator.wikimedia.org/T415825) [00:54:48] (03PS3) 10Eileen: Add guzzle support to GenerateBatchTest & fix update to exported for when they all work [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296490 [00:57:06] (03CR) 10Ejegg: "Do we want to style it like they do in their branding 'DAFpay' ?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296702 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:58:45] (03PS2) 10Eileen: Add DAF Pay payment instrument [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296702 (https://phabricator.wikimedia.org/T415825) [00:58:56] (03CR) 10Eileen: "yeah makes sense" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296702 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [01:12:57] (03CR) 10Ejegg: [C:03+2] Add DAF Pay payment instrument [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296702 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [01:13:01] 03Fundraising Sprint: Jack's Junk Drawer, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, 07fr-current-sprint: Update Metadbase chariot board for changes - https://phabricator.wikimedia.org/T428014 (10Eileenmcnaughton) 03NEW [01:24:26] (03PS1) 10Eileen: - Upgrading symfony/polyfill-mbstring (v1.37.0 => v1.38.1): Extracting archive - Upgrading psr/log (1.1.4 => 3.0.2): Extracting archive - Upgrading symfony/process (v6.4.39 => v7.4.13): Extracting archive - Upgrading knplabs/knp-snappy (v1.4.4 => v1.7.2): Extracting archive - Upgrading symfony/polyfill-intl-normalizer (v1.37.0 => v1.38.0): Extracting archive - Upgrading symfony/polyfill-intl-idn ( [01:24:27] Upgrading symfony/polyfill-php81 (v1.37.0 => v1.38.1): Extracting archive - Upgrading symfony/polyfill-php82 (v1.37.0 => v1.38.1): Extracting archive - Upgrading symfony/polyfill-php83 (v1.37.0 => v1.38.1): Extracting archive - Upgrading symfony/polyfill-php84 (v1.37.0 => v1.38.1): Extracting archive - Upgrading omnimail/omnimail (dev-master 6b5104b => dev-master 9b5434d): Checking out 9b5434d133 from cache - Upgrading [01:24:27] v1.2.4.11 871acfa): Checking out 871acfae61 from cache [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1296706 [01:24:44] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296707 [01:25:02] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296707 (owner: 10Eileen) [01:25:22] (03CR) 10Eileen: [C:03+2] - Upgrading symfony/polyfill-mbstring (v1.37.0 => v1.38.1): Extracting archive - Upgrading psr/log (1.1.4 => 3.0.2): Extracting archive [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1296706 (owner: 10Eileen) [01:31:43] FIRING: RedisQueueSize: Redis Queue contribution_tracking is high: [2030] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [01:36:27] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296707 (owner: 10Eileen) [01:36:43] FIRING: RedisQueueSize: Redis Queue contribution_tracking is Critical: [2586] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [01:41:43] FIRING: [2x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [3074] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [01:46:43] FIRING: [3x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [3757] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [01:56:05] (03Merged) 10jenkins-bot: - Upgrading symfony/polyfill-mbstring (v1.37.0 => v1.38.1): Extracting archive - Upgrading psr/log (1.1.4 => 3.0.2): Extracting archive - Upgrading symfony/process (v6.4.39 => v7.4.13): Extracting archive - Upgrading knplabs/knp-snappy (v1.4.4 => v1.7.2): Extracting archive - Upgrading symfony/polyfill-intl-normalizer (v1.37.0 => v1.38.0): Extracting archive - Upgrading symfony/polyfill-in [01:56:05] - Upgrading symfony/polyfill-php81 (v1.37.0 => v1.38.1): Extracting archive - Upgrading symfony/polyfill-php82 (v1.37.0 => v1.38.1): Extracting archive - Upgrading symfony/polyfill-php83 (v1.37.0 => v1.38.1): Extracting archive - Upgrading symfony/polyfill-php84 (v1.37.0 => v1.38.1): Extracting archive - Upgrading omnimail/omnimail (dev-master 6b5104b => dev-master 9b5434d): Checking out 9b5434d133 from cache - Upgradin [01:56:05] v1.2.4.11 871acfa): Checking out 871acfae61 from cache [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1296706 (owner: 10Eileen) [02:07:55] !log civicrm upgraded from 663c0c30 to f6839b65 [02:07:56] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [02:08:42] eileen: msg harassment but the queues arent currently running [02:08:52] cstone: huh? [02:09:10] hah the queues havent run since you deployed that process control change earlier [02:09:18] any idea how to kick them I tried undoing it [02:09:23] oh - did I break them? [02:09:34] i dont think so i think the deploy itself did it [02:09:42] i undid your change and they still arent running [02:09:46] im gona msg jeff idono what else [02:10:17] hmm - they 'sound' ok Press to restart process-control, or to skip. y [02:10:17] civi1002 no errors [02:10:17] civi2002 no errors [02:10:17] frdev1002 no errors [02:10:17] frdev1003 no errors [02:10:26] yep but nothings run since 23:13 [02:10:39] and the queues are all backed up [02:11:03] ug - just running donations_queue_consume [02:11:10] seems to run ok when run [02:11:30] yeah its process control not running it [02:11:51] scrolling back - I remember there was something odd but I don't think material - looking [02:13:54] it also pushed out process-control /srv/process-control [02:13:54] * postinstall [02:13:54] ◦ civi1002/paypal_frup_audit_download.yaml updated [02:13:55] ◦ civi1002/chariot_audit_download.yaml updated [02:14:08] could try removing that one - although the job ran fine [02:14:16] i already reverted the change you made there [02:14:25] and its still not running them i think maybe it just needs to be fully restarted? [02:14:25] the frup too? [02:14:30] i didnt see that in your change [02:15:10] no - I think it was already pushed but trying to figure out [02:21:43] FIRING: [4x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [8722] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [02:26:43] FIRING: [4x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [9265] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [02:31:43] FIRING: [4x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [9265] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [02:35:36] (03CR) 10Cstone: [C:03+2] Accept any folder in audit root for files path [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296701 (https://phabricator.wikimedia.org/T424083) (owner: 10Eileen) [02:36:43] RESOLVED: [4x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [9265] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [02:36:43] RESOLVED: [5x] RedisQueueSize: Redis Queue contribution_tracking is high: [9265] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [02:58:19] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Make it possible to configure the from name and email address for all transactional emails from Civi - https://phabricator.wikimedia.org/T428004#11979796 (10Lars) It would definitely be simpler to implement per message template, rather than per translatio... [02:58:34] (03Merged) 10jenkins-bot: Accept any folder in audit root for files path [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296701 (https://phabricator.wikimedia.org/T424083) (owner: 10Eileen) [02:58:35] (03CR) 10Eileen: [C:03+2] Update vendor libraries [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296616 (owner: 10Ejegg) [03:01:02] (03PS1) 10Eileen: Vendor update [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1296709 [03:01:16] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296710 [03:01:27] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296710 (owner: 10Eileen) [03:01:55] (03CR) 10Eileen: [C:03+2] Vendor update [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1296709 (owner: 10Eileen) [03:14:10] (03Abandoned) 10Eileen: Upgrading wikimedia/omnimail-silverpop (1.31 => 1.32) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283759 (owner: 10Eileen) [03:14:23] (03PS1) 10Eileen: Update omnimail-silverpop [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296711 [03:18:33] (03CR) 10Cstone: [C:03+2] Update omnimail-silverpop [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296711 (owner: 10Eileen) [03:22:26] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296710 (owner: 10Eileen) [03:32:45] !log civicrm upgraded from f6839b65 to 219cf085 [03:32:46] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [03:36:30] (03CR) 10CI reject: [V:04-1] Update omnimail-silverpop [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296711 (owner: 10Eileen) [03:46:48] (03PS1) 10Eileen: Add more logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296712 (https://phabricator.wikimedia.org/T424083) [03:52:05] (03CR) 10Cstone: [C:03+2] Add more logging [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296712 (https://phabricator.wikimedia.org/T424083) (owner: 10Eileen) [04:02:51] (03PS1) 10Eileen: Also use incoming directory for completed & ignored [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296714 (https://phabricator.wikimedia.org/T424083) [04:07:11] (03CR) 10Eileen: [C:04-1] "test fails relate - hmmm" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296711 (owner: 10Eileen) [04:18:24] (03CR) 10Cstone: [C:03+2] Also use incoming directory for completed & ignored [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296714 (https://phabricator.wikimedia.org/T424083) (owner: 10Eileen) [04:22:28] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296715 [04:22:37] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296715 (owner: 10Eileen) [04:23:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [04:28:09] !log civicrm upgraded from 219cf085 to 55adc0bb [04:28:10] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [04:28:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [04:33:37] 06Fundraising-Backlog: How will we use vue in combowiki - https://phabricator.wikimedia.org/T428019 (10Cstone) 03NEW [04:33:44] 06Fundraising-Backlog: How will we use vue in combowiki - https://phabricator.wikimedia.org/T428019#11979870 (10Cstone) [04:33:47] 06Fundraising-Backlog, 07Fundraising-ComboWiki-Dependency, 07Epic, 07fr-combowiki: [Epic] ComboWiki - https://phabricator.wikimedia.org/T417039#11979871 (10Cstone) [04:45:41] (03PS1) 10Eileen: Drop use of realpath for checks [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296717 (https://phabricator.wikimedia.org/T424083) [04:48:43] (03Merged) 10jenkins-bot: Also use incoming directory for completed & ignored [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296714 (https://phabricator.wikimedia.org/T424083) (owner: 10Eileen) [04:53:08] (03CR) 10Cstone: [C:03+2] Drop use of realpath for checks [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296717 (https://phabricator.wikimedia.org/T424083) (owner: 10Eileen) [05:14:56] (03Merged) 10jenkins-bot: Drop use of realpath for checks [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296717 (https://phabricator.wikimedia.org/T424083) (owner: 10Eileen) [05:45:00] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296930 [05:45:13] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296930 (owner: 10Eileen) [05:46:07] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1296930 (owner: 10Eileen) [05:46:44] !log civicrm upgraded from 55adc0bb to 6b61d3a5 [05:46:45] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [05:49:55] (03PS1) 10Cstone: WIP: Handle double currency conversion [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1296931 (https://phabricator.wikimedia.org/T426098) [06:12:00] (03PS1) 10Eileen: Get PaymentMethod in both places, DAFpay as payment method [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1296935 (https://phabricator.wikimedia.org/T415825) [06:12:29] (03CR) 10CI reject: [V:04-1] Get PaymentMethod in both places, DAFpay as payment method [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1296935 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [06:13:59] (03PS2) 10Eileen: Get PaymentMethod in both places, DAFpay as payment method [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1296935 (https://phabricator.wikimedia.org/T415825) [07:25:00] 10Fundraising analytics stack, 06Fundraising-Backlog, 10fundraising-tech-ops, 06collaboration-services, and 2 others: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#11980026 (10ABran-WMF) [07:27:57] (03PS1) 10Eileen: Extract buildAuditRows [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1296948 (https://phabricator.wikimedia.org/T415825) [07:28:32] (03CR) 10CI reject: [V:04-1] Extract buildAuditRows [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1296948 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [07:30:39] (03PS2) 10Eileen: Extract buildAuditRows [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1296948 (https://phabricator.wikimedia.org/T415825) [10:30:35] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Smarty templates failing to render when donor name has an apostrophe - https://phabricator.wikimedia.org/T421166#11980775 (10ppenloglou) Agreed not to update all these templates right now as: - This was occuring in T... [11:09:05] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Adyen webhooks for Gravy-orchestrated transactions arriving with a colon-separated merchantReference breaking Base62 conversion - https://phabricator.wikimedia.org/T428042 (10jgleeson) 03NEW [11:27:07] (03PS3) 10Jgleeson: Skip Base62 conversions for problematic Gravy-related Adyen webhooks. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297103 (https://phabricator.wikimedia.org/T428042) [11:49:32] (03CR) 10Damilare Adedoyin: [C:03+2] Skip Base62 conversions for problematic Gravy-related Adyen webhooks. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297103 (https://phabricator.wikimedia.org/T428042) (owner: 10Jgleeson) [11:50:24] (03Merged) 10jenkins-bot: Skip Base62 conversions for problematic Gravy-related Adyen webhooks. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297103 (https://phabricator.wikimedia.org/T428042) (owner: 10Jgleeson) [11:50:25] thanks for the patch jgleeson|skrun, lgtm! [12:02:03] thanks damilare I'll push that out [12:02:10] np [12:09:33] damilare: When I was standing on the school playground, I thought, "Maybe I should have limited that code change to just capture webhooks, as I think we do stuff with the others", so I'm gonna push up a small tweak to the earlier patch to do that. [12:10:06] they will still fail if we get a refund with this new format but we probably want to know about it [12:10:46] all of today's failures so far have been capture messages [12:10:53] so i think it's specific to those [12:11:45] true, yh lets fix the current problem first and if it persist we can push it back up [12:14:38] (03PS1) 10Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1297113 [12:16:02] (03CR) 10Jgleeson: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1297113 (owner: 10Jgleeson) [12:16:13] sound good [12:16:44] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1297113 (owner: 10Jgleeson) [12:17:53] !log SmashPig upgraded from 166abfbd to 99233b18 [12:17:54] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [12:17:56] ok that fix has gone out [12:25:37] (03PS5) 10Jgleeson: Restrict b10ee0a45 to only apply for CAPTURE events [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297112 (https://phabricator.wikimedia.org/T428042) [12:26:03] that's the patch to tighten it up damilare ^ [12:28:38] thanks jgleeson, since there's a dedicated capture message class I'm thinking we could update the property in the notifications array there [12:31:34] hmm what are you thinking [12:33:40] copying across the existing setup to constructFromJSON in Capture? [12:33:43] so we wouldn't have to list the event codes in the Adyen Message, we can do the logic of identifying the issue in the child class and maybe removing the property. [12:34:43] I think we'd have to replicate all the shared assignments so that we don't run into the Base62Helper::toUuid() failure [12:35:27] or maybe add a new method in the base that capture overrides [12:36:46] replicating all the setup in the constructor feels like overkill to me [12:37:19] damilare: if we redirect to the child, we need to do the other setup too [12:37:34] unless we introduce hooks that one/all use [12:38:25] this is what I had in mind: https://dotmdeditor.dev/?doc=6dc9rg7u [12:38:39] but the override method also sounds good too [12:40:09] I think unsetting it there will cause a few other issues [12:40:36] it's the orchestratorTransactionID we want to set to null [12:40:53] the $merchantReference is assigned in a few places in the parent constructor [12:41:09] so setting that to null will trigger warnings I think [12:41:28] I wonder if that uuid method accepts null [12:42:03] this would break Base62Helper::toUuid( $notification['merchantReference'] ); [12:42:07] as it expects a string [12:42:25] in the parent [12:42:54] that's true, adding the method to set the orchestratorTransactionID is a better idea [12:43:04] and overriding in the capture message [12:43:59] ok cool I'll turn it into a hook that we can override [12:52:11] FIRING: NodeDown: Node frdb1006 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [12:55:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [12:57:11] RESOLVED: NodeDown: Node frdb1006 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [13:01:56] 10fundraising-tech-ops: Fundraising access request and new user setup for Stann-Omar Jones - https://phabricator.wikimedia.org/T427292#11981266 (10Jgreen) [13:05:11] FIRING: NodeDown: Node frdb2005 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb2005 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [13:05:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [13:10:03] (03PS7) 10Jgleeson: Add hook for parseOrchestratorTransactionId() [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297112 (https://phabricator.wikimedia.org/T428042) [13:10:11] RESOLVED: [2x] NodeDown: Node frdb1006 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [13:10:16] ok damilare I agree that looks more tidy [13:10:27] check out that latest follow-on [13:11:23] thanks! [13:11:46] actually let me fix that commit message [13:11:56] I renamed the hook and forgot to update the commit [13:13:21] (03CR) 10Damilare Adedoyin: "Thanks for all the work on this. LGTM!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297112 (https://phabricator.wikimedia.org/T428042) (owner: 10Jgleeson) [13:13:33] (03CR) 10Damilare Adedoyin: [C:04-1] Add hook for parseOrchestratorTransactionId() [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297112 (https://phabricator.wikimedia.org/T428042) (owner: 10Jgleeson) [13:13:47] oops, just saw your last message jgleeson [13:13:54] (03PS8) 10Jgleeson: Add overridable hook for Adyen Base62 merchantReference parsing [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297112 (https://phabricator.wikimedia.org/T428042) [13:13:59] hence the -1 [13:14:15] does that commit look better :) [13:14:51] yea it does [13:15:05] awesome [13:15:14] (03CR) 10Damilare Adedoyin: [C:03+2] "LGTM!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297112 (https://phabricator.wikimedia.org/T428042) (owner: 10Jgleeson) [13:15:20] ty! [13:15:33] np! [13:15:35] I'll push that out and get back to what I originally planned to start my day with [13:15:44] @_@ [13:15:51] (03Merged) 10jenkins-bot: Add overridable hook for Adyen Base62 merchantReference parsing [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1297112 (https://phabricator.wikimedia.org/T428042) (owner: 10Jgleeson) [13:16:29] also, I noticed dotmd has php code support [13:16:38] it's coming along! [13:22:33] yh it's just code markdown, looking to add those beautiful colors 😃 [13:26:22] that would be cool [13:30:37] jgleeson: do you also get this error when you do the mysqldump on local civi docker: "TLS/SSL error: SSL is required, but the server does not support it" when trying to connect [13:31:01] can you paste the command [13:31:06] using the command in the setup script currently [13:31:18] docker compose exec civicrm mysqldump -hdatabase -uroot civicrm > ./.backup/sql/civicrm.sql [13:31:52] oh is this that post-install backup thing [13:31:57] yea [13:32:18] I also get it [13:32:33] I wonder how that broke [13:32:39] ok interesting, seems to work if you add --skip_ssl [13:32:51] however I'm getting db errror on civi [13:32:54] although to be honest we probably don't need them anymore. I think they were useful when our unit tests would break the data [13:32:59] so I'm wondering if its related [13:33:08] and it was quicker to retore than rebuild, on drupal [13:33:22] ohh ok, we should take that out then [13:33:32] I'd push up a patch to remove it [13:33:45] yeah. I don't think we do it for any of the others [13:34:29] one more thing though, can you try to reinstall the wmf civicrm extension on your civi project to see if its broken for you too [13:35:09] mine shows "DB Error: no such database" and I have all the project db setup [13:35:42] I did a reinstall on Monday, I think it was working before then, so might be related to that. [13:39:55] I'm just about to test out a recurring charge with civi using the payment_service_id, so I don't want to break civi just yet, but once I've finished this end-to-end test to confirm it all still works, I will give it a go [13:40:24] out of interest, maybe try checking out an older version of standlone and see if that builds [13:40:24] thanks! [13:41:02] makes sense, I'd try that. [14:13:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [43.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [14:28:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [40.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [14:36:40] damilare: can I take a look at the console where you were installing? [14:37:57] sure, what do you want to see? [14:38:52] we could join the standup call if you want [14:39:27] I think the most notable error is: POST https://wmf.localhost:32353/civicrm/admin/extensions 500 (Internal Server Error) [14:42:31] ok, I'll get in the standup call [15:05:24] ejegg: I pinged you last night about that payments attempts table approach. did you see it? what did you settle on [15:08:36] oops, i missed it jgleeson - on IRC? [15:08:49] yeah [15:08:52] lemme find it [15:09:17] Jack Gleeson [15:09:17] ejegg: what was the final design with the payments attempts table stuff [15:09:17] 10:17 PM [15:09:17] did you figure out a way to write to mysql from payments [15:10:17] jgleeson: yes, there is going to be a new server that accepts webservice calls from payments [15:10:24] and has its own mariadb instance [15:10:37] they are going to repurpose an older payment server for now [15:10:40] ah so it's a web service [15:10:46] or pair of them, so it can fail over [15:10:46] not mysql writes [15:11:02] right, payments-wiki itself doesn't do the mysql write [15:11:03] s [15:11:08] got it [15:11:42] payments-wiki calls the webservice which stores, gets the counts, derives more features, and feeds it to the XGBoost model [15:11:52] then returns the prediction to payments-wiki [15:15:53] so how does that plan impact our current fraud filters [15:16:39] I can understand how dami's model training can now happen on persisted data [15:17:20] will this service essentially be another filter [15:17:58] jgleeson: yep [15:19:18] Sucheta reached out recently saying the ML folks couldn't help us with liftwing due to their workload for the quarter. Does that impact this plan? [15:19:20] damilare: ^ [15:19:20] the service can return an array of scores: one from the xgboost model and others from any additional rule-based filters we port over from donationinterface [15:20:11] ah, that's a bummer - I guess we'll see how much we can train locally [15:20:14] I think we need to join a queue if we need their help [15:20:19] or rather how fast we can train locally [15:21:25] I know dami was able to train on almost 400k transactions exported from gravy just on his local machine [15:21:30] right - we might be able to use it next quarter [15:21:36] no I don't think so [15:23:22] ah that's good [15:23:41] I feel like we could frame our ask better [15:23:47] if needed [15:24:11] hmm, though we really need to keep all the payment attempt data in the frack cluster [15:24:38] from "we're experimenting with ML" to "if this works, it will increase revenue" [15:25:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [42.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [15:40:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [42.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [15:42:07] (03PS1) 10Damilare Adedoyin: Add certificates for mysql ssl [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1297182 [15:50:59] ERROR 2026 (HY000): TLS/SSL error: self-signed certificate in certificate chain [15:51:16] I guess we need to use a cert that's recognized on all the clients [15:52:31] we don't have a shared local CA cert, do we? [15:56:36] interesting, that's a different error [15:57:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [43.9] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [16:07:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [42.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [16:09:31] (03PS3) 10Wfan: Add the testkitchen in docker env [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1281400 [16:22:27] (03PS3) 10Jgleeson: Refactor: extract Donor Portal error handling into shared helper [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1296679 [16:22:30] (03CR) 10Ejegg: [C:03+2] Refactor: extract Donor Portal error handling into shared helper [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1296679 (owner: 10Jgleeson) [16:24:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [16:44:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [16:49:07] 06Fundraising-Backlog: Update payments.wiki annual donation phrase to "yearly" - https://phabricator.wikimedia.org/T428084 (10KMorrow-WMF) 03NEW [17:05:32] 10fundraising-tech-ops: Fundraising access request and new user setup for Stann-Omar Jones - https://phabricator.wikimedia.org/T427292#11982494 (10IAckerman-WMF) [17:05:54] 10fundraising-tech-ops: Fundraising access request and new user setup for Stann-Omar Jones - https://phabricator.wikimedia.org/T427292#11982496 (10IAckerman-WMF) (Added GitLab access) [17:19:30] (03Merged) 10jenkins-bot: Refactor: extract Donor Portal error handling into shared helper [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1296679 (owner: 10Jgleeson) [18:34:49] 03Fundraising Sprint: Jack's Junk Drawer, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 2 others: Update Metadbase chariot board for changes - https://phabricator.wikimedia.org/T428014#11982713 (10MDemosWMF) [19:48:57] (03PS1) 10Umherirrender: Use $wgExtensionDirectory in DonorPortal/Hooks [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1297223 [19:59:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [43.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [20:09:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [40.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [20:40:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [43.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [20:55:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [40.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [21:37:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [74.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [21:46:21] (03CR) 10Cstone: [C:03+2] "this worked locally for me, both with a new recurring with the payment_service_id, an older gravy one that didn't have it, and an older ad" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1278542 (https://phabricator.wikimedia.org/T422416) (owner: 10Jgleeson) [21:47:19] thank you cstone ! [21:47:24] thanks for all the work jgleeson [21:59:36] FIRING: StrongswanFailure: Strongswan is not fully established on frban1002.frack.eqiad.wmnet:9100 for peerhost frdev1003 - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [22:00:52] (03PS3) 10Wfan: Delete payment method for pix recurring [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1295959 (https://phabricator.wikimedia.org/T424245) [22:09:09] (03Merged) 10jenkins-bot: Store and replay payment_service_id for Gravy recurring charges [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1278542 (https://phabricator.wikimedia.org/T422416) (owner: 10Jgleeson) [22:12:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [65.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [23:51:28] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Remove advertised source_contact_id:name fields in SK which don't work - https://phabricator.wikimedia.org/T428005#11983522 (10Lars) p:05Triage→03Low [23:54:27] 03Fundraising Sprint: Jack's Junk Drawer, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Request for improvement to prospecting system to track moves management - https://phabricator.wikimedia.org/T386961#11983533 (10Lars) a:03Lars