[01:25:02] (03PS1) 10Ejegg: Mixin to target campaigns by user edit count [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1352670 (https://phabricator.wikimedia.org/T269758) [01:29:04] oh needs the i18n messages added to admin [01:38:06] (03PS2) 10Ejegg: Mixin to target campaigns by user edit count [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1352670 (https://phabricator.wikimedia.org/T269758) [01:43:15] (03PS3) 10Ejegg: Move 11 yr old mystery TODO [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1348867 [01:55:30] (03PS1) 10Ejegg: Configure beacon path for donutwiki [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1352680 [02:28:08] (03CR) 10Ejegg: [C:04-1] "Spotted one more issue" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1346137 (https://phabricator.wikimedia.org/T438355) (owner: 10Wfan) [02:33:50] (03CR) 10Ejegg: [V:03+2] Move 11 yr old mystery TODO [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1348867 (owner: 10Ejegg) [02:45:37] (03CR) 10Ejegg: [C:03+2] "Looks good! TIL the spaceship operator exists in MariaDB but does something totally different from what it does in PHP." [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1352605 (https://phabricator.wikimedia.org/T437967) (owner: 10Lars SG) [02:48:23] (03Merged) 10jenkins-bot: Choose individual contacts over organization contacts that share emails [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1352605 (https://phabricator.wikimedia.org/T437967) (owner: 10Lars SG) [03:02:52] (03PS2) 10Lars SG: Add both_funds_overall_latest_cy_native_total & _count [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1351489 (https://phabricator.wikimedia.org/T438035) [03:14:32] (03CR) 10Lars SG: Choose individual contacts over organization contacts that share emails (032 comments) [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1352605 (https://phabricator.wikimedia.org/T437967) (owner: 10Lars SG) [03:46:16] (03PS1) 10Cstone: Update smash-pig to v1.2.7.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1352725 [03:46:59] (03CR) 10Cstone: [C:03+2] Update smash-pig to v1.2.7.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1352725 (owner: 10Cstone) [03:55:34] (03Merged) 10jenkins-bot: Update smash-pig to v1.2.7.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1352725 (owner: 10Cstone) [04:12:13] (03PS1) 10Cstone: Update smash-pig to v1.2.7.1 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1352746 [04:13:39] (03CR) 10Cstone: [C:03+2] Update smash-pig to v1.2.7.1 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1352746 (owner: 10Cstone) [04:19:41] 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog: Adding new anonymous flag to Chariot email alerts - https://phabricator.wikimedia.org/T433441#12409625 (10Eileenmcnaughton) @MDemosWMF this is done isn't it? [04:21:38] (03Merged) 10jenkins-bot: Update smash-pig to v1.2.7.1 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1352746 (owner: 10Cstone) [04:24:23] (03PS1) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1352753 [04:25:18] (03CR) 10Cstone: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1352753 (owner: 10Cstone) [04:26:11] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1352753 (owner: 10Cstone) [04:27:48] !log civicrm upgraded from d3ad8686 to 83683472 [04:27:49] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [04:34:26] FIRING: [3x] EndpointResultCritical: Endpoint check for civiproxy on civi1002 is CRITICAL - https://alerts.wikimedia.org/?q=alertname%3DEndpointResultCritical [04:36:25] hmm did i do that with the civi deploy... [04:37:23] it was just deleting tests... [04:41:57] hmm with rollback its still down [04:53:58] sorry dwisehaupt [04:54:07] i dont know what i did haha [04:54:29] howdy [04:54:39] i rolled back the deploy which shouldnt have affected anyhting [04:54:41] looks like civiproxy is down on most hosts. [04:54:43] and its still down [04:54:51] ok so just suspicous timing right after my deploy hah [04:55:09] i'll check logs. just got back from wood turning. [04:55:36] thanks, i dont know where to look from my end for logs best I could find is [04:55:38] frdata:Oct 8 04:41:22 frdata1003 EmailPreferences: Error from civiproxy: CiviCRM is currently unreachable [04:56:32] i manually ran the checkoutcom audit parser after the deploy which then hung too but that shouldnt have anything to do with civiproxy [04:57:45] i'm seeing requests to civiproxy in the last minute. [04:58:40] is it just not responding? [05:00:20] that i'm not sure. [05:00:38] oh, it's responding with 500's [05:01:23] civiproxy talks to civicrm, so maybe there is something in that log to find. [05:02:48] i confirmed i rolled back the civi deploy [05:04:01] i'm going to try restarting nginx and apache. expecting no change but let's see. [05:05:04] yeah. still seeing "We are currently performing maintenance on our donor database. You may still request a login link, and we will send it as soon as the maintenance is complete" [05:06:18] what i was doing when it broke was using run-job -j to run the checkoutcom audit which in theory should not have anything to do with civi proxy [05:07:44] well, if it broke civi on the back end that could cause civiproxy to not work and toss 500s [05:08:07] i think I killed the job though or how can we see what is currently running [05:08:31] normal civi seems to work fine [05:08:34] i see the f_c_u change to 83683472 and then the rollback to d3ad8686. and the rsync blasters are clean too. [05:08:50] localsettings didn't change. [05:09:05] could there have been a change with how civiproxy auths? [05:09:23] or something that may stick around and not get clean up? [05:09:43] (i'm not 100% certain on all of civiproxy) [05:10:38] it shouldnt the code i deployed was 2 test deletions, smashpig code and a managed file change [05:10:44] unfortunately, the apache logs for civiproxy are less than helpful in these cases. and the general civi logs are full of everything. [05:10:55] yeah so we cant find the actual php error thats making the 500? [05:11:12] trying to see if i can. [05:11:47] lemme see if i can dig up the civi url that hte proxy calls to confirm that still works [05:14:31] 127.0.0.1 - - [08/Oct/2026:05:04:26 +0000] "GET /civicrm/ajax/api4/WMFContact/getDonorSummary?params=%7B%22checksum%22%3A%227e922cda4e5ebd92b263759477838d1f_1791435578_168%22%2C%22contact_id%22%3A52287939%7D HTTP/1.1" 500 364 "-" "-" [05:14:51] oh the configandlog [05:14:54] where is that [05:15:13] we'll want to reset the token for that contact_id i think. [05:15:46] ha. it's mine. [05:15:52] hah [05:16:15] i found some logs what is in here [05:16:57] /srv/org.wikimedia.civicrm/private/log i think [05:17:15] the one i lookde at in files hasnt had an update for 2 hours lemme look at that one [05:17:56] ok same [05:18:28] CiviCRM.7a880382d2e1d80611365ce1.log shows the consumer trying to build a link for my wm.o address that doesn't have a record. [05:18:33] which is correct. [05:18:45] but nothing related to my other one. [05:19:42] crazy suggestion, what if we just clear the cache files. [05:20:08] sounds good im at a loss [05:20:17] ok. i'll fire that off. [05:20:20] on the civi box? [05:20:31] yep, using civicrm_cache_clear [05:22:18] no luck there. [05:27:02] ok i just made an api call locally to civi for it and it returned my info [05:27:11] so makes me think the issue is on civiproxys end [05:27:19] does that have a cache to clear hah [05:27:47] interesting. [05:28:04] i'm not aware of any cache for it.... [07:53:43] Civi6.20 rc is cut - it includes a patch I've managed to get in for Melanie plus a nice improvment Lars got the core team to do which will allow us to set a site-default time out for Searchkits, overridable in the UI per search kit. Should be great. Also I merged a PR this week which addresses partial rollback on merge. I'll put together a version for deploy this am [08:02:59] (03PS1) 10Eileen: Temp fix for class loader issue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353003 [08:03:37] (03CR) 10Damilare Adedoyin: [C:03+2] Temp fix for class loader issue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353003 (owner: 10Eileen) [08:05:08] (03CR) 10CI reject: [V:04-1] Temp fix for class loader issue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353003 (owner: 10Eileen) [08:10:53] eileen: looks like the iframe extension path is not on CI [08:19:15] (03PS2) 10Eileen: Temp fix for class loader issue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353003 [08:25:13] damilare: yeah - && should be safer - ci is not hating it [08:25:31] (03CR) 10Damilare Adedoyin: [C:03+2] Temp fix for class loader issue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353003 (owner: 10Eileen) [08:28:22] (03PS1) 10Eileen: Temp fix for class loader issue [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353018 [08:28:49] (03CR) 10Eileen: [C:03+2] "merge to deploy" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353018 (owner: 10Eileen) [08:31:17] !log civicrm upgraded from 83683472 to 18cf9477 [08:31:18] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [08:37:35] (03PS1) 10Eileen: Temp short circuit [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353022 [08:38:29] (03CR) 10Damilare Adedoyin: [C:03+2] Temp short circuit [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353022 (owner: 10Eileen) [08:39:13] (03CR) 10Eileen: [V:03+2] Temp short circuit [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353022 (owner: 10Eileen) [08:41:16] (03PS1) 10Eileen: Further attempt at temp fix [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353023 [08:42:22] (03CR) 10Damilare Adedoyin: [C:03+2] Further attempt at temp fix [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353023 (owner: 10Eileen) [08:43:06] (03Merged) 10jenkins-bot: Further attempt at temp fix [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353023 (owner: 10Eileen) [08:51:15] (03PS1) 10Eileen: Still trying to temp fix [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353029 [08:51:28] argh Dami - I misread the file name - maybe that one [08:54:31] (03CR) 10Damilare Adedoyin: [C:03+2] Still trying to temp fix [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353029 (owner: 10Eileen) [08:55:16] (03Merged) 10jenkins-bot: Still trying to temp fix [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353029 (owner: 10Eileen) [09:39:12] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559 (10Eileenmcnaughton) 03NEW [09:39:56] FIRING: [22x] NodeDown: Node civi2002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [09:42:30] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12410393 (10Eileenmcnaughton) Here is the script I'm trying ` #!/bin/bash... [09:43:23] (03PS1) 10Eileen: Revert "Still trying to temp fix" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353065 [09:43:43] (03PS1) 10Eileen: Revert "Further attempt at temp fix" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353066 [09:44:02] (03PS1) 10Eileen: Revert "Temp short circuit" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353067 [09:44:56] RESOLVED: [22x] NodeDown: Node civi2002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [09:47:22] (03CR) 10Damilare Adedoyin: [C:03+2] Revert "Still trying to temp fix" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353065 (owner: 10Eileen) [09:47:29] (03CR) 10Damilare Adedoyin: [C:03+2] Revert "Further attempt at temp fix" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353066 (owner: 10Eileen) [09:47:39] (03CR) 10Damilare Adedoyin: [C:03+2] Revert "Temp short circuit" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353067 (owner: 10Eileen) [09:48:07] (03Merged) 10jenkins-bot: Revert "Still trying to temp fix" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353065 (owner: 10Eileen) [09:48:17] (03Merged) 10jenkins-bot: Revert "Further attempt at temp fix" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353066 (owner: 10Eileen) [09:48:19] (03Merged) 10jenkins-bot: Revert "Temp short circuit" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1353067 (owner: 10Eileen) [09:54:10] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12410433 (10Eileenmcnaughton) p:05Triage→03Unbreak! [10:15:45] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12410502 (10jgleeson) civi1002 queues have been paused to avoid failmails [10:43:48] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12410595 (10Eileenmcnaughton) So basically I worked with Claude to go through what other tables needed reverts & ultimately ALSO ran reverts on wmf-cv api3 Logging.revert log_conn_id=6ac75... [10:51:28] FIRING: RedisQueueSize: Redis Queue jobs_paypal is high: [5797] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [10:54:31] (03CR) 10CI reject: [V:04-1] build: Updating npm dependencies [extensions/FundraisingTranslateWorkflow] (REL1_43) - 10https://gerrit.wikimedia.org/r/1157359 (owner: 10Libraryupgrader) [11:21:28] FIRING: RedisQueueSize: Redis Queue jobs_paypal is Critical: [10942] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [11:27:56] FIRING: [26x] NodeDown: Node civi2002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [11:28:31] FIRING: HAProxyExporterDown: HAProxy on fransw2003.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [11:32:56] RESOLVED: [30x] NodeDown: Node civi2002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [11:33:31] RESOLVED: HAProxyExporterDown: HAProxy on fransw2003.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [11:41:28] FIRING: [2x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [2229] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [11:44:00] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12410925 (10Eileenmcnaughton) I hand edited some financial type labels back in - api v3 system flush seemed to work better on redis? No known issues now... [12:01:28] FIRING: [3x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [2813] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [12:06:28] FIRING: [3x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [2954] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [12:16:28] FIRING: [3x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [2954] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [12:17:57] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12411089 (10jgleeson) Queues are now back on - mostly ok apart from one recurring error `php 2026-10-08 12:17:03+0000 [debug] DB Error: syntax error SELECT id FROM WHERE entity_id = 29250... [12:21:28] RESOLVED: [3x] RedisQueueSize: Redis Queue contribution_tracking is Critical: [2954] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [12:21:28] FIRING: [4x] RedisQueueSize: Redis Queue contribution_tracking is high: [2954] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [12:26:28] RESOLVED: [3x] RedisQueueSize: Redis Queue jobs_paypal is high: [5228] - https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc - https://alerts.wikimedia.org/?q=alertname%3DRedisQueueSize [12:28:17] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12411122 (10Eileenmcnaughton) There is something wrong with these 2 fields causing the problem =they look 'ok' so I tried re-saving one "Major_Gifts_Engagement.Appeal",... [12:33:02] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12411137 (10Eileenmcnaughton) OK the trace was a problem with the custom table data - the value for `table_name` was lost for the Major Gifts Engagement group (48)- now fixed civicrm_value_m... [12:36:23] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12411142 (10Eileenmcnaughton) OK - there were 2 more - both extending activity MariaDB [civicrm]> update civicrm_custom_group SET table_name = 'civicrm_value_sms_consent_52' WHERE id = 52;... [12:38:17] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: civicrm.wmf.ALERT: UNHANDLED ERROR. Message was removed from queue `new-checksum-link` and sent to the damaged message table - https://phabricator.wikimedia.org/T440582 (10jgleeson) 03NEW [12:39:11] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12411171 (10Eileenmcnaughton) OK ^^ + a cache flush is looking better for NewChecksumLinkQueueConsumer job [13:40:58] ooh, looks like it's been a fun day so far [13:42:51] -_- [13:44:03] anything still broken, or did I have the good luck to come online just as the cleanup finished? :) [13:44:52] looking good atm [13:45:04] lots of log watching [13:45:10] nice [13:45:25] that logging.revert action is a life saver [13:45:33] right [13:46:17] and if things just get nuked, we should be able to restore a day-old db and salvage some stuff from queue logs [13:46:35] though the timing interdependence there would make that hairy [13:46:36] I'm still not 100% on why, when wmf-civicrm was disable and then reenabled, the custom fields were still inacitve [13:46:53] yeah, it really should have reactivated them [13:46:58] eileen: do you know why that never worked? [13:47:34] (03CR) 10Ejegg: [V:03+2] build: Updating npm dependencies [extensions/DonationInterface] (REL1_45) - 10https://gerrit.wikimedia.org/r/1338863 (owner: 10Libraryupgrader) [13:47:43] (03CR) 10Ejegg: [V:03+2] build: Updating npm dependencies [extensions/DonationInterface] (REL1_46) - 10https://gerrit.wikimedia.org/r/1338864 (owner: 10Libraryupgrader) [13:48:14] ejegg: this is one of those times when I like past me - makes up for all the "wtf - who wrote this " times [13:49:10] :) [13:49:15] Wmf-civicrm is kinda designed to not touch custom fields except on initial install - I had no expectation of it not creating havoc if disabled [13:49:27] right [13:49:53] we might be able to make it safer - although we have probably scared a generation of devs away from touching the 'disable' on it [13:50:06] ha [13:50:21] lol or just add a hook to abort disablement [13:50:38] "You don't want to do this" [13:51:39] here's hoping the CentralNotice deploy I need to do later today goes more smoothly! [13:52:11] well, on that note [13:52:15] 😅 🙏 [13:52:17] I need to go and pick my kids up [13:52:18] :D [13:53:55] (03PS5) 10Ejegg: Add lockTSE to banner choice cache options [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1347918 (https://phabricator.wikimedia.org/T439239) [13:54:17] Can anyone take a look at that last cache-related patch? ^^^ [13:54:38] ejegg: I think that looked good on tech-talk [13:54:55] I can +2 it [13:54:58] Thanks! [13:55:34] (03CR) 10Jgleeson: [C:03+2] Add lockTSE to banner choice cache options [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1347918 (https://phabricator.wikimedia.org/T439239) (owner: 10Ejegg) [13:56:05] ty! [13:56:40] Ah, looks like the next 'backport window' is in 6 hrs anyway. There's one currently happening but over in 4 min [13:57:07] laurabar: Deployments to the main wikis are coordinated here: https://wikitech.wikimedia.org/wiki/Deployments [13:57:33] (03Merged) 10jenkins-bot: Add lockTSE to banner choice cache options [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1347918 (https://phabricator.wikimedia.org/T439239) (owner: 10Ejegg) [14:00:24] the 'train' deployments send out all the new code merged to the master branch of mediawiki and its extensions, Tuesdays to test wikis and mediawiki.org, Wednesdays to more wikis, then Thursdays to the biggest wikis [14:00:30] https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys [14:00:50] But CentralNotice is an exception [14:01:37] We deploy batches of CentralNotice code during the 'backport windows' when you can deploy a small amount of code to all wikis at once [14:26:49] (03CR) 10Ejegg: [C:03+2] "Let's not squash the commits on merge to wmf_deploy next time, to minimize conflicts on the next merge." [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1333270 (owner: 10Jdlrobson) [14:28:00] (03PS1) 10Ejegg: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1353243 [14:38:20] 10Fundraising Payments token migration 2026, 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07payments-orchestration: Recurring token migration - Dlocal - https://phabricator.wikimedia.org/T434580#12411904 (10Damilare) [15:00:08] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1353243 (owner: 10Ejegg) [15:03:35] (03Merged) 10jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1353243 (owner: 10Ejegg) [15:24:05] (03PS2) 10Lars SG: Add Appeal afform [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1352151 (https://phabricator.wikimedia.org/T434201) [15:24:40] (03CR) 10Ejegg: [C:03+2] "Looks good, nice to have an example of how to add a toolbar button to a saved search display" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1352151 (https://phabricator.wikimedia.org/T434201) (owner: 10Lars SG) [15:33:18] (03Merged) 10jenkins-bot: Add Appeal afform [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1352151 (https://phabricator.wikimedia.org/T434201) (owner: 10Lars SG) [15:45:20] 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Wikimedia-CentralNotice-Administration, 13Patch-For-Review: Provide a way to display of a banner only to users who reach a certain number of edits - https://phabricator.wikimedia.org/T269758#12412490 (10SBurn... [15:51:29] 10Fundraising Payments token migration 2026, 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Recurring Token Migrate - Adyen CC - https://phabricator.wikimedia.org/T418759#12412558 (10Damilare) Gravy says they have enabled the RTAU on our account,... [16:15:46] 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Wikimedia-CentralNotice-Administration, 13Patch-For-Review: Provide a way to display of a banner only to users who reach a certain number of edits - https://phabricator.wikimedia.org/T269758#12412619 (10Ejegg... [16:20:18] 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Wikimedia-CentralNotice-Administration, 13Patch-For-Review: Provide a way to display of a banner only to users who reach a certain number of edits - https://phabricator.wikimedia.org/T269758#12412651 (10greg... [16:23:32] (03CR) 10Ejegg: [C:04-1] "The 'no results' template is optional here - let's leave it out and not add the extra i18n strings." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1352574 (https://phabricator.wikimedia.org/T440262) (owner: 10Wfan) [16:28:17] (03CR) 10Wfan: "Ok, This is the reason why choose lookup instead of combobox, so leave it blank if not found right?" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1352574 (https://phabricator.wikimedia.org/T440262) (owner: 10Wfan) [16:31:15] (03PS2) 10Wfan: Combowiki: Better UX for the country drop down [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1352574 (https://phabricator.wikimedia.org/T440262) [16:31:26] (03CR) 10Ejegg: "looks like combobox allows freetext: "If users can only select a predefined option, use Select (for a short list of options) or Lookup (fo" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1352574 (https://phabricator.wikimedia.org/T440262) (owner: 10Wfan) [17:31:02] fr-tech, speaking of Claude, https://www.youtube.com/watch?v=3TNpOD6bov8 :) [17:35:28] it's actually an awesome song [17:36:09] bye for no [17:36:10] w [17:40:48] 06Fundraising-Backlog, 10FR-Donor-portal: Donor Portal - have better error message when civiproxy isn't working - https://phabricator.wikimedia.org/T440634 (10Cstone) 03NEW [17:42:35] Reviewed by: BOT-001 (1,363 reviewers, 0 humans) [17:57:59] I don't like being called out by name [18:04:09] (03PS19) 10Damilare Adedoyin: Add Provider for Gravy Adyen direct paybybank connection for ACH/SEPA transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1344370 (https://phabricator.wikimedia.org/T438723) [18:17:18] (03PS21) 10Damilare Adedoyin: Add mapper to make allowance for the transactions through the new connection for Adyen ACH/SEPA. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1344370 (https://phabricator.wikimedia.org/T438723) [18:19:04] ejegg: I removed the custom provider and added some notes in the commit message for this patch ^ [18:19:18] 06Fundraising-Backlog, 10FR-Donor-portal: Donor Portal - have better error message when civiproxy isn't working - https://phabricator.wikimedia.org/T440634#12413203 (10SBurnett-WMF) p:05Triage→03Low [18:21:02] thanks damilare, looking [18:21:23] thanks! [18:22:33] (03Abandoned) 10Damilare Adedoyin: Add fields for payments through Gravy paybybank connector. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1344755 (https://phabricator.wikimedia.org/T438723) (owner: 10Damilare Adedoyin) [18:27:50] damilare: are you able to test this at all in the sandbox? [18:28:17] yes I ran some tests with the full patch [18:28:34] basically created a recurring on that connection [18:28:41] would you mind sharing a token I could add to my local? [18:28:45] charged the recurring through that connection [18:28:47] sure [18:29:38] 8a11e2ec-b716-4fe5-85b7-5e3f6e9eee4d [18:29:42] thanks! [18:30:42] the payment_service_id is 279f10eb-b24d-4482-8f12-5b97e1d2fcc4 [18:31:25] ok, cool [18:36:35] damilare: i forget, was it possible to create trustly recurrings in Gravy sandbox? [18:36:42] I'm failing to do it right now [18:37:16] you might need to turn on your proxy and it could take a while to capture [18:37:35] but yh it should be possible [18:37:37] Do you remember which bank I need to choose? [18:38:01] 06Fundraising-Backlog, 10FR-Donor-portal: Donor Portal - have better error message when civiproxy isn't working - https://phabricator.wikimedia.org/T440634#12413309 (10SBurnett-WMF) @ppenloglou - this is a low priority. We had an incident today that showed us that some tweaking on this feature could improve th... [18:38:50] its demo bank I use, but they change it often [18:38:54] let me find that doc [18:39:08] ah thank you. I was searching for 'test' [18:39:38] yep, working [18:39:58] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Deploy opps fall out - https://phabricator.wikimedia.org/T440559#12413313 (10SBurnett-WMF) [18:40:38] https://amer.developers.trustly.com/integrate/api-fundamentals/testing [18:40:53] bookmarked, thanks again damilare [18:40:59] np! [18:42:20] 10Fundraising Payments token migration 2026, 06Fundraising-Backlog: Check on adyen ideal Contract not found recurring failures - https://phabricator.wikimedia.org/T440419#12413318 (10SBurnett-WMF) @Damilare - Could you see if this belongs on the migration board? [18:42:50] 10Fundraising Payments token migration 2026, 06Fundraising-Backlog: Check on adyen ideal Contract not found recurring failures - https://phabricator.wikimedia.org/T440419#12413322 (10SBurnett-WMF) p:05Triage→03Medium [18:49:22] looking Stacy [18:53:08] ok damilare so that patch lets us map the new connector in notifications from their side [18:53:32] yes exactly [18:53:43] and then it looks like we're already sending payment_service_id in to the recurring payments when we have that stored in contribution_recur_smashpig [18:54:11] yes correct [18:54:31] so what's the missing piece with ach vs trustly? [18:56:11] anyway, that particular patch seems safe and well tested [18:56:17] currently we map Bank Transfer: ACH -> bt in civi and then to trustly in the Gravy/PaymentMethod.php [18:56:33] ah got it [18:56:50] so its just something to keep in mind when/if Trustly changes behaviour [18:57:37] (03CR) 10Ejegg: [C:03+2] "Looks good, and I'm very glad you were able to do it without needing the new top-level method category." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1344370 (https://phabricator.wikimedia.org/T438723) (owner: 10Damilare Adedoyin) [18:57:53] thanks ejegg [18:58:19] (03Merged) 10jenkins-bot: Add mapper to make allowance for the transactions through the new connection for Adyen ACH/SEPA. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1344370 (https://phabricator.wikimedia.org/T438723) (owner: 10Damilare Adedoyin) [18:58:31] I'd deploy that on the listener and test out Christine's stored method on civi [19:00:29] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1353369 [19:06:15] (03CR) 10Damilare Adedoyin: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1353369 (owner: 10Damilare Adedoyin) [19:06:38] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1353369 (owner: 10Damilare Adedoyin) [19:14:07] log SmashPig upgraded from 31c021ba to 7e3426cf [19:14:12] !log SmashPig upgraded from 31c021ba to 7e3426cf [19:14:14] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [19:31:22] 10Fundraising analytics stack, 10fundraising-tech-ops, 10FR-Tech-Analytics: create directory for fr-analytics to read files exported from dagster - https://phabricator.wikimedia.org/T440652 (10AStein-WMF) 03NEW [19:51:28] 10Fundraising Payments token migration 2026, 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Recurring Token Migrate - Adyen Ideal - https://phabricator.wikimedia.org/T432725#12413705 (10Damilare) Testing with a donor account, [[ https://wikimedia.... [19:52:47] 10Fundraising Payments token migration 2026, 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 07fr-current-sprint, 07payments-orchestration: Recurring Token Migrate - Adyen ACH - https://phabricator.wikimedia.org/T418762#12413710 (10Damilare) Did a test with a donor account, [[ https://wikimedia... [19:54:31] (03CR) 10Ejegg: "It's great to see that we have maintained pretty good compatibility between the two SmashPig-level wrappers. I'd prefer an abstract getGat" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1344317 (https://phabricator.wikimedia.org/T438879) (owner: 10Damilare Adedoyin) [19:58:35] (03CR) 10Ejegg: "It would be nice to minimize the differences even more. I'll add a quick SmashPig patch to make Adyen responses come back with getBackendP" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1344317 (https://phabricator.wikimedia.org/T438879) (owner: 10Damilare Adedoyin) [20:01:07] (03PS1) 10Pwangai: Seed the contribution-tracking test sequence with an integer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353394 (https://phabricator.wikimedia.org/T434024) [20:03:31] oh boy, central notice deploy time! [20:05:32] (03CR) 10Pwangai: "When running tests with paratest with multiple databases, PHPUnit outputs 2 failures:" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353394 (https://phabricator.wikimedia.org/T434024) (owner: 10Pwangai) [20:13:44] (03PS1) 10Ejegg: Set backendProcessor to 'adyen' for all Adyen responses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1353402 [20:14:09] (03CR) 10CI reject: [V:04-1] Set backendProcessor to 'adyen' for all Adyen responses [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1353402 (owner: 10Ejegg) [20:18:10] 10Fundraising analytics stack, 10fundraising-tech-ops, 10FR-Tech-Analytics: create directory for fr-analytics to read files exported from dagster - https://phabricator.wikimedia.org/T440652#12413800 (10AStein-WMF) 05Open→03Resolved [20:29:47] 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Remove old audience framework segment & status - https://phabricator.wikimedia.org/T432597#12413833 (10NNgu-WMF) I see - it's more that other Civi work on the list was a higher priority for us. We're not currently usin... [20:33:26] ejegg: problems with the deploy or you're about to do it? [20:41:22] about to, just waiting for the other folks with a deploy scheduled in this slot [20:54:27] 06Fundraising-Backlog, 07Fundraising-ComboWiki-Dependency, 06Fundraising-Tech-Roadmap, 07Epic, 07fr-combowiki: Combowiki A/B testing - https://phabricator.wikimedia.org/T427983#12413914 (10AKanji-WMF) @Sjones-ctr I don't know if we'd ever use test kitchen this way? [21:00:12] 10fundraising-tech-ops: Fundraising access request for Kadeem Khan - https://phabricator.wikimedia.org/T440658 (10Khantstop) 03NEW [21:09:24] 14Fundraising Sprint: Shelf Aware, 06Fundraising-Backlog, 10FR-email-preferences: EPC should be mobile responsive - https://phabricator.wikimedia.org/T439622#12413966 (10dmiranda) p:05Triage→03Medium [21:19:20] (03PS1) 10Laurabarluzzi: ComboWiki: show donor fields per country and payment method configurations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1353438 (https://phabricator.wikimedia.org/T439938) [21:41:41] (03PS1) 10Ejegg: Fix missed merge conflict [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1353446 [21:43:26] (03CR) 10Ejegg: [C:03+2] Fix missed merge conflict [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1353446 (owner: 10Ejegg) [21:44:14] 06Fundraising-Backlog, 10FR-email-preferences: Email Preference center - handle 500 errors better - https://phabricator.wikimedia.org/T440660 (10Cstone) 03NEW [21:45:25] (03Merged) 10jenkins-bot: Fix missed merge conflict [extensions/CentralNotice] (wmf_deploy) - 10https://gerrit.wikimedia.org/r/1353446 (owner: 10Ejegg) [21:56:30] (03PS5) 10Wfan: limit the line to max 30 for all failmail [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1346137 (https://phabricator.wikimedia.org/T438355) [22:16:18] (03CR) 10Dmiranda: [C:03+2] "Thanks @pwangai@wikimedia.org! LGTM" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353394 (https://phabricator.wikimedia.org/T434024) (owner: 10Pwangai) [22:25:00] (03Merged) 10jenkins-bot: Seed the contribution-tracking test sequence with an integer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353394 (https://phabricator.wikimedia.org/T434024) (owner: 10Pwangai) [22:33:14] (03PS1) 10SarthakSingh2904: Filter out forbidden countries from country drop down [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1353470 (https://phabricator.wikimedia.org/T440661) [22:57:22] (03PS1) 10SarthakSingh2904: combowiki: reset payment method when not supported by frequency [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1353484 (https://phabricator.wikimedia.org/T440640) [23:08:10] (03PS1) 10Lars SG: Update donor details iframe per DR requests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353488 (https://phabricator.wikimedia.org/T433866) [23:09:21] (03PS1) 10Lars SG: Update donor details iframe per DR requests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353489 (https://phabricator.wikimedia.org/T433866) [23:09:58] (03Abandoned) 10Lars SG: Update donor details iframe per DR requests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1353488 (https://phabricator.wikimedia.org/T433866) (owner: 10Lars SG) [23:32:02] 06Fundraising-Backlog, 10FR-Donor-portal: Donor Portal - have better error message when civiproxy isn't working - https://phabricator.wikimedia.org/T440634#12414297 (10ppenloglou) Thank you @SBurnett-WMF for the cc, I'll keep an eye out on this one. @Cstone If there's anything I can help with let me know! [23:41:08] 06Fundraising-Backlog, 10fundraising-tech-ops: Create process to actively monitor active mysql grants vs what the grant scripts hold - https://phabricator.wikimedia.org/T247371#12414299 (10Dwisehaupt) a:03Dwisehaupt New collector (`/usr/lib/prometheus/collector/mariadb_grants`) rolled out to hosts where we h... [23:50:03] 14Fundraising Sprint: Shelf Aware, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Adyen audit ignores PaymentCost rows - https://phabricator.wikimedia.org/T440672 (10dmiranda) 03NEW [23:54:22] (03PS1) 10Dmiranda: Adyen audit: book PaymentCost rows as fees [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1353506 (https://phabricator.wikimedia.org/T440672)