[00:01:11] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail, 10FR-WMF-Audit: Failmail alert paypal_audit - https://phabricator.wikimedia.org/T437215#12305856 (10Eileenmcnaughton) Sign - no I failed to log in I thought I got into paypal once - maybe I imagined it - @Cstone  or @Ejegg ?
[00:47:05] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Tracking donor strategies in Civi - https://phabricator.wikimedia.org/T437526#12305897 (10SBurnett-WMF) a:03SBurnett-WMF @DBautista-WMF Could you please attach a sample of google docs and spreadsheets you use without any PII? It would be helpful to see...
[01:13:28] <wikibugs>	 (03PS1) 10Ejegg: Stop logging changes to model score table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338347
[01:24:53] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] Stop logging changes to model score table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338347 (owner: 10Ejegg)
[01:46:06] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-failmail, 10FR-WMF-Audit: Failmail alert paypal_audit - https://phabricator.wikimedia.org/T437215#12306033 (10Ejegg) I searched in the PayPal console at https://www.paypal.com/mep/unifiedtransactions for the transaction ID and didn't find anythin...
[01:46:22] <wikibugs>	 (03Merged) 10jenkins-bot: Stop logging changes to model score table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338347 (owner: 10Ejegg)
[02:04:24] <jinxer-wm>	 FIRING: NodeTextfileScrapeError: Prometheus node collector scrape error (instance frpm1002.frack.eqiad.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileScrapeError
[02:09:24] <jinxer-wm>	 RESOLVED: NodeTextfileScrapeError: Prometheus node collector scrape error (instance frpm1002.frack.eqiad.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileScrapeError
[02:10:10] <wikibugs>	 (03PS4) 10Cstone: Better error display and reload whole app when trying again [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140)
[02:10:17] <wikibugs>	 (03CR) 10Cstone: Better error display and reload whole app when trying again (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140) (owner: 10Cstone)
[02:10:51] <wikibugs>	 (03PS4) 10Lars SG: Add single page display of all Payment Attempt fields for one id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338070 (https://phabricator.wikimedia.org/T437380)
[02:11:00] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] "This is great!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338070 (https://phabricator.wikimedia.org/T437380) (owner: 10Lars SG)
[02:11:58] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Better error display and reload whole app when trying again [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140) (owner: 10Cstone)
[02:12:39] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki Testing: error handling check - https://phabricator.wikimedia.org/T437542 (10Cstone) 03NEW
[02:15:39] <wikibugs>	 (03PS5) 10Cstone: Better error display and reload whole app when trying again [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140)
[02:16:43] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: Combowiki: variant? where paypal, venmo, apple pay, and google pay buttons are loaded to start - https://phabricator.wikimedia.org/T437335#12306095 (10Cstone) a:03Cstone
[02:19:34] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Better error display and reload whole app when trying again [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140) (owner: 10Cstone)
[02:22:39] <wikibugs>	 (03PS6) 10Cstone: Better error display and reload whole app when trying again [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140)
[02:24:28] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Better error display and reload whole app when trying again [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140) (owner: 10Cstone)
[02:30:13] <wikibugs>	 (03Merged) 10jenkins-bot: Add single page display of all Payment Attempt fields for one id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338070 (https://phabricator.wikimedia.org/T437380) (owner: 10Lars SG)
[02:41:49] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Remove alerts from recurring convert errors - https://phabricator.wikimedia.org/T437543 (10Cstone) 03NEW
[02:43:05] <wikibugs>	 (03PS7) 10Cstone: Better error display and reload whole app when trying again [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140)
[02:43:12] <cstone>	 7th times the charm??
[02:43:42] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Remove alerts from recurring convert errors - https://phabricator.wikimedia.org/T437543#12306139 (10Cstone) p:05Triage→03High
[02:43:59] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: fix the ty page params from combo wiki - https://phabricator.wikimedia.org/T436165#12306140 (10Cstone) p:05Triage→03High
[02:44:37] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki, 13Patch-For-Review: Error handling in combowiki - https://phabricator.wikimedia.org/T433140#12306141 (10Cstone) p:05Triage→03High
[02:44:42] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: filter payment method by supported country - https://phabricator.wikimedia.org/T437508#12306142 (10Cstone) p:05Triage→03High
[02:45:03] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 10MediaWiki-extensions-DonationInterface, 07fr-combowiki, 13Patch-For-Review: Combowiki: move over smsOptin variant - https://phabricator.wikimedia.org/T437075#12306143 (10Cstone) p:05Triage→03High
[02:45:19] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Traffic analysis for donate wiki - https://phabricator.wikimedia.org/T436168#12306144 (10Cstone) p:05Triage→03Low
[02:45:28] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Add display for blocked countries - https://phabricator.wikimedia.org/T435681#12306145 (10Cstone) p:05Triage→03High
[02:47:09] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: Strategies to speed up combo wiki initial load time - https://phabricator.wikimedia.org/T433012#12306148 (10Cstone) p:05Triage→03Medium
[02:47:14] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07Fundraising-ComboWiki-Dependency, 07fr-combowiki, 07fr-current-sprint: Use DonationInterface config in Combowiki - https://phabricator.wikimedia.org/T427982#12306150 (10Cstone) p:05Triage→03High
[02:47:22] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: Consider improving the wiring of the logger in ComboWiki.php and its classes - https://phabricator.wikimedia.org/T434831#12306151 (10Cstone) p:05Triage→03Medium
[02:49:15] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Clarify usage of DonationDetails dataSources and improve - https://phabricator.wikimedia.org/T434835#12306152 (10Cstone) p:05Triage→03Medium
[02:49:45] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Reduce to the minimum all gateway dependencies in ComboWiki.php - https://phabricator.wikimedia.org/T434827#12306154 (10Cstone) p:05Triage→03Medium
[02:49:47] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Optional: consider to have a stricter enforcement of value types in DonationDetails() - https://phabricator.wikimedia.org/T434815#12306156 (10Cstone) p:05Triage→03Medium
[02:50:41] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Clarify the types of values of fieldNames passed from DataIntegration() down the chain - https://phabricator.wikimedia.org/T434813#12306160 (10Cstone) p:05Triage→03Medium
[02:50:57] <ejegg>	 failstorm...
[02:51:10] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Update DonationInterface\ComboWiki\DataNormalizer::normalizeCountry - https://phabricator.wikimedia.org/T434807#12306164 (10Cstone) p:05Triage→03Medium
[02:51:21] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: Update the list of ::fieldNames in data fetched from request and session - https://phabricator.wikimedia.org/T434803#12306166 (10Cstone) p:05Triage→03Medium
[02:51:39] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Define fieldNames allowed in session - https://phabricator.wikimedia.org/T434804#12306170 (10Cstone) p:05Triage→03Medium
[02:52:00] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: how do we display endowment - https://phabricator.wikimedia.org/T434233#12306171 (10Cstone) p:05Triage→03High
[03:23:37] <jinxer-wm>	 FIRING: GnmiInterfaceCountersDrop: ...
[03:23:37] <jinxer-wm>	 fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=fasw1-f5a-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[03:47:00] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 3 others: Automate some Direct Mail white mail coding - https://phabricator.wikimedia.org/T386031#12306235 (10Lars) @MDemosWMF In discussion with Rosie, I've done the following:  Ne...
[03:54:54] <wikibugs>	 (03Abandoned) 10Lars SG: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338311 (https://phabricator.wikimedia.org/T437527) (owner: 10Lars SG)
[04:28:56] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Donate wiki opt in flow - https://phabricator.wikimedia.org/T435683#12306286 (10Cstone) p:05Triage→03High
[04:28:57] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Handle major gift amounts from Donate wiki - https://phabricator.wikimedia.org/T435682#12306288 (10Cstone) p:05Triage→03High
[04:29:00] <wikibugs>	 06Fundraising-Backlog, 07Fundraising-ComboWiki-Dependency, 07fr-combowiki: How would we deploy combo wiki - https://phabricator.wikimedia.org/T427985#12306289 (10Cstone) p:05Triage→03Low
[04:29:05] <wikibugs>	 06Fundraising-Backlog, 07Fundraising-ComboWiki-Dependency, 07fr-combowiki: How would we deploy combo wiki - https://phabricator.wikimedia.org/T427985#12306291 (10Cstone) p:05Low→03Medium
[04:29:09] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: Combowiki: variant? where paypal, venmo, apple pay, and google pay buttons are loaded to start - https://phabricator.wikimedia.org/T437335#12306293 (10Cstone) p:05Triage→03High
[04:29:11] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Tax Messaging from Donate Wiki - https://phabricator.wikimedia.org/T434020#12306295 (10Cstone) p:05Triage→03High
[04:29:15] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Currency selector? How to handle changing currency - https://phabricator.wikimedia.org/T436164#12306297 (10Cstone) p:05Triage→03Medium
[04:29:19] <wikibugs>	 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07fr-combowiki: Minimize use of 'Combowiki' in code - https://phabricator.wikimedia.org/T437062#12306298 (10Cstone) p:05Triage→03Low
[04:29:25] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Use uselang= for language in the URL - https://phabricator.wikimedia.org/T437483#12306300 (10Cstone) p:05Triage→03High
[04:29:29] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki, 05MW-1.47-notes (1.47.0-wmf.19; 2026-09-08): Variants in combowiki - https://phabricator.wikimedia.org/T433141#12306303 (10Cstone) p:05Triage→03High
[04:29:34] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki Testing: error handling check - https://phabricator.wikimedia.org/T437542#12306304 (10Cstone) p:05Triage→03High
[04:29:41] <cstone>	 oik im done with the spam
[06:00:11] <jinxer-wm>	 FIRING: NodeDown: Node fransw1003 is down.  - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[06:01:28] <jinxer-wm>	 FIRING: OOM: OOM killer active on fransw1003.frack.eqiad.wmnet:9100   - https://alerts.wikimedia.org/?q=alertname%3DOOM
[06:05:11] <jinxer-wm>	 RESOLVED: NodeDown: Node fransw1003 is down.  - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[06:06:28] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on fransw1003.frack.eqiad.wmnet:9100   - https://alerts.wikimedia.org/?q=alertname%3DOOM
[07:23:37] <jinxer-wm>	 FIRING: GnmiInterfaceCountersDrop: ...
[07:23:37] <jinxer-wm>	 fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=fasw1-f5a-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[10:44:06] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: Strategies to speed up combo wiki initial load time - https://phabricator.wikimedia.org/T433012#12307110 (10Laurabarluzzi) Some small things I'm looking into is about the country blob we stored in <script> and is re-parsed, re-rea...
[11:23:38] <jinxer-wm>	 FIRING: GnmiInterfaceCountersDrop: ...
[11:23:38] <jinxer-wm>	 fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=fasw1-f5a-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[12:48:13] <wikibugs>	 06Wikimedia-Fundraising, 10Wikimedia-Portals: Portal fundraising banners shown in incorrect countries - https://phabricator.wikimedia.org/T437587 (10Pcoombe) 03NEW
[12:51:21] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Stripe Major Gifts No payment type for message - https://phabricator.wikimedia.org/T436887#12307811 (10Damilare) @Ejegg looks like the test failure was caused by stale triggers on the contribution table...
[13:07:58] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Stripe Major Gifts No payment type for message - https://phabricator.wikimedia.org/T436887#12307918 (10Ejegg) Oh right, it looked like we did want the individual donation after all, so I updated the tes...
[13:08:58] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 05MW-1.47-notes (1.47.0-wmf.20; 2026-09-15): Countries falling through gateway chooser ending up with paypal_ec - https://phabricator.wikimedia.org/T435554#12307930 (10Ejegg) The new settings are deployed.
[13:53:47] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Stripe Major Gifts No payment type for message - https://phabricator.wikimedia.org/T436887#12308098 (10Damilare)
[14:13:38] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2728Q1 maintenance window - https://phabricator.wikimedia.org/T436725#12308180 (10AKanji-WMF)
[14:22:22] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: RefundQueue consumer failure: "Parameter "where" is required..." - https://phabricator.wikimedia.org/T437602 (10Damilare) 03NEW
[14:49:50] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Develop process to prevent direct mail being sent to long-deleted cids - https://phabricator.wikimedia.org/T406771#12308488 (10Lars) @Elbar53 @MDemosWMF Can you confirm if you had any issues or not...
[14:58:23] <jinxer-wm>	 RESOLVED: GnmiInterfaceCountersDrop: ...
[14:58:23] <jinxer-wm>	 fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=fasw1-f5a-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[15:04:35] <wikibugs>	 (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1338958
[15:04:38] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1338958 (owner: 10Ejegg)
[15:05:40] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1338958 (owner: 10Ejegg)
[15:08:32] <wikibugs>	 (03PS1) 10Lars SG: Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031)
[15:08:36] <wikibugs>	 06Wikimedia-Fundraising, 10Wikimedia-Portals: Portal fundraising banners shown in incorrect countries - https://phabricator.wikimedia.org/T437587#12308575 (10Pcoombe) I checked past banners, and they were all correctly using either `indexOf() > -1` or just `includes()`, so this wouldn't have been a problem. It...
[15:10:10] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[15:10:20] <wikibugs>	 (03PS2) 10Lars SG: Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031)
[15:11:12] <larssandergreen>	 ejegg: I'm hoping to get ^^^ merged before we do the trigger updates for the log table exclusion, since it will have a small trigger update as well
[15:18:16] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Develop process to prevent direct mail being sent to long-deleted cids - https://phabricator.wikimedia.org/T406771#12308642 (10Elbar53) I haven't noticed any deleted contacts. Sometimes the CID is...
[15:20:11] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Develop process to prevent direct mail being sent to long-deleted cids - https://phabricator.wikimedia.org/T406771#12308682 (10Lars) @Elbar53 So there were cases since Aug 10 where the cid was inco...
[15:24:59] <ejegg>	 looking larssandergreen 
[15:26:06] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Develop process to prevent direct mail being sent to long-deleted cids - https://phabricator.wikimedia.org/T406771#12308741 (10Elbar53) I only knew it was incorrect because I needed to move it to t...
[15:31:41] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[15:36:17] <ejegg>	 also, thank you for the reminder before I went and deployed that drop table without trigger updates :P
[15:36:31] <ejegg>	 hmm, hwat's that V-1?
[15:37:01] <ejegg>	 (saw the typo, was going to correct it, decided it sounded like an old british constable so left it)
[15:37:34] <ejegg>	 oh, just needs some test updates larssandergreen 
[15:37:48] <ejegg>	 15:31:36 -    'direct_mail_appeal' => 'Spontaneous Donation'
[15:37:49] <ejegg>	 15:31:36 +    'direct_mail_appeal' => 'spontaneous'
[15:38:02] <ejegg>	 looks like mostly SmashPigTest
[15:38:10] <ejegg>	 huh, shouldn't actually care about that field
[15:38:19] <ejegg>	 I'd just unset it for the comparison
[15:40:25] <wikibugs>	 (03CR) 10Ejegg: Add MG Engagement appeal custom field (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[15:40:58] <ejegg>	 want me to fix that test larssandergreen ?
[15:45:23] <wikibugs>	 (03PS1) 10Ejegg: SmashPigTest: Don't examine fields with default values [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338976
[15:45:30] <ejegg>	 larssandergreen: want to rebase over ^^^
[15:45:31] <ejegg>	 ?
[15:52:23] <wikibugs>	 (03PS3) 10Lars SG: Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031)
[15:53:33] <larssandergreen>	 thanks ejegg, will do
[15:53:34] <wikibugs>	 (03PS1) 10Ejegg: Fix updating cancelled to chargeback [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338980 (https://phabricator.wikimedia.org/T437602)
[15:54:46] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: RefundQueue consumer failure: "Parameter "where" is required..." - https://phabricator.wikimedia.org/T437602#12308935 (10Ejegg) Oops, I missed this but while reviewing Eileen's patch. Have just uploaded a patch to add the missing 'wh...
[15:55:47] <wikibugs>	 (03PS4) 10Lars SG: Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031)
[16:03:33] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] SmashPigTest: Don't examine fields with default values [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338976 (owner: 10Ejegg)
[16:05:09] <wikibugs>	 (03CR) 10Damilare Adedoyin: [C:03+2] Fix updating cancelled to chargeback [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338980 (https://phabricator.wikimedia.org/T437602) (owner: 10Ejegg)
[16:05:25] <wikibugs>	 (03CR) 10Wfan: [C:03+2] Fix updating cancelled to chargeback [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338980 (https://phabricator.wikimedia.org/T437602) (owner: 10Ejegg)
[16:06:12] <ejegg>	 thanks for the CR, folks!
[16:06:20] <wikibugs>	 (03PS1) 10Lars SG: Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338989 (https://phabricator.wikimedia.org/T386031)
[16:07:47] <wikibugs>	 (03PS5) 10Lars SG: Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031)
[16:08:52] <wikibugs>	 (03PS6) 10Lars SG: Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031)
[16:09:20] <wikibugs>	 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Remove alerts from recurring convert errors - https://phabricator.wikimedia.org/T437543#12308993 (10AnnWF) a:03AnnWF
[16:09:25] <wikibugs>	 (03Abandoned) 10Lars SG: Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338989 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[16:09:44] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki: Combowiki: Remove alerts from recurring convert errors - https://phabricator.wikimedia.org/T437543#12308995 (10AnnWF)
[16:09:55] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] "Hooray, less wrong!" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1338258 (owner: 10Lars SG)
[16:12:09] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki, 07good first task: Combowiki: Remove alerts from recurring convert errors - https://phabricator.wikimedia.org/T437543#12309002 (10AnnWF) Thank you for tagging this task with #good_first_task for Wikimedia newcomers!  Newcomers o...
[16:14:34] <wikibugs>	 (03PS2) 10Eileen: Additional RecurDonationMessage helpers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1337670 (https://phabricator.wikimedia.org/T426098)
[16:14:36] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Additional RecurDonationMessage helpers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1337670 (https://phabricator.wikimedia.org/T426098) (owner: 10Eileen)
[16:17:20] <wikibugs>	 (03Merged) 10jenkins-bot: [NFC] Removing old, incorrect comment [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1338258 (owner: 10Lars SG)
[16:20:44] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 07fr-combowiki, 07good first task: Combowiki: Remove alerts from recurring convert errors - https://phabricator.wikimedia.org/T437543#12309032 (10AnnWF) a:05AnnWF→03None
[16:28:49] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Develop process to prevent direct mail being sent to long-deleted cids - https://phabricator.wikimedia.org/T406771#12309062 (10Lars) Great, that's what we want to hear @Elbar53  I've re-enabled our...
[16:29:51] <wikibugs>	 (03Merged) 10jenkins-bot: SmashPigTest: Don't examine fields with default values [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338976 (owner: 10Ejegg)
[16:29:52] <wikibugs>	 (03Merged) 10jenkins-bot: Fix updating cancelled to chargeback [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338980 (https://phabricator.wikimedia.org/T437602) (owner: 10Ejegg)
[16:33:21] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: RefundQueue consumer failure: "Parameter "where" is required..." - https://phabricator.wikimedia.org/T437602#12309103 (10Laurabarluzzi) a:03Ejegg
[16:40:40] <wikibugs>	 (03Merged) 10jenkins-bot: Additional RecurDonationMessage helpers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1337670 (https://phabricator.wikimedia.org/T426098) (owner: 10Eileen)
[16:51:39] <wikibugs>	 06Wikimedia-Fundraising, 10Wikimedia-Portals: Portal fundraising banners shown in incorrect countries - https://phabricator.wikimedia.org/T437587#12309226 (10KMorrow-WMF) Hey @Pcoombe, here's [[ https://metabase.frdev.wikimedia.org/question/3993-fy2627-portal-september-donations-outside-of-en6c | a Metabase ta...
[16:55:12] <wikibugs>	 06Fundraising-Backlog, 07FR-Audience-Dev: Investigate conditional Google's script addition for Google Ads Grant (fundraising) - https://phabricator.wikimedia.org/T374079#12309242 (10Ejegg) @Lars mentioned today that there are ways of doing conversion tracking without adding client-side code. It looks like we j...
[16:55:25] <ejegg>	 larssandergreen: want to add any more info you have to ^^^ ?
[16:56:02] <ejegg>	 gonna grab some lunch. I can help with trigger updates in a bit
[16:56:21] <larssandergreen>	 ejegg: I think that covers it, looks simple enough to me.
[17:09:11] <jinxer-wm>	 FIRING: NodeDown: Node frqueue2004 is down.  - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frqueue2004 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[17:14:11] <jinxer-wm>	 FIRING: [7x] NodeDown: Node civi2002 is down.   - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[17:19:11] <jinxer-wm>	 RESOLVED: [7x] NodeDown: Node civi2002 is down.   - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[17:34:29] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Tracking donor strategies in Civi - https://phabricator.wikimedia.org/T437526#12309339 (10SBurnett-WMF) @DBautista-WMF - not needed before Q3, and for scoping, she will wait until the gift officers are back from travel to obtain the specific asks.
[17:38:48] <ejegg>	 hmm larssandergreen I think we might need to do the trigger update in two steps anyway
[17:39:07] <ejegg>	 I need to drop triggers on model scores before I run the upgrader script for my patch
[17:39:30] <ejegg>	 then we need to run the upgrader for yours before we have the new triggers added to the script
[17:40:05] <ejegg>	 Jeff_Green or dwisehaupt is one of you available to run some trigger scripts?
[17:40:08] <larssandergreen>	 my upgrader is a no-op on prod
[17:40:20] <ejegg>	 oh i see larssandergreen - what is the trigger change then?
[17:40:29] <larssandergreen>	 and doesn't change triggers either
[17:40:58] <larssandergreen>	 it's from the added field in the mgd, just need to run wmf-cv flush before generating triggers
[17:41:10] <ejegg>	 oh did i misunderstand your comment earlier?
[17:41:11] <ejegg>	  < larssandergreen> ejegg: I'm hoping to get ^^^ merged before we do the trigger updates for the log table exclusion,
[17:41:50] <larssandergreen>	 i just wanted to get the patch merged so that mgd adds the custom field so it gets included in the trigger update
[17:42:10] <ejegg>	 ok
[17:42:22] <larssandergreen>	 it should who up on the civicrm_value_mg_engagement table
[17:42:27] <ejegg>	 still i need to drop the other triggers before dropping their destination table though
[17:42:45] <larssandergreen>	 for sure, trigger update, then run upgrader after
[17:43:07] <ejegg>	 lemme see if I can do it on staging
[17:43:35] <ejegg>	 oh but staging might not be all that current with user-added fields
[17:43:47] <larssandergreen>	 this one needs the +2 still and deploy: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1338962
[17:44:09] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[17:44:16] <larssandergreen>	 yeah, has to be done on prod
[17:45:19] <ejegg>	 ohhh right but we CAN do the mgd reconcile to get your field before we do the upgrader
[17:45:22] <ejegg>	 now i get it
[17:45:52] <larssandergreen>	 exactly
[17:48:31] <Jeff_Green>	 ejegg: I'm around, but it looks like you don't need to adjust triggers after all?
[17:50:34] <ejegg>	 Jeff_Green: Thanks! we will, but it'll be the standard one-step way, no need to do a preliminary drop step
[17:51:32] <Jeff_Green>	 ok
[17:54:22] <wikibugs>	 (03PS8) 10Damilare Adedoyin: Remove duplicated lines between adapters Apple pay api classes [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1337640
[17:55:42] <wikibugs>	 (03PS9) 10Damilare Adedoyin: Remove duplicated lines between adapters Apple pay api classes [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1337640
[17:56:21] <wikibugs>	 (03CR) 10Damilare Adedoyin: "Thanks for the CR ejegg, good call to use namespaces for simplicity, done!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1337640 (owner: 10Damilare Adedoyin)
[17:59:33] <wikibugs>	 (03CR) 10Laurabarluzzi: Better error display and reload whole app when trying again (033 comments) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140) (owner: 10Cstone)
[18:02:31] <wikibugs>	 (03CR) 10Ejegg: [V:03+2 C:03+2] Add MG Engagement appeal custom field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338962 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[18:03:43] <wikibugs>	 (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1339038
[18:03:45] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1339038 (owner: 10Ejegg)
[18:07:41] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1339038 (owner: 10Ejegg)
[18:08:49] <ejegg>	 !log fundraising civicrm upgraded from 89d5c0e6 to ba563432
[18:08:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL
[18:08:54] <ejegg>	  code is up
[18:08:59] <ejegg>	 will run managed reconcile
[18:09:56] <ejegg>	 ok, that's run
[18:10:10] <ejegg>	 larssandergreen: do you want to check that all is correct before I dump new triggers?
[18:11:11] <jinxer-wm>	 FIRING: NodeDown: Node franio2002 is down.  - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=franio2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[18:11:12] <ejegg>	 oh wait i needed wmf-cv api4 WMFConfig.syncCustomFields
[18:11:17] <ejegg>	 not just mgd
[18:11:28] <larssandergreen>	 not sure we do
[18:11:52] <ejegg>	 ah well i ran it too
[18:12:18] <larssandergreen>	 shouldn't hurt
[18:12:21] <larssandergreen>	 but looks good to me
[18:12:45] <ejegg>	 ok, just power-cycled the logging setting, let's see where the file got dumped
[18:12:47] <larssandergreen>	 it doesn't adjust existing fields on prod
[18:13:02] <larssandergreen>	 in /private/log
[18:13:19] <ejegg>	 and it always dumps 3 for some reason
[18:14:01] <ejegg>	 so we want the big one
[18:14:38] <larssandergreen>	 yeah, two makes sense, but I don't get why we get that middle one
[18:15:16] <ejegg>	 oh nothing about model scores in that diff
[18:15:29] <ejegg>	 right, we never actually updated the triggers after creating that table!
[18:15:55] <wikibugs>	 (03PS1) 10Ejegg: Trigger update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339048
[18:16:11] <jinxer-wm>	 RESOLVED: NodeDown: Node franio2002 is down.  - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=franio2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[18:16:56] <ejegg>	 guess I can run the upgrader whenever
[18:18:05] <ejegg>	 ready for review larssandergreen https://gerrit.wikimedia.org/r/1339048
[18:20:37] <larssandergreen>	 hah! after all that, sorry that I got the wrong end of the stick on that one 
[18:21:32] <ejegg>	 no worries, it was a pretty safe assumption the table was there
[18:21:40] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] Trigger update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339048 (owner: 10Ejegg)
[18:22:16] <larssandergreen>	 I don't think we actually need to deploy the trigger update, nothing that we really need in there, so we can just wait until later
[18:22:28] <ejegg>	 ok
[18:22:35] <ejegg>	 will just go ahead with the upgrader
[18:25:03] <ejegg>	 k, upgrader is run
[18:27:24] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: RefundQueue consumer failure: "Parameter "where" is required..." - https://phabricator.wikimedia.org/T437602#12309507 (10Ejegg) Patch deployed, messages requeued
[18:29:07] <wikibugs>	 (03PS5) 10Ejegg: Save ML calculated features as JSON blob [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338054 (https://phabricator.wikimedia.org/T437304)
[18:30:01] <ejegg>	 fr-tech ^^^ will store the calculated features - the ML service is already sending them so we just need that patch to start storing them
[18:30:16] <ejegg>	 then I'll update the single-record view that lars made to add those
[18:30:34] <ejegg>	 actually I could add that to the same patch
[18:32:47] <ejegg>	 oh let's see if we can make that page use order_id too
[18:34:27] <wikibugs>	 (03CR) 10Cstone: Better error display and reload whole app when trying again (033 comments) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1332843 (https://phabricator.wikimedia.org/T433140) (owner: 10Cstone)
[18:41:00] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 10Recurring-Donations: Trustly subsequent recurring fails with invalid_service_credentials - https://phabricator.wikimedia.org/T434933#12309557 (10Ejegg) Dami, this is related to the Bank of America API update you mentioned in sta...
[18:41:08] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 3 others: Automate some Direct Mail white mail coding - https://phabricator.wikimedia.org/T386031#12309569 (10Lars) @RLewis For the email notification, we decided that in order to b...
[18:41:59] <wikibugs>	 (03Merged) 10jenkins-bot: Trigger update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339048 (owner: 10Ejegg)
[18:42:43] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: venmo recurring failing with Payment method has not been successfully stored - https://phabricator.wikimedia.org/T434741#12309573 (10Ejegg) If we know they're going to fail, can we just go ahead and cancel the recurrings?
[18:55:55] <wikibugs>	 06Fundraising-Backlog, 06FR-donorrelations: populate donor PII at Trustly console - https://phabricator.wikimedia.org/T436862#12309591 (10MBeat33) Trustly's latest response is:  Our Internal Team has concluded we are not able to modify or amend the items in the merchant portal. They mentioned that we obtain th...
[19:20:34] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: venmo recurring failing with Payment method has not been successfully stored - https://phabricator.wikimedia.org/T434741#12309716 (10Laurabarluzzi) I think we can, but I'm not sure why it wasn't considered (I think I asked if we do that, and I was told we...
[19:43:54] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: venmo recurring failing with Payment method has not been successfully stored - https://phabricator.wikimedia.org/T434741#12309822 (10Damilare) Thanks @Laurabarluzzi, we did consider it and decided to let it go through the failure process in order to trigg...
[20:42:47] <wikibugs>	 (03CR) 10CI reject: [V:04-1] build: Updating npm dependencies [extensions/FundraisingTranslateWorkflow] (REL1_43) - 10https://gerrit.wikimedia.org/r/1157359 (owner: 10Libraryupgrader)
[20:47:14] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: [trustly recon Auditor] Required field/s missing from message: gateway_txn_id - https://phabricator.wikimedia.org/T434916#12309994 (10Ejegg) Looks like this was deployed on Aug 18th
[20:47:56] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Smashpig: Gravy IPN job runner should retry if it doesn't find data which should be in pending table - https://phabricator.wikimedia.org/T405484#12310011 (10Ejegg) Moving back to the backlog as there's nothing pending deploy
[20:55:38] <wikibugs>	 (03PS1) 10Ejegg: Update Civi-RIP extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339150
[20:56:54] <wikibugs>	 (03PS1) 10Ejegg: Update Contact Layout Editor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339151
[20:58:24] <wikibugs>	 (03PS1) 10Ejegg: Update Fuzion tokens extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339152
[21:00:59] <wikibugs>	 (03PS1) 10Ejegg: Update composer dependencies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339155
[21:01:51] <wikibugs>	 (03PS1) 10Ejegg: Update composer libraries [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1339156
[21:03:38] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: INVALID_MESSAGE No payment type found for message [stripe_audit] - https://phabricator.wikimedia.org/T434871#12310075 (10Ejegg) Looks like this was deployed on Aug 18th too
[21:10:45] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: venmo recurring failing with Payment method has not been successfully stored - https://phabricator.wikimedia.org/T434741#12310091 (10Ejegg) Looks like there is only one outstanding. I suggest that in order to close this ticket and save ourselves a puzzle...
[21:25:46] <jinxer-wm>	 FIRING: [4x] HAProxyBackendDown: HAProxy backend tls_backend on pay-lb1001.frack.eqiad.wmnet is down   - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendDown
[21:26:06] <wikibugs>	 (03PS1) 10Dmiranda: Add disclaimer and drop 'recurring' from login email [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1339169 (https://phabricator.wikimedia.org/T435107)
[21:45:46] <jinxer-wm>	 RESOLVED: [4x] HAProxyBackendDown: HAProxy backend tls_backend on pay-lb1001.frack.eqiad.wmnet is down   - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendDown
[22:00:38] <wikibugs>	 (03PS1) 10Lars SG: Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031)
[22:00:57] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[22:01:28] <wikibugs>	 (03PS2) 10Lars SG: Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031)
[22:07:06] <wikibugs>	 (03PS3) 10Lars SG: Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031)
[22:07:22] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[22:09:52] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences, and 3 others: Add additional questions to donor portal trouble logging in email - https://phabricator.wikimedia.org/T435107#12310241 (10dmiranda) Thanks @AMJohnson and @krobinson for the...
[22:10:17] <wikibugs>	 (03PS4) 10Lars SG: Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031)
[22:14:13] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] Update Civi-RIP extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339150 (owner: 10Ejegg)
[22:14:46] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] Update Contact Layout Editor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339151 (owner: 10Ejegg)
[22:15:07] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] Update Fuzion tokens extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339152 (owner: 10Ejegg)
[22:18:18] <ejegg>	 fr-tech would anyone be able to look at this patch? I have to rebase it every time there's another change to Upgrader.php: https://gerrit.wikimedia.org/r/1338054
[22:21:40] <wikibugs>	 (03PS3) 10Eileen: Queue original message with only necessary changes to DonationMessage [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1337671 (https://phabricator.wikimedia.org/T426098)
[22:21:42] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Queue original message with only necessary changes to DonationMessage [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1337671 (https://phabricator.wikimedia.org/T426098) (owner: 10Eileen)
[22:28:17] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences, and 3 others: Add additional questions to donor portal trouble logging in email - https://phabricator.wikimedia.org/T435107#12310332 (10Lars) @dmiranda I think the "I am unable to log ......
[22:31:54] <cstone>	 looking ejegg 
[22:32:34] <wikibugs>	 (03Merged) 10jenkins-bot: Update Civi-RIP extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339150 (owner: 10Ejegg)
[22:32:35] <wikibugs>	 (03Merged) 10jenkins-bot: Update Contact Layout Editor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339151 (owner: 10Ejegg)
[22:37:13] <wikibugs>	 (03Merged) 10jenkins-bot: Update Fuzion tokens extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339152 (owner: 10Ejegg)
[22:37:49] <wikibugs>	 (03CR) 10Cstone: [C:03+2] "I see the blob" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338054 (https://phabricator.wikimedia.org/T437304) (owner: 10Ejegg)
[22:39:35] <wikibugs>	 (03Merged) 10jenkins-bot: Queue original message with only necessary changes to DonationMessage [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1337671 (https://phabricator.wikimedia.org/T426098) (owner: 10Eileen)
[22:40:31] <wikibugs>	 (03PS1) 10Lars SG: Make direct mail appeal & package custom fields use option values [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339214 (https://phabricator.wikimedia.org/T386031)
[22:48:54] <ejegg>	 thank you cstone !
[22:50:16] <ejegg>	 I'm going to deploy those extension & composer library upgrades in a bit. Don't see any breaking changes anywhere
[22:50:30] <ejegg>	 this is the full diff for the composer packages: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/vendor/+/1339156
[22:50:40] <wikibugs>	 (03PS2) 10Ejegg: Update composer dependencies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339155
[22:50:46] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Update composer dependencies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339155 (owner: 10Ejegg)
[22:50:48] <wikibugs>	 (03CR) 10Ejegg: [V:03+2 C:03+2] Update composer dependencies [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339155 (owner: 10Ejegg)
[22:52:03] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 4 others: Automate some Direct Mail white mail coding - https://phabricator.wikimedia.org/T386031#12310396 (10Lars) I'm converting the string direct_mail_data.direct_mail_appeal and...
[22:52:35] <ejegg>	 I bet there's already a Civi issue for upgrading to tc-lib-pdf
[22:53:43] <wikibugs>	 12Fundraising Sprint: Quiz Show Podium, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 4 others: Automate some Direct Mail white mail coding - https://phabricator.wikimedia.org/T386031#12310409 (10Lars) @RLewis Here's a screenshot of the email (it only shows the expec...
[22:53:52] <ejegg>	 yeah: https://lab.civicrm.org/dev/core/-/work_items/6726
[23:01:49] <wikibugs>	 (03Merged) 10jenkins-bot: Save ML calculated features as JSON blob [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1338054 (https://phabricator.wikimedia.org/T437304) (owner: 10Ejegg)
[23:03:06] <wikibugs>	 (03PS1) 10Lars SG: WIP: Add gift coding for White Mail donations and those expected by MG Engagement activities [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339216 (https://phabricator.wikimedia.org/T386031)
[23:03:42] <wikibugs>	 (03CR) 10CI reject: [V:04-1] WIP: Add gift coding for White Mail donations and those expected by MG Engagement activities [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339216 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[23:04:20] <ejegg>	 sorry larssandergreen i passed the upgrader rebase curse to you
[23:06:23] <wikibugs>	 (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1339219
[23:06:26] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1339219 (owner: 10Ejegg)
[23:06:58] <wikibugs>	 (03PS5) 10Lars SG: Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031)
[23:07:15] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG)
[23:09:41] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1339219 (owner: 10Ejegg)
[23:11:01] <wikibugs>	 (03PS6) 10Lars SG: Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031)
[23:15:57] <wikibugs>	 (03PS7) 10Lars SG: Create missing option values for DM package [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339195 (https://phabricator.wikimedia.org/T386031)
[23:15:57] <wikibugs>	 (03PS2) 10Lars SG: Make direct mail appeal & package custom fields use option values [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1339214 (https://phabricator.wikimedia.org/T386031)
[23:16:57] <larssandergreen>	 evidently ejegg
[23:21:35] <jinxer-wm>	 FIRING: StrongswanFailure: Strongswan is not fully established on frlog2002.frack.codfw.wmnet:9100 for peerhost civi2002   - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure
[23:26:35] <jinxer-wm>	 RESOLVED: StrongswanFailure: Strongswan is not fully established on frlog2002.frack.codfw.wmnet:9100 for peerhost civi2002   - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure
[23:38:35] <ejegg>	 !log fundraising civicrm upgraded from ba563432 to 9436540f
[23:38:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL
[23:51:44] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Refund Contribution searchkit task fails on second search - https://phabricator.wikimedia.org/T437451#12310583 (10Lars) @ejegg I can't reproduce this one, can you provide more details to replicate?