[00:01:11] RESOLVED: [7x] NodeDown: Node frdb2005 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:01:56] !log onfig 1c59ae42 -> fa79267c [00:01:57] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [00:06:11] FIRING: [6x] NodeDown: Node frqueue2004 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:06:17] FIRING: [2x] HAProxyExporterDown: HAProxy on pay-lb2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [00:09:24] (03CR) 10Cstone: [C:03+2] chariot: also validate early on the DonationAudit processor [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1308559 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:10:02] (03Merged) 10jenkins-bot: chariot: also validate early on the DonationAudit processor [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1308559 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:10:04] (03CR) 10Cstone: [C:03+2] Use property rather than passing api parameter around [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1307005 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:10:39] (03Merged) 10jenkins-bot: Use property rather than passing api parameter around [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1307005 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:11:11] FIRING: NodeDown: Node payments2006 has been down for long. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments2006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:11:11] RESOLVED: [4x] NodeDown: Node pay-lb2001 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:11:17] RESOLVED: [2x] HAProxyExporterDown: HAProxy on pay-lb2001.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [00:13:59] (03CR) 10CI reject: [V:04-1] chariot: fix benevity fees [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308590 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:14:49] (03CR) 10Cstone: [C:03+2] Chariot: add maintenance script to list & set properties [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1307006 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:15:24] (03Merged) 10jenkins-bot: Chariot: add maintenance script to list & set properties [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1307006 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:16:02] (03CR) 10Cstone: [C:03+2] Chariot - rename individual & matching_gift amount to total_amount [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308262 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:16:11] RESOLVED: NodeDown: Node payments2006 has been down for long. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=payments2006 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:16:11] (03CR) 10Cstone: [C:03+2] Chariot audit: add new fields to test csv files [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308263 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:16:25] (03CR) 10Cstone: [C:03+2] Notice fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308344 (owner: 10Eileen) [00:16:36] (03CR) 10Cstone: [C:03+2] chariot: fix comparison for string zero values [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308579 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:24:51] (03PS1) 10Eileen: Chariot: Declare a few more fields [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1308795 (https://phabricator.wikimedia.org/T415825) [00:25:22] (03CR) 10CI reject: [V:04-1] Chariot: Declare a few more fields [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1308795 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:26:11] FIRING: [4x] NodeDown: Node frmx2002 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:27:41] (03PS2) 10Eileen: Chariot: Declare a few more fields [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1308795 (https://phabricator.wikimedia.org/T415825) [00:31:11] RESOLVED: NodeDown: Node frdb2003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb2003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [00:39:00] (03Merged) 10jenkins-bot: Chariot - rename individual & matching_gift amount to total_amount [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308262 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:39:39] (03Merged) 10jenkins-bot: Chariot audit: add new fields to test csv files [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308263 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:39:42] (03Merged) 10jenkins-bot: Notice fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308344 (owner: 10Eileen) [00:39:57] (03Merged) 10jenkins-bot: chariot: fix comparison for string zero values [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308579 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [00:51:17] (03PS3) 10Eileen: chariot: fix benevity fees [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308590 (https://phabricator.wikimedia.org/T415825) [00:55:55] (03PS1) 10Eileen: Upgrading wikimedia/smash-pig (v1.2.4.19 => v1.2.4.20) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1308796 [00:57:13] (03CR) 10Eileen: [C:03+2] Upgrading wikimedia/smash-pig (v1.2.4.19 => v1.2.4.20) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1308796 (owner: 10Eileen) [01:09:10] (03CR) 10CI reject: [V:04-1] chariot: fix benevity fees [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308590 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [01:14:45] (03Merged) 10jenkins-bot: Upgrading wikimedia/smash-pig (v1.2.4.19 => v1.2.4.20) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1308796 (owner: 10Eileen) [01:31:10] (03PS4) 10Eileen: chariot: fix benevity fees [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308590 (https://phabricator.wikimedia.org/T415825) [01:55:32] (03CR) 10Cstone: [C:03+2] chariot: fix benevity fees [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308590 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [01:56:29] (03CR) 10Cstone: [C:03+2] Initial checkout.com audit processor (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1307867 (https://phabricator.wikimedia.org/T431240) (owner: 10Eileen) [02:00:20] (03CR) 10Cstone: [C:03+2] Chariot: Declare a few more fields [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1308795 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [02:00:59] (03Merged) 10jenkins-bot: Chariot: Declare a few more fields [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1308795 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [02:10:58] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1308800 [02:11:13] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1308800 (owner: 10Eileen) [02:12:01] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1308800 (owner: 10Eileen) [02:12:53] !log SmashPig upgraded from 2042f7f7 to 87dfbd6b [02:12:54] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [02:17:41] (03Merged) 10jenkins-bot: chariot: fix benevity fees [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308590 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [02:17:44] (03Merged) 10jenkins-bot: Initial checkout.com audit processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1307867 (https://phabricator.wikimedia.org/T431240) (owner: 10Eileen) [02:26:05] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1308801 [02:26:16] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1308801 (owner: 10Eileen) [02:27:12] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1308801 (owner: 10Eileen) [02:28:04] !log civicrm upgraded from f4ff79c8 to 3a2952ff [02:28:06] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [02:45:47] (03PS1) 10Eileen: Do not send thank you for workplace giving [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308803 [02:46:03] cstone: one tiny extra one ^^ [02:56:35] (03CR) 10Cstone: [C:03+2] Do not send thank you for workplace giving [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308803 (owner: 10Eileen) [03:09:44] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1308804 [03:10:02] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1308804 (owner: 10Eileen) [03:12:54] !log civicrm upgraded from 3a2952ff to 3a1a0891 [03:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [04:20:14] (03PS1) 10Eileen: chariot: handle 0.00 where there is no value [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308810 (https://phabricator.wikimedia.org/T415825) [04:24:39] (03CR) 10Lars SG: [C:03+2] chariot: handle 0.00 where there is no value [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308810 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [04:47:44] (03Merged) 10jenkins-bot: chariot: handle 0.00 where there is no value [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1308810 (https://phabricator.wikimedia.org/T415825) (owner: 10Eileen) [04:52:38] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1308813 [05:02:17] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1308813 (owner: 10Eileen) [05:02:58] !log civicrm upgraded from 3a1a0891 to 20b3dabe [05:02:59] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [10:07:04] (03PS1) 10Laurabarluzzi: Cancel Double Opt-In activities if donor later opts out [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1309117 (https://phabricator.wikimedia.org/T416702) [11:07:24] 03Fundraising Sprint: Loud carpet, 06Fundraising-Backlog, 10Fundraising-CiviCRM-Starting-Tasks, 07fundraising-supporter-opt-in, and 6 others: Cancel Double Opt-In activities if donor later opts out. - https://phabricator.wikimedia.org/T416702#12104255 (10Laurabarluzzi) @Lars I pushed a patch that implement... [11:59:16] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Content-Transform-Team (Work In Progress), 07Essential-Work, 07Wikimedia-production-error: PHP Deprecated: Use of MediaWiki\Output\OutputPage::parserOptions was deprecated in MediaWiki 1.44.... - https://phabricator.wikimedia.org/T390041#12104660 [11:59:26] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Content-Transform-Team (Work In Progress), 07Essential-Work, 07Wikimedia-production-error: PHP Deprecated: Use of MediaWiki\Output\OutputPage::parserOptions was deprecated in MediaWiki 1.44... - https://phabricator.wikimedia.org/T390040#12104661 [13:20:12] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Omnirecipient.load null return - https://phabricator.wikimedia.org/T431468#12105199 (10Lars) Backfill finally finished overnight, after removing the new throttle_count limit (turns out 'thottle' was a bit of a load bearing typo). [14:13:34] clear [14:53:23] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Venmo donation missing PII in Civi 6/28/26 - https://phabricator.wikimedia.org/T430699#12105782 (10Lars) It looks like we had ~180 since June 28, but none in the last 24h, so maybe Gravy fixed : https://civicrm.wikimedia.org/civicrm... [15:27:15] 06Fundraising-Backlog, 06Security-Team, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07SecTeam-Processed: Allow app.goacoustic.com in CSP for fundraising contact info collection - https://phabricator.wikimedia.org/T419752#12105983 (10sbassett) a:05sbassett→03BCornwall [16:30:06] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Merging contacts can create multiple segment/status change rows in wmf_donor_history - https://phabricator.wikimedia.org/T431741 (10Lars) 03NEW [16:40:37] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Prevent to assign order id to processor_contact_id - https://phabricator.wikimedia.org/T431327#12106264 (10AnnWF) a:03AnnWF [16:41:31] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: dlocal recurring has no subscription ID - https://phabricator.wikimedia.org/T423717#12106275 (10AnnWF) p:05Medium→03Low [18:18:20] laurabar: `alias c='clear` [18:18:37] it's liberating 😅 [18:32:25] ahah wow I don't remember, I must have had a terminal instinct [18:44:11] FIRING: NodeDown: Node franio1001 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=franio1001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:49:11] RESOLVED: NodeDown: Node franio1001 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=franio1001 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:55:41] FIRING: [3x] NodeDown: Node franio1001 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:58:37] that's odd. franio1001 isn't down. [18:59:29] possibly just a caching oddity on reporting since it was definitely up then. [19:00:41] RESOLVED: [3x] NodeDown: Node franio1001 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:13:11] FIRING: NodeDown: Node frnetmon1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frnetmon1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:18:11] RESOLVED: NodeDown: Node frnetmon1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frnetmon1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:21:40] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: FormBuilder default operator of = isn't displayed since 6.17 - https://phabricator.wikimedia.org/T431509#12106927 (10Lars) 05Open→03Invalid Seems to have fixed itself, perhaps with a cache flush [20:34:57] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 2 others: Improve matching gift verification & thank you process - https://phabricator.wikimedia.org/T425577#12106956 (10MDemosWMF) @Lars Thanks for thinking through how to make this process mo... [20:43:19] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 2 others: Improve matching gift verification & thank you process - https://phabricator.wikimedia.org/T425577#12106979 (10Lars) @MDemosWMF I think before October is reasonable, hopefully sooner.... [21:43:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: SearchDisplay totals aggregation select per column doesn't seem to save changes after 6.17 upgrade - https://phabricator.wikimedia.org/T431508#12107099 (10Lars) a:03Lars [[ https://github.com/civicrm/civicrm-core/pull/... [23:12:11] FIRING: NodeDown: Node frauth1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frauth1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:13:47] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Validation error during recurring charge, in field: last_name - https://phabricator.wikimedia.org/T431459#12107309 (10AnnWF) Take this as example: Validation error during recurring charge, in field: last_name. Message: required. Payment Method: apple. P... [23:17:11] RESOLVED: NodeDown: Node frauth1002 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frauth1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:31:38] (03PS1) 10Lars SG: Don't overwrite SearchDisplay totals column aggregate selection when loading [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1309308 (https://phabricator.wikimedia.org/T431508)