[00:16:27] <wikibugs>	 (03PS11) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161)
[00:19:30] <wikibugs>	 (03PS1) 10Eileen: Handle chargeback_reversals [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1230532
[00:20:06] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Handle chargeback_reversals [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1230532 (owner: 10Eileen)
[00:21:39] <wikibugs>	 (03PS2) 10Eileen: Handle chargeback_reversals [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1230532 (https://phabricator.wikimedia.org/T414531)
[00:24:25] <cstone>	 eileen: idono if itll mess you up locally but i rebased the sftp one onto smashpig master to get that library
[00:25:02] <eileen>	 cstone: should be OK - the patch I was just working on is hopefully done not but it doesn't actually need to be in a chain either
[00:39:57] <wikibugs>	 (03PS3) 10Eileen: Handle chargeback_reversals [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1230532 (https://phabricator.wikimedia.org/T414531)
[00:40:51] <eileen>	 I rebased it onto master ^^
[00:49:43] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Audience-Dev: Donor Segment and Status update when Recurring Status changes - https://phabricator.wikimedia.org/T415319#11547325 (10Lars)
[00:52:55] <cstone>	 i like this comment :P // Finalize.
[00:58:17] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Contribution not merged to merged contact - https://phabricator.wikimedia.org/T415327 (10AnnWF) 03NEW
[01:00:04] <wikibugs>	 (03CR) 10Cstone: [C:03+2] "thanks for all this! straightforward and appreciate all the examples in the commit msg" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) (owner: 10Eileen)
[01:00:45] <wikibugs>	 (03Merged) 10jenkins-bot: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) (owner: 10Eileen)
[01:01:22] <eileen>	 cstone: I will pass your compliments on to the friend who wrote them....
[01:01:46] <cstone>	 if they are writing nice examples i cant argue :P
[01:01:55] <eileen>	 I did ask them to....
[01:02:19] <cstone>	 i guess I didnt test them all so one could be a trap :P
[01:02:42] <eileen>	 lol yeah I think I tested most of the variations tbh 
[01:03:26] <wikibugs>	 (03CR) 10Cstone: [C:03+2] "could be useful in the future!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229832 (https://phabricator.wikimedia.org/T414999) (owner: 10Eileen)
[01:03:31] <eileen>	  I didn't test private-key-file-password - adding that was my friends idea so if you are looking for a back door ...
[01:04:25] <cstone>	 id figured wed ask dallas what he thought to do with that part and we can just update if needed?
[01:06:21] <eileen>	 yeah - so at the moment it's in a config file & the file has a comment to say 'this would be better in a key file' - I would up not being able to download so I put it in a key file in my .ssh on prod to test if that was it - but it wasn't - it was the phpsec2 didn't have modern enough protocols - but anyway - I tested using  a key file - I just didn't put a password on it
[01:07:28] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Audience-Dev: Donor Segment and Status update when Recurring Status changes - https://phabricator.wikimedia.org/T415319#11547431 (10Lars) Currently, I don't think we don't change the status when the recurring contribution is cancelled at all (not j...
[01:10:24] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Contribution not merged to merged contact - https://phabricator.wikimedia.org/T415327#11547437 (10AnnWF) Had to move over the contribution to the new cid manually, wonder if that's expected or we should move all contribution over when merge cids
[01:12:27] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email: Message template preview gives invalidChecksum and doesn't fill in email - https://phabricator.wikimedia.org/T415329 (10Lars) 03NEW
[01:12:39] <wikibugs>	 (03CR) 10Wfan: [C:03+2] Do not skip Trustly refund when Contribution is not found. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230376 (https://phabricator.wikimedia.org/T412984) (owner: 10Damilare Adedoyin)
[01:12:44] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: {wmf_url.unsubscribe} has "invalidChecksum" - https://phabricator.wikimedia.org/T415097#11547451 (10Lars) @ppenloglou This fixes the invalidChecksum issue for {wmf_url.unsubscribe}. You'll note that it still shows invali...
[01:17:24] <wikibugs>	 (03PS1) 10Lars SG: Save date format on the user job, when set on the Data source form for contact imports [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230546 (https://phabricator.wikimedia.org/T415314)
[01:20:34] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Delete all dedupe exceptions for Japan - https://phabricator.wikimedia.org/T415331 (10Lars) 03NEW
[01:25:05] <wikibugs>	 (03CR) 10Eileen: [C:03+2] Save date format on the user job, when set on the Data source form for contact imports [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230546 (https://phabricator.wikimedia.org/T415314) (owner: 10Lars SG)
[01:28:02] <wikibugs>	 (03Merged) 10jenkins-bot: Do not skip Trustly refund when Contribution is not found. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230376 (https://phabricator.wikimedia.org/T412984) (owner: 10Damilare Adedoyin)
[01:32:17] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Contribution not merged to merged contact - https://phabricator.wikimedia.org/T415327#11547533 (10Lars) This problem was because the contribution was added to a batch entry import before the contacts were merged and then...
[01:32:52] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Contribution imported to trashed contact because the contact was merged after data entered in batch - https://phabricator.wikimedia.org/T415327#11547535 (10Lars)
[02:12:51] <wikibugs>	 (03CR) 10Cstone: [C:03+2] "not reversing the +2" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1230532 (https://phabricator.wikimedia.org/T414531) (owner: 10Eileen)
[02:13:52] <wikibugs>	 (03Merged) 10jenkins-bot: Handle chargeback_reversals [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1230532 (https://phabricator.wikimedia.org/T414531) (owner: 10Eileen)
[02:28:20] <wikibugs>	 (03CR) 10Cstone: [C:03+2] "thanks for this!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1225015 (https://phabricator.wikimedia.org/T412307) (owner: 10Damilare Adedoyin)
[02:28:51] <wikibugs>	 (03Merged) 10jenkins-bot: Add script to update Contribution Tracking row for contributions with missing UTM parameters in tracking table using UTM parameters that are available in the logs. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1225015 (https://phabricator.wikimedia.org/T412307) (owner: 10Damilare Adedoyin)
[02:28:53] <wikibugs>	 (03Merged) 10jenkins-bot: Specify the type of trustly transactions to send chargeback messages for. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1230372 (https://phabricator.wikimedia.org/T409471) (owner: 10Damilare Adedoyin)
[02:48:39] <wikibugs>	 (03CR) 10Cstone: [C:03+2] "thanks looks good!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229943 (https://phabricator.wikimedia.org/T409362) (owner: 10Eileen)
[02:48:44] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Move gravy settlement reports to ignored [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229944 (https://phabricator.wikimedia.org/T409362) (owner: 10Eileen)
[02:48:48] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Ignore dlocal Cleared.csv files [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229945 (https://phabricator.wikimedia.org/T409362) (owner: 10Eileen)
[03:01:01] <wikibugs>	 (03PS1) 10Cstone: Update smash-pig to v1.1.1.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230578
[03:01:32] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Update smash-pig to v1.1.1.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230578 (owner: 10Cstone)
[03:05:39] <wikibugs>	 (03Merged) 10jenkins-bot: Move ignored reconciliation files to an ignored directory [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229943 (https://phabricator.wikimedia.org/T409362) (owner: 10Eileen)
[03:11:26] <wikibugs>	 (03Merged) 10jenkins-bot: Move gravy settlement reports to ignored [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229944 (https://phabricator.wikimedia.org/T409362) (owner: 10Eileen)
[03:11:28] <wikibugs>	 (03Merged) 10jenkins-bot: Ignore dlocal Cleared.csv files [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229945 (https://phabricator.wikimedia.org/T409362) (owner: 10Eileen)
[03:28:20] <wikibugs>	 (03Merged) 10jenkins-bot: Update smash-pig to v1.1.1.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230578 (owner: 10Cstone)
[03:36:36] <wikibugs>	 (03PS1) 10Eileen: Upgrading wikimedia/smash-pig (v1.1.1 => v1.1.1.1) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1230597
[03:36:48] <wikibugs>	 (03CR) 10Eileen: [C:03+2] Upgrading wikimedia/smash-pig (v1.1.1 => v1.1.1.1) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1230597 (owner: 10Eileen)
[03:37:21] <wikibugs>	 (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1230598
[03:37:30] <wikibugs>	 (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1230598 (owner: 10Eileen)
[03:50:27] <eileen>	 !log civicrm upgraded from 9e754954 to f7064a46
[03:50:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:52:39] <wikibugs>	 (03Merged) 10jenkins-bot: Upgrading wikimedia/smash-pig (v1.1.1 => v1.1.1.1) [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1230597 (owner: 10Eileen)
[04:00:56] <wikibugs>	 (03PS1) 10Eileen: Declare city as valid auditMessage field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230610
[04:09:33] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: When a contact is merged and then unmerged, the contribution stays with the merged contact - https://phabricator.wikimedia.org/T387948#11547670 (10Lars) 05Open→03Invalid @MBeat33 just unmerged a pair of contacts...
[04:18:49] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Declare city as valid auditMessage field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230610 (owner: 10Eileen)
[06:12:54] <wikibugs>	 (03PS1) 10Cstone: Move the recurring failure sending and activity creation to a helper [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230706
[06:18:22] <wikibugs>	 (03PS6) 10Cstone: Add second recurring failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199940 (https://phabricator.wikimedia.org/T365488)
[06:19:08] <wikibugs>	 (03PS2) 10Cstone: Move the recurring failure sending and activity creation to a helper [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230706
[11:20:27] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: {wmf_url.unsubscribe} has "invalidChecksum" - https://phabricator.wikimedia.org/T415097#11548433 (10ppenloglou) Again, much appreciated, you seem to resolve these at lightning speed! @Lars
[15:12:19] <wikibugs>	 14Fundraising Sprint Cheese String Theory, 14Fundraising Sprint Discworld reformatted as ntfs, 14Fundraising Sprint e^🥧👀=yum, 06Fundraising-Backlog, and 4 others: Civi Searchkit should allow user to export list to acoustic - https://phabricator.wikimedia.org/T301514#11549165 (10Lars)
[15:17:18] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Add one-click ‘copy email address’ function - https://phabricator.wikimedia.org/T415374 (10Lars) 03NEW
[15:31:39] <wikibugs>	 (03CR) 10Damilare Adedoyin: [C:03+2] move logic in HTML to script (032 comments) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1224243 (owner: 10Wfan)
[15:34:59] <wikibugs>	 (03CR) 10Ejegg: "Looks like an unintended rename got checked in" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230610 (owner: 10Eileen)
[15:39:06] <wikibugs>	 (03PS1) 10Ejegg: WIP map IDs for merged contacts in recipient load [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230940 (https://phabricator.wikimedia.org/T404404)
[15:40:27] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Add link in Zendesk to contact in CiviCRM - https://phabricator.wikimedia.org/T415378 (10Lars) 03NEW
[15:50:25] <wikibugs>	 (03Merged) 10jenkins-bot: move logic in HTML to script [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1224243 (owner: 10Wfan)
[15:52:51] <wikibugs>	 (03PS1) 10Ejegg: Avoid validating just-generated checksum [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230947 (https://phabricator.wikimedia.org/T402837)
[15:56:53] <wikibugs>	 (03CR) 10CI reject: [V:04-1] WIP map IDs for merged contacts in recipient load [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230940 (https://phabricator.wikimedia.org/T404404) (owner: 10Ejegg)
[16:01:02] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07fr-payments-ACH: Recurring ACH looks like a success but then is actually failed - https://phabricator.wikimedia.org/T409471#11549356 (10Damilare) Found another 29 transactions that got declined but still in Civi. Here's the list for [[ https://civicrm...
[16:49:43] <wikibugs>	 (03CR) 10Ejegg: "Looks good, except for this issue with the additional active recur check. Can be fixed in a follow-on patch." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199940 (https://phabricator.wikimedia.org/T365488) (owner: 10Cstone)
[16:49:49] <wikibugs>	 (03PS7) 10Cstone: Add second recurring failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199940 (https://phabricator.wikimedia.org/T365488)
[16:49:51] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Add second recurring failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199940 (https://phabricator.wikimedia.org/T365488) (owner: 10Cstone)
[16:54:50] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] "Looks like it just needs an alias, i.e. ->addSelect('COUNT(id) as count')" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199940 (https://phabricator.wikimedia.org/T365488) (owner: 10Cstone)
[16:59:05] <wikibugs>	 (03CR) 10Ejegg: [C:04-1] "Wouldn't it make more sense to extend FailureEmail.render to take a workflow parameter?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230706 (owner: 10Cstone)
[16:59:25] <wikibugs>	 (03CR) 10Ejegg: [C:04-1] "(and FailureEmail.send)" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230706 (owner: 10Cstone)
[17:09:20] <wikibugs>	 (03Merged) 10jenkins-bot: Add second recurring failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199940 (https://phabricator.wikimedia.org/T365488) (owner: 10Cstone)
[17:10:48] <wikibugs>	 10fundraising-tech-ops: Metabase access for Sarah Holko - https://phabricator.wikimedia.org/T415385#11549589 (10greg) @CHudson-WMF if you use the template link from https://wikitech.wikimedia.org/wiki/Fundraising/techops/docs/fundraising_access_request it'll automatically add the correct project tag. I just adde...
[17:10:52] <wikibugs>	 (03PS1) 10Ejegg: WIP FailureEmail::send can sending first or second [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230968 (https://phabricator.wikimedia.org/T365488)
[17:17:59] <greg-g>	 stashbot: help
[17:17:59] <stashbot>	 See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help.
[17:18:07] <greg-g>	 !log test where does this go?
[17:18:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:18:51] <greg-g>	 looks like we need to get bryan to help us make a separate feed for us so we're not mixed up in the deluge of production logs
[17:26:35] <dwisehaupt>	 that'd be cool
[17:36:29] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific channel - https://phabricator.wikimedia.org/T415389 (10greg) 03NEW
[17:36:58] <greg-g>	 now it's a task ^ :)
[17:37:58] <wikibugs>	 10fundraising-tech-ops: Metabase access for Sarah Holko - https://phabricator.wikimedia.org/T415385#11549672 (10CHudson-WMF) Thanks, @greg -- is it just the link to create an access request in the document? I'm getting a permission denied error when I click on it: {F71597983}
[17:45:12] <wikibugs>	 10fundraising-tech-ops: Metabase access for Sarah Holko - https://phabricator.wikimedia.org/T415385#11549688 (10greg) Looks like the form that the template uses is restricted to reduce spam (presumably). I just added you to #wmf-nda so you can now use it. Just, usual statement: now you can see things in Phabrica...
[17:45:17] <icinga-wm>	 PROBLEM - check_disk on frdb1005 is CRITICAL: DISK CRITICAL - free space: /dev 257265MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4254MiB (60% inode=87%): /dev/shm 257295MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 235MiB (55% inode=99%): /home 12257MiB (99% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25233MiB (95% inode=99%): /var/log 23615MiB (88% inode=99%): /var/tmp 422MiB (99% inode=99%): /var/log/audit
[17:45:17] <icinga-wm>	 99% inode=99%): /srv 724242MiB (9% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_disk
[17:45:35] <greg-g>	 uhhh, that seems full
[17:50:07] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Delete all dedupe exceptions for Japan - https://phabricator.wikimedia.org/T415331#11549696 (10Lars) This is done.
[17:50:17] <icinga-wm>	 PROBLEM - check_disk on frdb1005 is CRITICAL: DISK CRITICAL - free space: /dev 257265MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4254MiB (60% inode=87%): /dev/shm 257295MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 235MiB (55% inode=99%): /home 12257MiB (99% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25233MiB (95% inode=99%): /var/log 23615MiB (88% inode=99%): /var/tmp 422MiB (99% inode=99%): /var/log/audit
[17:50:17] <icinga-wm>	 99% inode=99%): /srv 381647MiB (4% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_disk
[17:52:45] <greg-g>	 dwisehaupt: ^
[17:52:57] <dwisehaupt>	 yeah. looking into that.
[17:53:00] <greg-g>	 ty
[17:55:17] <icinga-wm>	 PROBLEM - check_disk on frdb1005 is CRITICAL: DISK CRITICAL - free space: /dev 257265MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4254MiB (60% inode=87%): /dev/shm 257295MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 235MiB (55% inode=99%): /home 12257MiB (99% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25233MiB (95% inode=99%): /var/log 23615MiB (88% inode=99%): /var/tmp 422MiB (99% inode=99%): /var/log/audit
[17:55:17] <icinga-wm>	 99% inode=99%): /srv 568219MiB (7% inode=99%): /run/user/573 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_disk
[17:56:01] <dwisehaupt>	 ok. there were 4 backed up queries that were creating massive sort indexes.
[18:00:17] <icinga-wm>	 RECOVERY - check_disk on frdb1005 is OK: DISK OK - free space: /dev 257265MiB (100% inode=99%): /run 51457MiB (99% inode=99%): / 4254MiB (60% inode=87%): /dev/shm 257295MiB (100% inode=99%): /run/lock 5MiB (100% inode=99%): /boot 235MiB (55% inode=99%): /home 12257MiB (99% inode=99%): /tmp 12350MiB (99% inode=99%): /var 25233MiB (95% inode=99%): /var/log 23615MiB (88% inode=99%): /var/tmp 422MiB (99% inode=99%): /var/log/audit 422MiB (99
[18:00:17] <icinga-wm>	 9%): /srv 5138117MiB (67% inode=99%): /run/user/573 51459MiB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_disk
[18:07:16] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Repeat sql queries created giant sort indexes and needed to be terminated - https://phabricator.wikimedia.org/T415392 (10Dwisehaupt) 03NEW
[18:07:23] <dwisehaupt>	 that was close: If they would have kept running, we would have shortly run out of disk on the FRDB origin server. https://frmon.wikimedia.org/goto/afb25dtfv8mpsd?orgId=1
[18:07:36] <greg-g>	 eeek
[18:08:34] <dwisehaupt>	 seems like something that we may be able to look at for a 'velocity' based prometheus alert. aka increase over X% in a short period of time.
[18:11:27] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Repeat sql queries created giant sort indexes and needed to be terminated - https://phabricator.wikimedia.org/T415392#11549781 (10Dwisehaupt) This (rapid increase of disk usage) may be a good place to look at prometheus based alerts for the velocity of a change. T...
[18:13:28] <greg-g>	 +1
[18:26:25] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Repeat sql queries created giant sort indexes and needed to be terminated - https://phabricator.wikimedia.org/T415392#11549794 (10Lars) @Damilare These ones are your user id. Looks like SearchKit, do you recall what the search was?
[19:02:44] <wikibugs>	 06Fundraising-Backlog: When civiproxy is down display a better donor preference page - https://phabricator.wikimedia.org/T415393 (10Cstone) 03NEW
[19:03:25] <wikibugs>	 06Fundraising-Backlog, 10FR-Donor-portal: When civiproxy is down display a better donor preference page - https://phabricator.wikimedia.org/T415393#11549850 (10Cstone)
[19:20:55] <wikibugs>	 10fundraising-tech-ops: Renew FR-Tech nessus license - https://phabricator.wikimedia.org/T415395 (10Dwisehaupt) 03NEW
[19:21:49] <ejegg>	 cstone: I'm failing to make that COUNT(id) as count thing work...
[19:22:02] <ejegg>	 but there is a rowCount property on the result object
[19:25:48] <ejegg>	 oh lol is it just that I just couldn't use 'count' as an alias b/c reserved sql word?
[19:25:58] <cstone>	 ooooh
[19:26:16] <cstone>	 `count`?
[19:26:21] <cstone>	 oh haha it wont even do them
[19:31:06] <ejegg>	 i'm v confused - seem to be getting different results in api explorer vs code
[19:31:23] <ejegg>	 adding a group by seems to mean no result rows at all in both
[19:31:47] <ejegg>	 but in the api explorer I do get a single result row with the count when i have no group by
[19:32:06] <cstone>	 i took it from an apiv3 query but when i checked i thought apiv4 had the same syntax
[19:32:07] <ejegg>	 even when the count is 0
[19:35:08] <wfan>	 cstone https://developer.apple.com/apple-pay/sandbox-testing/ I got test card for my account with those, but local still failing, gonna rebuild the payments
[19:35:10] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] "Seems correct, and unlikely to slow things down. Thanks!" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1228970 (https://phabricator.wikimedia.org/T413812) (owner: 10Lars SG)
[19:36:50] <wikibugs>	 (03Merged) 10jenkins-bot: Add double_opt_in_activity to Acoustic export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1228970 (https://phabricator.wikimedia.org/T413812) (owner: 10Lars SG)
[19:36:57] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for jmoreira - https://phabricator.wikimedia.org/T415114#11549955 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[19:42:50] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for eroden - https://phabricator.wikimedia.org/T415126#11549963 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[19:50:29] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for ewilfong - https://phabricator.wikimedia.org/T415124#11549974 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[19:52:15] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add set modified date to now SearchKit task - https://phabricator.wikimedia.org/T415397 (10Lars) 03NEW
[19:52:37] <wikibugs>	 03Fundraising Sprint: ARMchair, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add set modified date to now SearchKit task - https://phabricator.wikimedia.org/T415397#11549987 (10Lars)
[19:53:18] <wikibugs>	 (03PS1) 10Lars SG: Add set modified date to now SearchKit task [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230999 (https://phabricator.wikimedia.org/T415397)
[19:54:58] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for osefu - https://phabricator.wikimedia.org/T415117#11549992 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[20:32:45] <wikibugs>	 (03PS1) 10Ejegg: Get recurring count in API4 w/o warning [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231006 (https://phabricator.wikimedia.org/T365488)
[20:38:23] <ejegg>	 cstone: so ^^^ is what I can figure out
[20:38:40] <ejegg>	 likely about as efficient for most folks
[20:39:38] <ejegg>	 oh, and I could keep going on this WIP patch to make the second... job actually use it: https://gerrit.wikimedia.org/r/1230968
[20:44:05] <wikibugs>	 (03CR) 10Lars SG: "Yeah, I don't think COUNT would be supported in API4, so this is the way to do it, but see minor issue." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231006 (https://phabricator.wikimedia.org/T365488) (owner: 10Ejegg)
[20:47:30] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: New sftp credentials for gravy to connect to paypal - https://phabricator.wikimedia.org/T414750#11550088 (10Dwisehaupt) Thanks. I have added it to the private::passwords::gravy module also. We aren't currently deploying the settings in that file to configuration f...
[20:51:27] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for greg - https://phabricator.wikimedia.org/T415399 (10Dwisehaupt) 03NEW
[20:51:28] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for rholt - https://phabricator.wikimedia.org/T415400 (10Dwisehaupt) 03NEW
[20:51:32] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for ccogdill - https://phabricator.wikimedia.org/T415401 (10Dwisehaupt) 03NEW
[20:51:34] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for jmando - https://phabricator.wikimedia.org/T415402 (10Dwisehaupt) 03NEW
[20:51:37] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for dmorgan-ctr - https://phabricator.wikimedia.org/T415403 (10Dwisehaupt) 03NEW
[20:51:40] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for llugo - https://phabricator.wikimedia.org/T415404 (10Dwisehaupt) 03NEW
[20:53:11] <greg-g>	 heh, my turn ^
[20:57:43] <wikibugs>	 (03CR) 10Eileen: [C:03+2] Add set modified date to now SearchKit task [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230999 (https://phabricator.wikimedia.org/T415397) (owner: 10Lars SG)
[20:58:56] <wikibugs>	 (03PS2) 10Ejegg: FailureEmail::send can send first or second [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230968 (https://phabricator.wikimedia.org/T365488)
[20:59:55] <wikibugs>	 (03PS2) 10Ejegg: Get recurring count in API4 w/o warning [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231006 (https://phabricator.wikimedia.org/T365488)
[21:00:08] <wikibugs>	 (03PS2) 10Eileen: Declare city as valid auditMessage field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230610
[21:00:16] <ejegg>	 thanks larssandergreen, fixed the count() in PS2
[21:00:17] <wikibugs>	 (03CR) 10Eileen: "does doesn't it ... fixed" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230610 (owner: 10Eileen)
[21:00:42] <wikibugs>	 (03PS3) 10Eileen: Declare city as valid auditMessage field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230610
[21:00:43] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Declare city as valid auditMessage field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230610 (owner: 10Eileen)
[21:01:26] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] Get recurring count in API4 w/o warning [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231006 (https://phabricator.wikimedia.org/T365488) (owner: 10Ejegg)
[21:01:40] <ejegg>	 thanks!
[21:02:01] <wikibugs>	 (03PS3) 10Ejegg: FailureEmail::send can send first or second [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230968 (https://phabricator.wikimedia.org/T365488)
[21:02:09] <cstone>	 Thanks for digging into that ejegg larssandergreen guess my attempt to move it from 3 to 4 was not right 
[21:04:15] <larssandergreen>	 for sure. I think if you use the API explorer to generate that's probably the easiest way, since a lot changed between 3 and 4. And the docs on API4 usage are helpful on those methods you can tack on like count(): https://docs.civicrm.org/dev/en/latest/api/v4/usage/
[21:04:28] <cstone>	 I wanted to then move it to a place where both that and the recur charge job can use it but the job is still all api3
[21:05:07] <larssandergreen>	 you can call api4 from api3, if that's what you mean
[21:05:47] <cstone>	 yeah i htink its a separate call more so we should update it all ? so its not 3 and 4
[21:05:55] <cstone>	 i guess the recur helper is where that could live
[21:08:59] <wikibugs>	 (03PS2) 10Ejegg: Map IDs for merged contacts in omnirecipient load [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230940 (https://phabricator.wikimedia.org/T404404)
[21:10:13] <wikibugs>	 (03CR) 10Ejegg: [C:04-1] "Alternative proposal: https://gerrit.wikimedia.org/r/1230968" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230706 (owner: 10Cstone)
[21:11:02] <cstone>	 yeah ejegg I like that idea ^
[21:11:03] <wikibugs>	 (03PS1) 10Eileen: Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022
[21:11:32] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022 (owner: 10Eileen)
[21:12:54] <wikibugs>	 (03PS2) 10Eileen: Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022
[21:13:23] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022 (owner: 10Eileen)
[21:14:49] <wikibugs>	 (03PS3) 10Eileen: Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022
[21:15:22] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022 (owner: 10Eileen)
[21:17:27] <larssandergreen>	 cstone: not sure if ejegg's thing is what you're talking about, but I think it's kind of up to you, nice to upgrade to 4 but not required
[21:18:41] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Repeat sql queries created giant sort indexes and needed to be terminated - https://phabricator.wikimedia.org/T415392#11550195 (10Damilare) Yes @Lars, I do I was trying to get all the contribution tracking records created yesterday [on this SearchKit](https://civi...
[21:18:46] <wikibugs>	 (03Merged) 10jenkins-bot: Declare city as valid auditMessage field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230610 (owner: 10Eileen)
[21:19:48] <wikibugs>	 (03PS4) 10Eileen: Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022
[21:22:25] <wikibugs>	 (03PS1) 10Eileen: Do not bubble up gateway_status [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231025
[21:22:53] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Do not bubble up gateway_status [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231025 (owner: 10Eileen)
[21:23:06] <wikibugs>	 (03Merged) 10jenkins-bot: Get recurring count in API4 w/o warning [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231006 (https://phabricator.wikimedia.org/T365488) (owner: 10Ejegg)
[21:23:40] <wikibugs>	 (03CR) 10Eileen: "Actually it looks like we do map to a Civi field in contribution_extra" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231025 (owner: 10Eileen)
[21:27:06] <wikibugs>	 (03PS1) 10Eileen: Declare gateway_status field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231027
[21:30:04] <wikibugs>	 (03PS1) 10Eileen: Stop if host mismatch [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231029
[21:30:18] <cstone>	 yeah sorry larssandergreen being delayed in responses, the recurring charge job is all apiv3, i copied a query over into the 2nd recrruing failure email and attempted to convert it to apiv4 when then you both fixed
[21:30:26] <cstone>	 which then*
[21:30:39] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Map IDs for merged contacts in omnirecipient load [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1230940 (https://phabricator.wikimedia.org/T404404) (owner: 10Ejegg)
[21:38:10] <wikibugs>	 (03PS1) 10Eileen: Dial back noise on PayPal TRR processing [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231030
[21:38:37] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Dial back noise on PayPal TRR processing [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231030 (owner: 10Eileen)
[21:40:51] <wikibugs>	 (03PS2) 10Eileen: Dial back noise on PayPal TRR processing [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231030
[21:41:17] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Dial back noise on PayPal TRR processing [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231030 (owner: 10Eileen)
[21:42:26] <wikibugs>	 (03PS3) 10Eileen: Dial back noise on PayPal TRR processing [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231030
[22:19:13] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Chariot import fails when there is no exact match found for organization name - https://phabricator.wikimedia.org/T415295#11550319 (10MDemosWMF) @Lars Here is the error report for a recent Cha...
[23:59:33] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07fr-payments-ACH: Recurring ACH looks like a success but then is actually failed - https://phabricator.wikimedia.org/T409471#11550619 (10MBeat33) Many thanks, @Damilare   @RLewis [[ https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=6990544...