[00:44:39] FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [70.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [00:49:39] RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [55.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore [01:07:27] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_43) - 10https://gerrit.wikimedia.org/r/1293299 (owner: 10L10n-bot) [02:46:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [11.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [02:46:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [17.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [02:46:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [17.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [02:51:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [02:51:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [02:51:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [03:05:31] 06Fundraising-Backlog: May 2026 Gravy recurring payment failure issues - https://phabricator.wikimedia.org/T427341 (10Cstone) 03NEW [03:05:52] 06Fundraising-Backlog: May 2026 Gravy recurring payment failure issues - https://phabricator.wikimedia.org/T427341#11957857 (10Cstone) [03:05:53] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: gpay validation error as "Could not find an acquirer account" - https://phabricator.wikimedia.org/T426112#11957858 (10Cstone) [03:06:45] 06Fundraising-Backlog: May 2026 Gravy recurring payment failure issues - https://phabricator.wikimedia.org/T427341#11957874 (10Cstone) [03:06:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: recurring failure due to invalid cvv or invalid number - https://phabricator.wikimedia.org/T426119#11957873 (10Cstone) [04:30:01] 06Fundraising-Backlog: May 2026 Gravy recurring payment failure issues - https://phabricator.wikimedia.org/T427341#11957895 (10Cstone) [04:30:02] 06Fundraising-Backlog, 07Epic: Gr4vy Chaos Epic - https://phabricator.wikimedia.org/T403769#11957896 (10Cstone) [04:30:10] 06Fundraising-Backlog: May 2026 Gravy recurring payment failure issues - https://phabricator.wikimedia.org/T427341#11957898 (10Cstone) [04:30:12] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Recurring-Donations, 07payments-orchestration: Recurring bank charges fail with "Payment method has not been successfully stored." - https://phabricator.wikimedia.org/T426963#11957897 (10Cstone) [04:30:13] 06Fundraising-Backlog, 07Epic: Gr4vy Chaos Epic - https://phabricator.wikimedia.org/T403769#11957899 (10Cstone) [04:32:47] 06Fundraising-Backlog: Recurring paypal fails with Validation error during recurring charge, in field: payment_method/id. - https://phabricator.wikimedia.org/T427344 (10Cstone) 03NEW [04:32:59] 06Fundraising-Backlog: Recurring paypal fails with Validation error during recurring charge, in field: payment_method/id. - https://phabricator.wikimedia.org/T427344#11957910 (10Cstone) [04:33:01] 06Fundraising-Backlog: May 2026 Gravy recurring payment failure issues - https://phabricator.wikimedia.org/T427341#11957911 (10Cstone) [04:36:31] (03PS1) 10Cstone: WIP: Contribution modify queue consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1293840 (https://phabricator.wikimedia.org/T422091) [05:47:56] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1293900 (owner: 10L10n-bot) [08:26:00] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Adyen batches incorrect - https://phabricator.wikimedia.org/T427339#11958212 (10RKumar_WMF) I have reviewed the deposit debited from your Merchant Account WikimediaDonations. The reason you did not receive the full payout for batch 1189... [08:26:38] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Adyen batches incorrect - https://phabricator.wikimedia.org/T427339#11958214 (10RKumar_WMF) {F84575922} [09:27:16] 14Fundraising Sprint: Vixen, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Edit Civi Grant Deadline reminder email message - https://phabricator.wikimedia.org/T150862#11958555 (10RLewis) @DBautista-WMF - do you think it would be possible to make a dashlet for this that the gift officer... [10:14:40] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Manual Civi send now Thank You Email for Endowment DAF gifts includes tax information - https://phabricator.wikimedia.org/T427371 (10RLewis) 03NEW [11:12:04] (03PS1) 10Hashar: qunit: disable Firefox telemetry data submission for ~4% speedup [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1294252 (https://phabricator.wikimedia.org/T427103) [11:52:59] 06Fundraising-Backlog: AVS blocking transactions in Greece - https://phabricator.wikimedia.org/T427254#11959243 (10Damilare) @RKumar_WMF thanks for making the phab. This was added and deployed yesterday. [11:53:22] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog: AVS blocking transactions in Greece - https://phabricator.wikimedia.org/T427254#11959245 (10Damilare) [12:44:54] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Clean up communications opt out fields - https://phabricator.wikimedia.org/T425993#11959399 (10Lars) →14Duplicate dup:03T426745 [12:44:56] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07Epic: Epic: Merge duplicative CiviCRM opt out and similar fields - https://phabricator.wikimedia.org/T426745#11959397 (10Lars) [13:25:40] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Implement new audience framework in CiviCRM and export to Acoustic - https://phabricator.wikimedia.org/T426818#11959553 (10Lars) One wrinkle here is how we handle DAF contacts once we are counting the soft-credited donations to the individuals. In this ca... [13:33:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Manual Civi send now Thank You Email for Endowment DAF gifts includes tax information - https://phabricator.wikimedia.org/T427371#11959600 (10Lars) It's not just the manual thank you send, it is the automati... [13:59:27] (03CR) 10Ejegg: [C:03+2] qunit: disable Firefox telemetry data submission for ~4% speedup [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1294252 (https://phabricator.wikimedia.org/T427103) (owner: 10Hashar) [14:09:42] (03Merged) 10jenkins-bot: qunit: disable Firefox telemetry data submission for ~4% speedup [core] (fundraising/REL1_43) - 10https://gerrit.wikimedia.org/r/1294252 (https://phabricator.wikimedia.org/T427103) (owner: 10Hashar) [14:26:40] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-MGFE: Manual Civi send now Thank You Email for Endowment DAF gifts includes tax information - https://phabricator.wikimedia.org/T427371#11959780 (10Lars) p:05Triage→03Lowest [15:34:13] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Smarty templates failing to render when donor name has an apostrophe - https://phabricator.wikimedia.org/T421166#11960113 (10Lars) a:03Lars I updated a [[ https://github.com/civicrm/civicrm-core/pull/35742 | few upstre... [15:58:07] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11960245 (10Jgreen) [16:08:10] 06Fundraising-Backlog: Recurring paypal fails with Validation error during recurring charge, in field: payment_method/id. - https://phabricator.wikimedia.org/T427344#11960301 (10Ejegg) Probably rows still to clean up after the fix to {T423638} [16:14:20] 06Fundraising-Backlog: Update retryable gravy responses - https://phabricator.wikimedia.org/T427420 (10Cstone) 03NEW [16:46:43] 10Fundraising analytics stack, 06Fundraising-Backlog, 10fundraising-tech-ops, 06collaboration-services, and 2 others: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#11960537 (10SBurnett-WMF) 05Open→03Resolved As per Ilse, we can close. [16:57:13] (03PS1) 10Cstone: Updating the retryable reasons based on https://docs.gr4vy.com/guides/api/error-codes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1294354 (https://phabricator.wikimedia.org/T413905) [16:57:33] (03PS2) 10Cstone: Update the retryable reasons based on https://docs.gr4vy.com/guides/api/error-codes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1294354 (https://phabricator.wikimedia.org/T413905) [17:02:20] (03PS1) 10Ejegg: Civix upgrade for geocoder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1294357 [17:02:20] (03PS1) 10Ejegg: Update dependencies for Geocoder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1294358 [17:06:35] (03CR) 10Lars SG: [C:03+2] Civix upgrade for geocoder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1294357 (owner: 10Ejegg) [17:08:13] (03CR) 10Lars SG: [C:03+2] Update dependencies for Geocoder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1294358 (owner: 10Ejegg) [17:25:08] 10Fundraising analytics stack, 06Fundraising-Backlog, 10fundraising-tech-ops, 06collaboration-services, and 2 others: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#11960752 (10IAckerman-WMF) Ah, I meant we could take it off the FR-Tech plan for F... [17:25:32] 10Fundraising analytics stack, 06Fundraising-Backlog, 10fundraising-tech-ops, 06collaboration-services, and 2 others: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#11960754 (10IAckerman-WMF) 05Resolved→03In progress [17:32:32] (03Merged) 10jenkins-bot: Civix upgrade for geocoder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1294357 (owner: 10Ejegg) [17:32:33] (03Merged) 10jenkins-bot: Update dependencies for Geocoder [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1294358 (owner: 10Ejegg) [17:35:52] thanks larssandergreen ! [17:38:55] sorry, I should smoke test that some more before deploy though [17:39:53] ok, it's adding city and state from zip as it should [17:40:08] I'm not seeing geodata but maybe that's just the contact layout [17:40:17] gonna have lunch and test a bit more. [17:58:16] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: If default dashlets are disabled and contact has no active dashlets, DB error loading dashboard - https://phabricator.wikimedia.org/T427289#11960812 (10Lars) a:03Lars [18:01:51] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11960819 (10Jgreen) [18:38:53] 10fundraising-tech-ops, 10observability: Deprecate Fundraising nsca icinga alert collection - https://phabricator.wikimedia.org/T425424#11960923 (10Jgreen) 05Open→03Resolved Closing this, afaik there's nothing left to do. [18:46:49] 06Fundraising-Backlog, 10fundraising-tech-ops: Error logline about Queue host on Payment adapters - https://phabricator.wikimedia.org/T412519#11960944 (10Jgreen) 05Open→03Resolved a:03Jgreen Looking at the data collected, these seem rare enough (19 total in the past 2 days) I suspect it's attributabl... [18:59:15] ok, it is indeed adding the latitude and longitude [18:59:20] will prep the deploy [19:01:14] (03PS1) 10Ejegg: Update geocoder libraries [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1294383 [19:01:16] (03CR) 10Ejegg: [C:03+2] Update geocoder libraries [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1294383 (owner: 10Ejegg) [19:01:51] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1294384 [19:05:17] (03CR) 10Damilare Adedoyin: [C:03+2] Update the retryable reasons based on https://docs.gr4vy.com/guides/api/error-codes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1294354 (https://phabricator.wikimedia.org/T413905) (owner: 10Cstone) [19:05:51] (03Merged) 10jenkins-bot: Update the retryable reasons based on https://docs.gr4vy.com/guides/api/error-codes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1294354 (https://phabricator.wikimedia.org/T413905) (owner: 10Cstone) [19:27:01] (03Merged) 10jenkins-bot: Update geocoder libraries [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1294383 (owner: 10Ejegg) [19:48:28] 06Fundraising-Backlog: a test issue - https://phabricator.wikimedia.org/T427445 (10FR-Tech-Request-Bot) 03NEW [19:49:46] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Figure out how to ensure that all donations are settling - https://phabricator.wikimedia.org/T427238#11961139 (10Eileenmcnaughton) Note the missing paypal ones are fundraiseup it seems [19:59:17] (03PS1) 10Umherirrender: build: Update predis/predis to 2.4 [extensions/FundraisingEmailUnsubscribe] - 10https://gerrit.wikimedia.org/r/1294398 [20:02:11] FIRING: NodeDown: Node frav1003 is down. - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frav1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:03:11] 06Fundraising-Backlog: christine test - https://phabricator.wikimedia.org/T427447 (10FR-Tech-Request-Bot) 03NEW [20:05:33] FIRING: RsyslogBacklog: 3 ryslog files queued up on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [20:05:48] (03CR) 10Umherirrender: "check php" [extensions/FundraisingEmailUnsubscribe] - 10https://gerrit.wikimedia.org/r/1294398 (owner: 10Umherirrender) [20:07:11] RESOLVED: [2x] NodeDown: Node frav1003 is down. - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:10:33] RESOLVED: RsyslogBacklog: 3 ryslog files queued up on civi1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [20:27:43] (03PS2) 10Umherirrender: build: Update predis/predis to 2.4 [extensions/FundraisingEmailUnsubscribe] - 10https://gerrit.wikimedia.org/r/1294398 (https://phabricator.wikimedia.org/T425578) [20:29:08] (03PS6) 10Damilare Adedoyin: Add fraud prediction service to dev stack [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1275424 (https://phabricator.wikimedia.org/T408750) [20:30:15] (03PS7) 10Damilare Adedoyin: Add fraud prediction service to dev stack [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1275424 (https://phabricator.wikimedia.org/T408750) [20:31:33] (03PS1) 10Bvibber: Update name and address for bvibber [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1294406 [20:31:47] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, 07fr-current-sprint: Chariot import - get api access and investigate pulling in DAF (in particular PayPal grants) directly - https://phabricator.wikimedia.org/T415825#11961259 (10Eileenmcnaughton)... [20:41:07] (03PS8) 10Ejegg: Add fraud prediction service to dev stack [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1275424 (https://phabricator.wikimedia.org/T408750) (owner: 10Damilare Adedoyin) [20:43:19] I rebased that damilare ^^^ [20:44:01] ohh ok, thanks, I had rebased it earlier, so I was wondering why some changes came back [20:44:36] I changed the docker compose mac with mount to extend the main docker compose instead of redefining the services [20:45:05] ahh, weird, a different version of that change merged [20:45:43] no worries, I'd add it back [20:46:17] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/1271073 [20:47:07] looks like PS11 or 12 had the extends: stuff [20:47:08] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/1271073/11/docker-compose-mac-with-volume-mount.yml [20:48:39] seems the rebase on 13 is the culprit. I'd add them back now [20:48:51] so we don't have to be monitoring the 2 files [20:48:57] ok, great [20:55:36] (03PS1) 10Damilare Adedoyin: Extend docker compose services in docker with volume mount [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1294423 [20:57:50] (03PS9) 10Damilare Adedoyin: Add fraud prediction service to dev stack [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1275424 (https://phabricator.wikimedia.org/T408750) [21:01:05] ok that should fix it ^ [21:01:40] I've also pushed up the email file to config private and the added the kserve code to the gitlab repo [21:10:23] (03CR) 10Umherirrender: [C:03+2] Update name and address for bvibber [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1294406 (owner: 10Bvibber) [21:10:28] FIRING: OOM: OOM killer active on fransw1003.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [21:15:28] RESOLVED: OOM: OOM killer active on fransw1003.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [21:27:30] (03Merged) 10jenkins-bot: Update name and address for bvibber [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1294406 (owner: 10Bvibber) [21:41:03] FIRING: StrongswanFailure: Strongswan is not fully established on frban1002.frack.eqiad.wmnet:9100 for peerhost frdev1003 - https://alerts.wikimedia.org/?q=alertname%3DStrongswanFailure [22:00:18] ^^^ just re-ack'd this for longer while i continue the build out testing. [22:05:28] FIRING: OOM: OOM killer active on fransw1001.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [22:10:28] RESOLVED: OOM: OOM killer active on fransw1001.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [22:15:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [14.3] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [22:23:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [10.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [22:23:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [10.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [22:23:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [10.0] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [22:28:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [6.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [22:28:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [6.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [22:28:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [6.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [22:28:47] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Settlement data visualisation - https://phabricator.wikimedia.org/T427456 (10Eileenmcnaughton) 03NEW [22:45:09] fr-tech: as a reference, the HighPayment*Messages alerts take time to bubble up and arrive about 5 mins after the email about suspect transactions arrives. [22:45:41] so useful if the email gets missed, but the email report is the most timely as of now. [23:01:22] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11961532 (10Dwisehaupt) [23:02:01] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11961534 (10Dwisehaupt) Network bonding fixes in `13c1a98785c7703bfeb234fb7fd29add151a0585` [23:20:10] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, 07fr-current-sprint: Get Paypal Fundraise Up Audit reports coming in by SFTP - https://phabricator.wikimedia.org/T424083#11961574 (10Cstone) okay we have a username and pw! @Dwisehaupt I don't ha... [23:25:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.8] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [23:33:38] 03Fundraising Sprint: Infinity Pool, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit, 07fr-current-sprint: Get Paypal Fundraise Up Audit reports coming in by SFTP - https://phabricator.wikimedia.org/T424083#11961597 (10Eileenmcnaughton) woohoo