[00:15:40] (03PS1) 10Eileen: Submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/726996 [00:15:54] (03CR) 10Eileen: [C: 03+2] Submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/726996 (owner: 10Eileen) [00:26:40] (03Merged) 10jenkins-bot: Submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/726996 (owner: 10Eileen) [00:56:58] (03CR) 10Ejegg: [C: 04-1] "OK, I removed the existing template and re-synced with system.flush and now they show up in the list with titles. However, the en-US one s" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725182 (owner: 10Eileen) [00:58:26] (03CR) 10Eileen: Install translated versions on recurring failed (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725182 (owner: 10Eileen) [00:59:10] (03CR) 10Ejegg: [C: 04-1] "Also, let's not rename the Latvian ones - lt is for Lithuanian, and the subject text in the lv files is detected as Latvian by Google Tran" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/725182 (owner: 10Eileen) [00:59:26] ohhh [00:59:42] eileen: oh hmm, maybe it's missing preferred languages? [00:59:59] ejegg: yeah - I guess it must be - I'll have to dig [01:00:05] lemme see if I have Ukrainian [01:00:23] (03PS4) 10Eileen: Move our handling for the {now.MMMM} token to our custom token processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/726992 [01:01:15] FYI I moved ^^ to be first in the chain because I decided we would have to keep that token as a wmf custom for the medium term - & then found some typos [01:01:26] ok [01:02:10] one wonders how many Latvian recurring donations we have... [01:02:20] or even Danish [01:02:26] enough to write a translation (unless we misnamed it anyway) [01:08:00] (03CR) 10Ejegg: [C: 03+2] "looks good!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/726992 (owner: 10Eileen) [01:18:29] (03Merged) 10jenkins-bot: Move our handling for the {now.MMMM} token to our custom token processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/726992 (owner: 10Eileen) [01:19:45] (03CR) 10Cstone: [C: 03+2] "Thanks for updating this, works great!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/726717 (owner: 10Ejegg) [01:20:03] thanks cstone! [01:20:18] (03Merged) 10jenkins-bot: Update Adyen recur test script for new API [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/726717 (owner: 10Ejegg) [01:20:21] although i didnt like when I tried to SEPA direct debit some USD :P [01:20:36] oh yeah, currency not supported [01:51:02] (03CR) 10Cstone: [C: 03+2] "Thanks for looking at this, a lot cleaner way to do it" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/725154 (owner: 10Ejegg) [01:51:33] (03Merged) 10jenkins-bot: Don't require payment_method for recurring auth [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/725154 (owner: 10Ejegg) [01:52:39] thanks again cstone. I think i'll put up another minor-minor version of the lib and update it in payments-wiki so the tests pass [01:52:50] thanks ejegg [02:01:05] (03PS1) 10Ejegg: Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727007 [02:04:13] (03CR) 10jerkins-bot: [V: 04-1] Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727007 (owner: 10Ejegg) [02:04:27] (03PS2) 10Ejegg: Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727007 [02:05:08] bah, got lazy and squashed the vendor update into the security merge rather than try to rebase it before it [02:06:57] oh boo, needs a fuller composer update [02:07:06] ok, let's see exactly which packages need it [02:07:19] (03CR) 10jerkins-bot: [V: 04-1] Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727007 (owner: 10Ejegg) [02:08:01] (03Abandoned) 10Eileen: Use new civi core token for date [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/726349 (owner: 10Eileen) [02:09:54] (03PS3) 10Ejegg: Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727007 [02:10:06] ah, I just forgot to squash the lock file update in with it [02:11:32] and... i think i did that wrong again [02:13:43] hnmn? ok, the new symfony package IS in composer.lock even if I don't see it in log -p [02:26:49] (03CR) 10Ejegg: [C: 03+2] Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727007 (owner: 10Ejegg) [02:37:27] (03Merged) 10jenkins-bot: Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727007 (owner: 10Ejegg) [03:00:40] !log updated payments-wiki from 23d0ffac66 to 6d3560d083 [03:00:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:05:20] (03PS1) 10Ejegg: DO NOT MERGE bogus way to trigger 422 error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/727009 (https://phabricator.wikimedia.org/T290923) [03:13:36] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/727011 [03:13:49] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/727011 (owner: 10Eileen) [03:18:01] (03PS1) 10Ejegg: WIP Adyen Checkout: catch validation exceptions [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/727012 (https://phabricator.wikimedia.org/T290923) [03:20:00] (03CR) 10jerkins-bot: [V: 04-1] WIP Adyen Checkout: catch validation exceptions [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/727012 (https://phabricator.wikimedia.org/T290923) (owner: 10Ejegg) [06:50:12] 10Wikimedia-Fundraising, 10MediaWiki-extensions-CentralNotice, 10MediaWiki-extensions-Translate, 10SRE, and 8 others: DBPerformance warning "Query returned XXXX rows: query: SELECT * FROM `translate_metadata`" - https://phabricator.wikimedia.org/T204026 (10Nikerabbit) [06:51:31] 10Wikimedia-Fundraising, 10MediaWiki-extensions-CentralNotice, 10MediaWiki-extensions-Translate, 10SRE, and 8 others: DBPerformance warning "Query returned XXXX rows: query: SELECT * FROM `translate_metadata`" - https://phabricator.wikimedia.org/T204026 (10Nikerabbit) It looks like the number-of-rows-retur... [07:06:02] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/727058 (owner: 10L10n-bot) [07:23:08] (03CR) 10Damilare Adedoyin: [C: 03+2] Avoid writing empty prioritylangs rows to translate_metadata [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/720743 (https://phabricator.wikimedia.org/T204026) (owner: 10Nikerabbit) [07:23:49] (03CR) 10Damilare Adedoyin: [C: 03+1] "Looks good to me, thank you." [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/720743 (https://phabricator.wikimedia.org/T204026) (owner: 10Nikerabbit) [07:26:47] (03CR) 10Damilare Adedoyin: [C: 03+2] "Looks good to me, thank you." [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/720743 (https://phabricator.wikimedia.org/T204026) (owner: 10Nikerabbit) [07:31:13] (03Merged) 10jenkins-bot: Avoid writing empty prioritylangs rows to translate_metadata [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/720743 (https://phabricator.wikimedia.org/T204026) (owner: 10Nikerabbit) [12:32:19] 10Fundraising-Backlog, 10fr-donorservices, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): "No App" error from monthly convert on mobile - https://phabricator.wikimedia.org/T291199 (10Dbrant) This is now in the latest production app, and should therefore be fixed. > If we see any more of this e... [12:45:25] 10Fundraising-Backlog, 10fr-donorservices, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): "No App" error from monthly convert on mobile - https://phabricator.wikimedia.org/T291199 (10Dbrant) 05Open→03Resolved [13:16:19] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:(Need By: TBD) rack/setup (4) fundraising hosts - https://phabricator.wikimedia.org/T289812 (10Cmjohnson) [13:54:08] (03CR) 10Ejegg: [C: 03+2] "HTML looks fine,Jenkins plz merge" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/727058 (owner: 10L10n-bot) [13:54:33] hi fr-tech! [13:55:05] ejegg: fr-tech hiiiii :) [13:59:13] 10Fundraising-Backlog, 10fundraising-tech-ops: Current Engage Civi User Needs Additional Certificate - https://phabricator.wikimedia.org/T292649 (10MDemosWMF) Hi @Dwisehaupt the user asked if you can resend the certificate. Thanks! [14:10:16] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/727354 [14:10:41] fr-tech I'mma do a DonationInterface deploy to get those spanish MC translations out for testing [14:11:01] and pcoombe's off switch [14:14:59] ejegg: va que va [14:30:53] 10Fundraising-Backlog, 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, and 2 others: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10sgrabarczuk) We (the Product department, Wikimedia Foundation) are working on an announcement. Within a week, w... [14:34:38] hi fr-tech [14:35:03] (03Abandoned) 10Raimond Spekking: Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/726793 (owner: 10L10n-bot) [15:08:03] hi jgleeson [15:12:56] hi ejegg, were those ecertificates any good? [15:13:05] certificates.. even [15:13:05] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/727354 (owner: 10Ejegg) [15:13:20] jgleeson: I think those are all the right certificates [15:13:41] but what I needed there were the domain validation .txt files [15:13:54] also, hope everyone's feeling much better [15:14:06] oh did not I attach those also? [15:15:00] oh shoot, was it just one file for all the different domains? [15:15:02] I think they're the ones [15:15:13] hah, I was assuming it was 6 different files [15:15:19] lemme see [15:15:20] or they're at least the ones Matt sent over [15:16:18] ok, so the ones I downloaded do seem to be different for each domain [15:17:37] hmm maybe Matt generated test and prod merchant association txt files for just payments.wikimedia.org? [15:17:57] ah, could be [15:18:43] ok, I'll check in those six to the private config repo [15:20:10] hmm, i guess they go in payments [15:21:05] we could also do with removing the applepay directory added to the payments config for the old approach [15:21:29] ah, I'll remove the old files and replace 'em with these domain validation thangs [15:21:41] We've got a ticket to track the progress on that https://phabricator.wikimedia.org/T290108 [15:23:57] looks like you got that GatewayPorts patch merged and some hiera data added? Thanks! [15:24:23] so thinking about it, it does make sense to have a new set of files as we're using a new domain, wmcloud.org and not wikimedia.org so those files are probably not useful for testing [15:24:53] yeah that should fix our sshd_config thingy [15:29:51] sorry I just noticed your well wishes ejegg, thanks! the kids are still under the weather but through the worst of it we think. Oscar's PCR test came back negative today which was good news so feels like we're just riding out a chest infection instead of covid, which is good overall. [15:30:30] whew, glad to hear it's not COVID. chest infection still sounds unfun though. [15:31:43] 10Fundraising-Backlog, 10fundraising sprint Quantum Leapfrog, 10fundraising sprint Roller Toaster, 10fundraising sprint Screaming Goats Yelling Zoo, and 2 others: Switch over to using WMF-generated certificates with Adyen Apple Pay - https://phabricator.wikimedia.org/T290108 (10Ejegg) Test domains have bee... [15:38:37] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/727354 (owner: 10Ejegg) [15:47:13] (03PS1) 10Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727373 [15:47:36] (03CR) 10Ejegg: [C: 03+2] Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727373 (owner: 10Ejegg) [15:57:53] (03Merged) 10jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727373 (owner: 10Ejegg) [15:59:28] (03CR) 10Ejegg: [C: 03+2] "recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/726949 (owner: 10Cstone) [16:01:05] oops, that revert patch didn't get merged [16:01:11] will fold that in before deploy [16:08:55] (03Merged) 10jenkins-bot: Revert "Add payment_method to authorize request" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/726949 (owner: 10Cstone) [16:11:13] guh, paypal EC timeouts [16:11:42] ooh, minfraud timeouts at the same time? [16:12:08] @_@ [16:12:15] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Adyen_endpoint_critical 1 [=1],Paypal_endpoint_critical 4 [=1],Yubico_endpoint_critical 1 [=1],cURL_error 178 [=5],ipset_error 1 [=1],minFraud_endpoint_critical 8 [=1] [16:13:07] yep icinga-wm, it's looking wonky alright [16:13:08] huh [16:13:23] hmm, no Jeff_Green or dwisehaupt on right now [16:13:50] where are you seeing the minfraud timeouts ejegg ? [16:14:31] payments.error [16:14:44] Failed to connect to minfraud.maxmind.com port 443: Connection timed out [16:14:58] ooh, can't push reviews to gerrit either [16:15:15] generalized connectivity issues? [16:15:35] meet is still streaming away [16:16:06] mbeattie: heads up, there seems to be some connectivity problem causing some donation failures [16:16:15] hopefully transient [16:16:32] ejegg: chatting in operations [16:16:51] thanks jgleeson, will pop on over [16:17:05] chatter* even [16:17:14] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 5 [=1],Yubico_endpoint_critical 1 [=1],cURL_error 167 [=5],ipset_error 19 [=1],minFraud_endpoint_critical 8 [=1] [16:20:25] shoot, my connection to frlog seems to have dropped as well [16:22:00] and now phab's not loading... [16:22:16] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Adyen_endpoint_critical 1 [=1],Paypal_endpoint_critical 5 [=1],Yubico_endpoint_critical 1 [=1],cURL_error 138 [=5],ipset_error 25 [=1],minFraud_endpoint_critical 8 [=1] [16:22:43] phab ok for me ejegg [16:22:46] feels like DNS? [16:23:28] dwisehaupt: another fun problem for the start of the day - I started seeing some connectivity issues outbound from payments [16:23:36] from another channel, it looks like it may be a routing loop with cloudflare. [16:23:58] thanks for the update [16:24:04] work is being done to work around it right now. :) [16:24:34] hmm, and this isn't the first time we've had cloudflare issues. I hope they've been absorbing lots of DDOS for us [16:24:39] just hopping into the fire. although it's been interesting to listen to the staff meeting on the kid commute/dog walk this morning. [16:25:53] been quite a while since i've done that. :) [16:27:25] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 1 [=1],Yubico_endpoint_critical 1 [=1],cURL_error 280 [=5],ipset_error 25 [=1],minFraud_endpoint_critical 8 [=1] [16:27:36] it wasn't even on my radar that enterprise could push us out of the non-profit status! what a great question [16:27:47] I never imagined it would make that much [16:27:53] (relative to fundraising) [16:28:58] ok. looks like recovery is happening. [16:32:31] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Yubico_endpoint_critical 1 [=1],cURL_error 344 [=5],ipset_error 25 [=1],minFraud_endpoint_critical 8 [=1] [16:33:01] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/727395 [16:37:26] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Yubico_endpoint_critical 1 [=1],cURL_error 304 [=5],ipset_error 25 [=1],minFraud_endpoint_critical 8 [=1] [16:42:25] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: cURL_error 317 [=5],ipset_error 25 [=1],minFraud_endpoint_critical 8 [=1] [16:45:02] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/727395 (owner: 10Ejegg) [16:47:21] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Yubico_endpoint_critical 1 [=1],cURL_error 327 [=5],ipset_error 25 [=1],minFraud_endpoint_critical 8 [=1] [16:48:48] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/727395 (owner: 10Ejegg) [16:52:25] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: cURL_error 314 [=5],ipset_error 26 [=1],minFraud_endpoint_critical 8 [=1] [16:55:10] (03PS1) 10Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727414 [16:55:13] (03CR) 10Ejegg: [C: 03+2] Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727414 (owner: 10Ejegg) [16:57:21] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: cURL_error 296 [=5],ipset_error 25 [=1],minFraud_endpoint_critical 8 [=1] [16:58:58] (03CR) 10jerkins-bot: [V: 04-1] Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727414 (owner: 10Ejegg) [16:59:56] oh grr, npm registry making my patch fail [17:00:00] npm ERR! request to https://registry.npmjs.org/zwitch/-/zwitch-1.0.5.tgz failed, reason: getaddrinfo EAI_AGAIN registry.npmjs.org [17:02:25] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Yubico_endpoint_critical 1 [=1],cURL_error 288 [=5],ipset_error 25 [=1],minFraud_endpoint_critical 8 [=1] [17:07:21] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: cURL_error 58 [=5],ipset_error 5 [=1],minFraud_endpoint_critical 2 [=1] [17:10:23] 10Fundraising-Backlog: Wikimedia NOT accepting notifications for Adyen - https://phabricator.wikimedia.org/T292766 (10EMartin) [17:10:43] 10Fundraising-Backlog: Wikimedia NOT accepting notifications for Adyen - https://phabricator.wikimedia.org/T292766 (10EMartin) Note: This was flagged as urgent by Adyen. [17:12:21] RECOVERY - check_log_messages on frav1002 is OK: OK [17:12:57] (03CR) 10Ejegg: [C: 03+2] "recheck" [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727414 (owner: 10Ejegg) [17:13:12] fr-tech: re: the DX app monitoring alerts - they are being tracked in T292603 and folks are aware. [17:13:13] T292603: DX App Synthetic Monitoring App - watchmouse alert flapping due to CA expiration - https://phabricator.wikimedia.org/T292603 [17:13:28] thanks dwisehaupt [17:13:29] 10fundraising-tech-ops, 10DC-Ops, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install civi1002.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T292767 (10RobH) [17:14:09] 10fundraising-tech-ops, 10DC-Ops, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install civi1002.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T292767 (10RobH) [17:20:44] (03Merged) 10jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/727414 (owner: 10Ejegg) [17:32:23] 10Fundraising-Backlog: Wikimedia NOT accepting notifications for Adyen - https://phabricator.wikimedia.org/T292766 (10Ejegg) 05Open→03Resolved [17:39:35] 10Fundraising-Backlog: Figure out what to do with OFFER_CLOSED Adyen ipn messages - https://phabricator.wikimedia.org/T292475 (10jgleeson) @Cstone figured out that these are all related to iDEAL trxns. These notifications are probably happening due to donors cancelling the payment on their bank's side after we g... [17:55:32] (03CR) 10Ejegg: [C: 04-2] DO NOT MERGE bogus way to trigger 422 error [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/727009 (https://phabricator.wikimedia.org/T290923) (owner: 10Ejegg) [17:59:56] 10Fundraising-Backlog, 10fundraising-tech-ops: Current Engage Civi User Needs Additional Certificate - https://phabricator.wikimedia.org/T292649 (10Dwisehaupt) Resent certificate via email and password via SMS. [18:10:04] !log updated payments-wiki from 6d3560d083 to 030b11da1a [18:10:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:15:34] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:(Need By: TBD) rack/setup (4) fundraising hosts - https://phabricator.wikimedia.org/T289812 (10Cmjohnson) All the idracs are set up @jgreen please DM about these. Thanks [18:22:29] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Janna Layton - https://phabricator.wikimedia.org/T292222 (10Dwisehaupt) [18:23:44] 10Fundraising-Backlog, 10fundraising-tech-ops: Fundraising access request for Janna Layton - https://phabricator.wikimedia.org/T292222 (10Dwisehaupt) SSL client certificate has been created and sent via email. Certificate password sent via SMS. Superset account created and assigned base permissions. Temporary... [18:29:50] 10Fundraising-Backlog, 10fundraising-tech-ops: Current Engage Civi User Needs Additional Certificate - https://phabricator.wikimedia.org/T292649 (10MDemosWMF) Thank you! She just let me know the next time she will be in the office is early next week and will get it set up then. [18:59:01] (03PS1) 10Ejegg: Ignore OFFER_CLOSED messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/727464 (https://phabricator.wikimedia.org/T292475) [19:13:40] (03CR) 10Jgleeson: [C: 03+2] "Thanks for this!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/727464 (https://phabricator.wikimedia.org/T292475) (owner: 10Ejegg) [19:14:17] (03Merged) 10jenkins-bot: Ignore OFFER_CLOSED messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/727464 (https://phabricator.wikimedia.org/T292475) (owner: 10Ejegg) [20:15:17] 10Fundraising-Backlog: Unable to add contacts to a group via search kit - https://phabricator.wikimedia.org/T292784 (10NNichols) [21:06:11] 10Fundraising-Backlog, 10GlobalUserPage, 10MediaWiki-extensions-CentralNotice: CentralNotice banner goes behind of global user page's top menu buttons - https://phabricator.wikimedia.org/T175865 (10Jdlrobson) [21:33:14] (03PS1) 10Umherirrender: Remove php entry point [extensions/LandingCheck] - 10https://gerrit.wikimedia.org/r/727607 [21:35:19] (03CR) 10Umherirrender: [V: 03+2] build: Updating ansi-regex to 5.0.1 [extensions/DonationInterface] (REL1_36) - 10https://gerrit.wikimedia.org/r/727541 (owner: 10Libraryupgrader) [21:35:21] (03CR) 10Umherirrender: [V: 03+2] build: Updating ansi-regex to 5.0.1 [extensions/DonationInterface] (REL1_37) - 10https://gerrit.wikimedia.org/r/727542 (owner: 10Libraryupgrader) [23:55:06] (03PS2) 10Ejegg: Adyen Checkout: catch validation exceptions [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/727012 (https://phabricator.wikimedia.org/T290923) [23:55:39] phew fr-tech i went way down the rabbit hole of trying to tie into the adyen components validation, and just couldn't make it work [23:59:41] ejegg: I think that meeting tomorrow is still happening to ask that there too?