[00:04:01] 10Fundraising-Backlog, 10Wikimedia-Fundraising-Banners, 10Wikimedia-CentralNotice-Administration, 10Design, 10Performance-Team (Radar): Study options and designs for banner formats to prevent search ranking decline - https://phabricator.wikimedia.org/T280477 (10Jdlrobson) [00:07:09] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_results 0 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [01:17:35] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [01:23:19] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 31998 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 19 days 23 hours - memory use is 24.28M (peak 106.04M, 0.37% of max, fragmentation 1.26%), connected_slaves is 3, donations is 1, jobs is 0, jobs-adyen is 0, jobs-paypal is 1, payments-init is 7, pending is 0, recurring is 1, refund is 0, unsubscribe is 1 https://icinga.wiki [01:23:19] /cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:28:19] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 46505 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 20 days 2 minutes - memory use is 33.48M (peak 106.04M, 0.49% of max, fragmentation 1.19%), connected_slaves is 3, donations is 44, jobs is 0, jobs-adyen is 6, jobs-paypal is 0, payments-init is 106, pending is 0, recurring is 2, refund is 0, unsubscribe is 1 https://icinga. [01:28:19] .org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:33:22] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 59936 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 20 days 7 minutes - memory use is 41.26M (peak 106.04M, 0.58% of max, fragmentation 1.15%), connected_slaves is 3, donations is 11, jobs is 0, jobs-adyen is 1, jobs-paypal is 1, payments-init is 6, pending is 0, recurring is 0, refund is 0, unsubscribe is 1 https://icinga.wi [01:33:22] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:38:22] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 73565 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 20 days 12 minutes - memory use is 49.44M (peak 106.04M, 0.68% of max, fragmentation 1.12%), connected_slaves is 3, donations is 78, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 98, pending is 0, recurring is 0, refund is 0, unsubscribe is 1 https://icinga.w [01:38:22] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:43:22] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 87346 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 20 days 17 minutes - memory use is 57.74M (peak 106.04M, 0.78% of max, fragmentation 1.11%), connected_slaves is 3, donations is 45, jobs is 0, jobs-adyen is 2, jobs-paypal is 0, payments-init is 10, pending is 0, recurring is 0, refund is 0, unsubscribe is 3 https://icinga. [01:43:22] .org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:48:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 84395 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 20 days 22 minutes - memory use is 50.37M (peak 106.04M, 0.69% of max, fragmentation 1.13%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 2, jobs-paypal is 0, payments-init is 101, pending is 0, recurring is 0, refund is 0, unsubscribe is 4 https://icinga.w [01:48:18] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:53:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 62375 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 20 days 27 minutes - memory use is 37.57M (peak 106.04M, 0.55% of max, fragmentation 1.19%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 7, pending is 0, recurring is 0, refund is 0, unsubscribe is 4 https://icinga.wik [01:53:18] g/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:58:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 39733 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 20 days 32 minutes - memory use is 25.32M (peak 106.04M, 0.39% of max, fragmentation 1.27%), connected_slaves is 3, donations is 68, jobs is 0, jobs-adyen is 2, jobs-paypal is 1, payments-init is 104, pending is 0, recurring is 0, refund is 0, unsubscribe is 5 https://icinga [01:58:18] a.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [02:03:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 16605 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 20 days 37 minutes - memory use is 12.72M (peak 106.04M, 0.25% of max, fragmentation 1.57%), connected_slaves is 3, donations is 27, jobs is 0, jobs-adyen is 1, jobs-paypal is 1, payments-init is 9, pending is 0, recurring is 0, refund is 0, unsubscribe is 1 https://icinga.w [02:03:18] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [02:08:18] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 20 days 42 minutes - memory use is 4.50M (peak 106.04M, 0.14% of max, fragmentation 2.61%), connected_slaves is 3, donations is 30, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 1420, payments-init is 100, pending is 0, recurring is 0, refund is 0, unsubscribe is 1 https://icinga.wikimedia.org/cgi-bi [02:08:18] extinfo.cgi?type=2&host=frqueue1003&service=check_redis [06:44:28] (03CR) 10Wangombe: Update references to MessageGroups::class (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/861718 (https://phabricator.wikimedia.org/T313242) (owner: 10Wangombe) [10:17:57] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Revalidate the paymentstest*.wmcloud.org ApplePay dev Merchant Domains - https://phabricator.wikimedia.org/T328700 (10jgleeson) a:03jgleeson [10:44:48] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Revalidate the paymentstest*.wmcloud.org ApplePay dev Merchant Domains - https://phabricator.wikimedia.org/T328700 (10jgleeson) I've revalidated paymentstest[1-6].wmcloud.org. They should be good for another three months, until Apr 17, 2023. [11:12:27] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Reverify payments.wikimedia.org for ApplePay (Production) - https://phabricator.wikimedia.org/T329723 (10jgleeson) [11:37:46] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Get all fr-tech engineers added to the WMF Developer Account - https://phabricator.wikimedia.org/T329724 (10jgleeson) [11:39:30] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Get all fr-tech engineers added to the WMF Developer Account - https://phabricator.wikimedia.org/T329724 (10jgleeson) [11:42:00] 10Fundraising Sprint Drop It Like It's Fraud, 10Fundraising-Backlog, 10FR-Smashpig, 10FR-dlocal, 10Recurring-Donations: Implement backend for DLocal recurring card payments in SmashPig - https://phabricator.wikimedia.org/T324296 (10Damilare) a:03Damilare [15:59:58] 10fundraising-tech-ops, 10decommission-hardware, 10Patch-For-Review: decommission frpm1001.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T329752 (10Jgreen) [16:07:59] (03PS1) 10Damilare Adedoyin: Set gateway_txn_id on createPaymentResponse [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 [16:43:14] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Contribution Tracking queue consumer running for 34 minutes instead of 1 - https://phabricator.wikimedia.org/T329600 (10Cstone) [17:01:01] (03CR) 10Jgleeson: "Thanks for this. Is there a convenient test we could confirm this change with?" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 (owner: 10Damilare Adedoyin) [17:25:09] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Contribution Tracking queue consumer running for 34 minutes instead of 1 - https://phabricator.wikimedia.org/T329600 (10greg) [17:27:59] (03PS2) 10Damilare Adedoyin: Set gateway_txn_id on createPaymentResponse [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 [17:28:26] (03CR) 10CI reject: [V: 04-1] Set gateway_txn_id on createPaymentResponse [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 (owner: 10Damilare Adedoyin) [17:32:40] (03PS3) 10Damilare Adedoyin: Set gateway_txn_id on createPaymentResponse [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 [17:33:09] (03CR) 10CI reject: [V: 04-1] Set gateway_txn_id on createPaymentResponse [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 (owner: 10Damilare Adedoyin) [17:34:14] (03PS4) 10Damilare Adedoyin: Set gateway_txn_id on createPaymentResponse [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 [17:39:53] 10Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for pcoombe - https://phabricator.wikimedia.org/T329648 (10Dwisehaupt) SSL certificate renewed and sent via email. Password sent via SMS. [17:42:30] 10Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for jcuriel - https://phabricator.wikimedia.org/T329537 (10Dwisehaupt) SSL certificate renewed and sent via email. Password sent via SMS. [17:45:44] 10Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for spuri - https://phabricator.wikimedia.org/T328068 (10Dwisehaupt) SSL certificate renewed and sent via email. Password sent via SMS. [17:57:21] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Add a time limit to contribution tracking queue consume - https://phabricator.wikimedia.org/T329600 (10Cstone) [18:07:32] 10Fundraising-Backlog: Update fundraising analysis tools to use Python 3 - https://phabricator.wikimedia.org/T213949 (10Pcoombe) p:05Low→03Medium Thanks for the reminder @Dwisehaupt! I'll try and find time to work on this soon. [18:11:47] 10Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for dallas - https://phabricator.wikimedia.org/T329315 (10Dwisehaupt) SSL certificate renewed and files placed in users home dir on frpm1002. [18:13:39] 10Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for jgreen - https://phabricator.wikimedia.org/T329542 (10Dwisehaupt) SSL certificate renewed and files placed in users home directory on frpm1002. [18:19:36] (03PS1) 10Damilare Adedoyin: Implement backend for DLocal recurring card payments in SmashPig [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889588 (https://phabricator.wikimedia.org/T324296) [18:21:57] (03PS2) 10Damilare Adedoyin: Implement backend for DLocal recurring card payments in SmashPig [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889588 (https://phabricator.wikimedia.org/T324296) [18:34:30] (03PS3) 10Damilare Adedoyin: Implement backend for DLocal recurring card payments in SmashPig [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889588 (https://phabricator.wikimedia.org/T324296) [18:40:14] 10Fundraising Sprint Drop It Like It's Fraud, 10Fundraising-Backlog, 10FR-dlocal: Investigate potential temporary code solution for receiving old and new data formats from dlocal - https://phabricator.wikimedia.org/T329289 (10Cstone) Solution from the dlocal call today: Keep the /astropay enpoint in our con... [18:42:03] thanks cstone ! [18:42:10] glad that was easy to figure out [18:42:27] yeah I was wonder it was gonna require a bunch of work we'd have to throw away at first [18:42:35] same [18:42:42] wondering if it* [18:43:02] testing account idea was great too [18:45:28] I still haven't thought through how we're gonna set that up for testing. It might take some locking down of the dlocal form [18:45:44] as I imagine it would need to be a live URL [18:47:46] hmm for adyen it was just using adyencheckout right? but did we use staging for braintree? [18:49:06] I'm not sure if we've used staging before for this type of thing but I do vaguely remember a time when we deployed something new into production for testing [18:50:34] oh it might have been Adyen. Did we have old and new Adyen running at the same time for a period at the beginning? [18:50:56] we added a new 'checkout' directory didn't we [18:51:26] to accommodate both [18:53:01] yeah jgleeson [19:26:19] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[superset] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [19:26:56] back later [19:31:15] RECOVERY - check_puppetrun on fran1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [19:36:01] 10fundraising-tech-ops, 10FR-Tech-Analytics: Change SUPERSET_WEBSERVER_TIMEOUT flag in config from 1 minute to 3 minutes - https://phabricator.wikimedia.org/T329789 (10Dwisehaupt) This was brought about by looking at T294771. Even though their final solution was different for their error case, it informed our... [19:41:05] 10fundraising-tech-ops, 10FR-Tech-Analytics: Change SUPERSET_WEBSERVER_TIMEOUT flag in config from 1 minute to 3 minutes - https://phabricator.wikimedia.org/T329789 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt This change has been deployed: ` [frack::puppet] 24744f68 Update SUPERSET_WEBSERVER_TIMEOUT... [19:52:28] (03PS1) 10Damilare Adedoyin: Add callback_url to redirect API params [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889605 [19:53:49] (03CR) 10Damilare Adedoyin: "Thanks wfan, looks like the test links added are for AstroPay not Dlocal." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/889188 (https://phabricator.wikimedia.org/T328976) (owner: 10Wfan) [19:56:51] (03CR) 10Damilare Adedoyin: "thanks wfan, I'm thinking we should add the redirect_url to the var_map so we can add the wmf_token to the request like we do in adyen." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/874932 (https://phabricator.wikimedia.org/T324276) (owner: 10Wfan) [20:19:13] (03PS1) 10Wfan: Add cancel payment for authorized not captured payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889629 (https://phabricator.wikimedia.org/T324727) [20:19:34] (03CR) 10CI reject: [V: 04-1] Add cancel payment for authorized not captured payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889629 (https://phabricator.wikimedia.org/T324727) (owner: 10Wfan) [20:28:12] (03PS2) 10Wfan: Add cancel payment for authorized not captured payment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889629 (https://phabricator.wikimedia.org/T324727) [21:02:03] (03CR) 10Wfan: [C: 03+2] "LGTM" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 (owner: 10Damilare Adedoyin) [21:02:35] (03Merged) 10jenkins-bot: Set gateway_txn_id on createPaymentResponse [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/889565 (owner: 10Damilare Adedoyin) [21:02:49] thanks wfan! [22:22:47] (03CR) 10Ejegg: Initiate recurring profiles in PayPal resolver (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/886129 (https://phabricator.wikimedia.org/T324642) (owner: 10Ejegg)