[00:48:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:3) 3 [=1], paypal_transaction_failure 89 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [00:53:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:3) 3 [=1], check_endpoints_warning (Paypal:1) 1 [=1], paypal_transaction_failure 88 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [00:56:37] looks like some more transient paypal failures. [00:56:43] i'll keep an eye for a few mins [00:57:21] 10Fundraising Tech - Chaos Crew: Apple Pay Payment Processing Certificate Expires September 12, 2023 - https://phabricator.wikimedia.org/T344288 (10AnnWF) Created a ticket https://phabricator.wikimedia.org/T345307 to upload the pem key to server and then update the main.yaml with new certificate at local-config,... [00:58:17] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:3) 3 [=1], paypal_transaction_failure 90 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [01:03:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:3) 3 [=1], paypal_transaction_failure 69 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [01:07:31] same issue with same ip. gonna let it ride out. [01:08:19] PROBLEM - check_log_messages on frav1003 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:3) 3 [=1], check_endpoints_warning (Paypal:2) 2 [=1], paypal_transaction_failure 47 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [01:13:19] RECOVERY - check_log_messages on frav1003 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1003&service=check_log_messages [09:22:13] (03PS7) 10Jgleeson: Handled dLocal 'Wallet disabled' IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/952491 (https://phabricator.wikimedia.org/T341300) [09:22:15] (03PS4) 10Jgleeson: Add tests for dLocal PaidMessage IPN Job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/954141 [09:36:56] (03PS8) 10Jgleeson: Handled dLocal 'Wallet disabled' IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/952491 (https://phabricator.wikimedia.org/T341300) [09:36:58] (03PS5) 10Jgleeson: Add tests for dLocal PaidMessage IPN Job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/954141 [09:55:10] (03PS9) 10Jgleeson: Handle dLocal 'Wallet disabled' IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/952491 (https://phabricator.wikimedia.org/T341300) [09:55:12] (03PS6) 10Jgleeson: Add tests for dLocal PaidMessage IPN Job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/954141 [13:15:00] (03CR) 10Jforrester: [C: 03+1] Migrate calls to TablePager::getBody to ::getBodyOutput (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/954113 (https://phabricator.wikimedia.org/T345369) (owner: 10Jforrester) [13:45:45] (03PS1) 10Damilare Adedoyin: Disabling Gift Sources - Community Gift and Benefactor Gift [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954296 (https://phabricator.wikimedia.org/T343735) [13:47:29] (03PS2) 10Damilare Adedoyin: Disabling Gift Sources - Community Gift and Benefactor Gift [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954296 (https://phabricator.wikimedia.org/T343735) [14:02:41] (03CR) 10CI reject: [V: 04-1] Disabling Gift Sources - Community Gift and Benefactor Gift [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954296 (https://phabricator.wikimedia.org/T343735) (owner: 10Damilare Adedoyin) [14:12:52] (03PS1) 10Damilare Adedoyin: Add Placeholder value in fiscal_number field for Dlocal ZA transactions [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/954299 (https://phabricator.wikimedia.org/T307743) [14:16:23] (03PS3) 10Damilare Adedoyin: Disable Fiscal Number Field in Dlocal ZA [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/953594 (https://phabricator.wikimedia.org/T307743) [14:36:10] 10Fundraising Sprint Quit playing games with my Phab 🎶, 10Fundraising-Backlog: Implement Auto-Rescue Webhook handling - https://phabricator.wikimedia.org/T340164 (10Damilare) a:03Damilare [14:38:58] (03PS3) 10Damilare Adedoyin: Disabling Gift Sources - Community Gift and Benefactor Gift [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954296 (https://phabricator.wikimedia.org/T343735) [15:18:24] 10fundraising-tech-ops: Puppet 7.x support in Fundraising environment - https://phabricator.wikimedia.org/T345207 (10Jgreen) [15:20:13] 10fundraising-tech-ops: Puppet 7.x support in Fundraising environment - https://phabricator.wikimedia.org/T345207 (10Jgreen) [15:20:30] 10fundraising-tech-ops: Puppet 7.x support in Fundraising environment - https://phabricator.wikimedia.org/T345207 (10Jgreen) [15:41:43] (03PS1) 10Jgleeson: Update UPI Consumer to support canceling subscriptions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954309 (https://phabricator.wikimedia.org/T341300) [15:45:06] gl [15:45:30] (03PS2) 10Jgleeson: Update UPI Consumer to support canceling subscriptions [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954309 (https://phabricator.wikimedia.org/T341300) [16:47:24] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10MediaWiki-extensions-CLDR, 10Language-Team (Language-2023-July-September), and 2 others: quibble-fundraising-cldr-REL1_39-php74-docker failure on CLDR master - https://phabricator.wikimedia.org/T328240 (10greg) p:05Medium→03High [17:30:54] (03PS1) 10Jgleeson: WIP: UpiDonationsQueueConsumerTest refactor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954319 [17:41:02] (03PS2) 10Jgleeson: WIP: UpiDonationsQueueConsumerTest refactor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954319 [17:57:17] (03CR) 10CI reject: [V: 04-1] WIP: UpiDonationsQueueConsumerTest refactor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/954319 (owner: 10Jgleeson) [18:14:09] 10Fundraising-Backlog: Edit formchooser to redirect Ingenico requests - https://phabricator.wikimedia.org/T345461 (10AKanji-WMF) [18:49:52] 10fundraising-tech-ops: Deactivate fundraising accounts for twhiteantelope - https://phabricator.wikimedia.org/T345466 (10Dwisehaupt) [19:01:15] 10fundraising-tech-ops: Deactivate fundraising accounts for twhiteantelope - https://phabricator.wikimedia.org/T345466 (10Dwisehaupt) [19:03:04] 10fundraising-tech-ops: Deactivate fundraising accounts for twhiteantelope - https://phabricator.wikimedia.org/T345466 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt SSL cert revoke and CRL pushed out. Civi account moved to blocked. All other items verified as not applicable. [19:26:10] 10fundraising-tech-ops: Puppet 7.x support in Fundraising environment - https://phabricator.wikimedia.org/T345207 (10Jgreen) [19:37:02] It looks like I put this task about new Civi fields into fr-ops - but is that the right place? https://phabricator.wikimedia.org/T336890 [20:06:13] PROBLEM - check_puppetrun on frdb1006 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 5 minutes ago with 36 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_puppetrun [20:06:19] PROBLEM - check_puppetrun on frpig1002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 5 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_puppetrun [20:11:13] PROBLEM - check_puppetrun on frdb1006 is CRITICAL: CRITICAL: Puppet has 36 failures. Last run 10 minutes ago with 36 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_puppetrun [20:11:17] RECOVERY - check_puppetrun on frpig1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig1002&service=check_puppetrun [20:16:19] RECOVERY - check_puppetrun on frdb1006 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_puppetrun [21:09:34] 10Fundraising Sprint Quit playing games with my Phab 🎶, 10Fundraising-Backlog: Change non campaign markets from Ingenico to Adyen - https://phabricator.wikimedia.org/T328964 (10AnnWF) @EMartin, Currently ARS, INR ,COP, BRL, PEN 's default gateway is dlocal for both cc bank transfer and cash, then UYU and MXN's... [23:49:01] 10Fundraising-Backlog: Data Axle project - New fields in Civi - https://phabricator.wikimedia.org/T336890 (10AKanji-WMF) @Dwisehaupt did I misfile this in ops? @Eileenmcnaughton @MDemosWMF flagged that this task should be completed before {T341902} [23:54:27] 10Fundraising-Backlog: Data Axle project - New fields in Civi - https://phabricator.wikimedia.org/T336890 (10Dwisehaupt) @AKanji-WMF Yes this is misfiled. As far as I can tell, this isn't something we would be leading.