[07:28:25] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_35) - 10https://gerrit.wikimedia.org/r/983619 (owner: 10L10n-bot) [16:04:36] 10fundraising-tech-ops: Investigate pygsheets for fundraising analytics - https://phabricator.wikimedia.org/T353658 (10Jgreen) [18:08:13] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 5 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [18:08:17] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 5 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [18:08:17] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 5 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [18:09:32] looks like my attempt to keep them quiet didn't work. [18:10:13] PROBLEM - check_puppetrun on frlog2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 7 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [18:10:13] PROBLEM - check_puppetrun on frmon2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 7 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2002&service=check_puppetrun [18:10:17] PROBLEM - check_puppetrun on frmx2001 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 7 minutes ago with 29 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [18:11:13] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 8 minutes ago with 32 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [18:11:15] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 8 minutes ago with 32 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [18:11:17] PROBLEM - check_puppetrun on frauth2002 is CRITICAL: CRITICAL: Puppet has 34 failures. Last run 8 minutes ago with 34 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [18:11:17] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 8 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [18:11:23] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 8 minutes ago with 35 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [18:12:33] ok. popped in a downtime for a few minutes for them to pick up a clean run. [18:15:13] RECOVERY - check_puppetrun on frmon2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2002&service=check_puppetrun [18:15:17] RECOVERY - check_puppetrun on frmx2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [18:16:13] RECOVERY - check_puppetrun on pay-lvs2001 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [18:16:15] RECOVERY - check_puppetrun on pay-lvs2002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [18:16:15] RECOVERY - check_puppetrun on frauth2002 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [18:16:23] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [18:18:13] RECOVERY - check_puppetrun on frqueue2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [18:18:17] RECOVERY - check_puppetrun on payments2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [18:18:17] RECOVERY - check_puppetrun on payments2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [18:20:13] RECOVERY - check_puppetrun on frlog2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [18:21:17] RECOVERY - check_puppetrun on payments2002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [18:23:11] (03PS1) 10Wfan: Save rescue_reference for to civicrm_contribution_recur_smashpig [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/983899 [18:29:56] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog: Import and verify small batch of Ingenico tokens from Adyen CSVs - https://phabricator.wikimedia.org/T348771 (10EMartin) Great, Ingenico is preparing to migrate us off of Wells Fargo and to Chase this week.... [19:55:22] (03PS1) 10Wfan: WIP: Save retry.rescueReference from payments result [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/983936 (https://phabricator.wikimedia.org/T353293) [19:56:02] (03CR) 10CI reject: [V: 04-1] WIP: Save retry.rescueReference from payments result [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/983936 (https://phabricator.wikimedia.org/T353293) (owner: 10Wfan) [20:17:15] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10FR-Docker, 10Recurring-Donations: fundraising-dev needs two redis instances - https://phabricator.wikimedia.org/T352928 (10XenoRyet) 05Open→03Resolved [20:18:00] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10FR-Docker: Rename fr-dev 'mediawiki' docker image - https://phabricator.wikimedia.org/T350442 (10XenoRyet) 05Open→03Resolved [20:19:20] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog: Import and verify small batch of Ingenico tokens from Adyen CSVs - https://phabricator.wikimedia.org/T348771 (10XenoRyet) 05Open→03Resolved [20:19:26] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Patch-For-Review: All Ingenico/Worldline active tokens will be migrated to Adyen - https://phabricator.wikimedia.org/T344645 (10XenoRyet) [20:19:47] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Recurring-Donations, 10MW-1.42-notes (1.42.0-wmf.10; 2023-12-19): Error submitting recur upgrade form in production - https://phabricator.wikimedia.org/T352907 (10XenoRyet) 05Open→03Resolved [20:21:19] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Update thank you main & endowment mail to use new stock fields - https://phabricator.wikimedia.org/T350331 (10XenoRyet) 05Open→03Resolved [20:22:26] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.5; 2023-11-14): Edit configuration to send explicit Ingenico requests to Adyen - https://phabricator.wikimedia.org/T345461 (10XenoRyet) 05Open→03Resolved [20:22:53] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.9; 2023-12-12): Enable variant with no matching gift field for GPay/ApplePay/Venmo - https://phabricator.wikimedia.org/T353064 (10XenoRyet) 05Open→03Resolved [20:23:21] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog: Import bug - https://phabricator.wikimedia.org/T352720 (10XenoRyet) 05Open→03Resolved [20:23:54] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.7; 2023-11-28): Test out the readiness of our backup payment processor. - https://phabricator.wikimedia.org/T351714 (10XenoRyet) 05Open→03Resolved [20:24:35] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10FR-donorservices, 10Recurring-Donations: Get CiviProxy -> CiviCRM auth working in production for API4 calls - https://phabricator.wikimedia.org/T349946 (10XenoRyet) 05Open→03Resolved [20:24:38] 10Fundraising-Backlog, 10FR-donorservices, 10Recurring-Donations, 10Epic: How can we enable upgrading of existing recurring donation amount? - https://phabricator.wikimedia.org/T325317 (10XenoRyet) [20:25:41] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Matching Gift information link broken - https://phabricator.wikimedia.org/T352771 (10XenoRyet) 05Open→03Resolved [20:26:21] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Successful Adyen transactions not making it into CiviCRM - https://phabricator.wikimedia.org/T345736 (10XenoRyet) 05Open→03Resolved [20:27:18] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fundraising-tech-ops: Remove Redis Connection Timeout in redis.conf - https://phabricator.wikimedia.org/T352711 (10XenoRyet) 05Open→03Resolved [20:27:21] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Predis\Connection\ConnectionException: Error while reading line from the server - https://phabricator.wikimedia.org/T349719 (10XenoRyet) [20:27:56] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.9; 2023-12-12): Venmo monthly convert - new screen when accepting and email when decliing - https://phabricator.wikimedia.org/T352341 (10XenoRyet) 05Open→03Resolved [20:28:15] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Predis\Connection\ConnectionException: Error while reading line from the server - https://phabricator.wikimedia.org/T349719 (10XenoRyet) 05Open→03Resolved [20:28:45] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Audit: create missing contribution_tracking entries instead of skipping importing them - https://phabricator.wikimedia.org/T339182 (10XenoRyet) 05Open→03Resolved [20:29:03] 10Fundraising Tech - Chaos Crew, 10FR-Tech-Analytics, 10FR-tech-data-integrity: Odd Refund Recording in Civi - https://phabricator.wikimedia.org/T351651 (10XenoRyet) 05Open→03Resolved [20:29:28] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-donorservices: Donation attempts via Venmo get to error page. - https://phabricator.wikimedia.org/T353067 (10XenoRyet) 05Open→03Resolved [20:29:56] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Smashpig, 10FR-dlocal: DLocal IPN failures due to card type lookup - https://phabricator.wikimedia.org/T351673 (10XenoRyet) 05Open→03Resolved [20:30:45] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Smashpig, 10FR-dlocal: DLocal IPN failures due to card type lookup - https://phabricator.wikimedia.org/T351673 (10XenoRyet) [20:30:49] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10LandingCheck, 10MW-1.42-notes (1.42.0-wmf.9; 2023-12-12), 10Wikimedia-production-error: Remove calls to GeoIP 1 in Extension:LandingCheck - https://phabricator.wikimedia.org/T352247 (10XenoRyet) 05Open→03Resolved [20:55:29] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: testing..... - https://phabricator.wikimedia.org/T353674 (10greg) [20:55:41] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: testing..... - https://phabricator.wikimedia.org/T353674 (10greg) 05Open→03Invalid [20:57:23] 10Fundraising-Backlog: fail to log adyen apple donation to civi table - https://phabricator.wikimedia.org/T353494 (10AKanji-WMF) p:05Triage→03High Needs investigation, marking as High so we can review most recent data. [20:58:36] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-production-error: Error in fundraising banner: TypeError: Cannot read properties of undefined (reading 'style') - https://phabricator.wikimedia.org/T353475 (10AKanji-WMF) p:05Triage→03Medium [21:06:27] 10Fundraising-Backlog: Find all the places we aren't rounding properly resulting in weird floating point issues - https://phabricator.wikimedia.org/T353378 (10AKanji-WMF) p:05Triage→03Low [21:13:28] 10Fundraising-Backlog, 10Developer Productivity: Migrate fundraising-dev repo to gitlab - https://phabricator.wikimedia.org/T353374 (10AKanji-WMF) Will also add to Asana roadmap to evaluate against other FR-Tech OKR3 priorities [21:53:06] 10Fundraising-Backlog: Reattempt failed Ingencio transactions from Sept, Oct, Nov, Dec 2023 - https://phabricator.wikimedia.org/T353677 (10EMartin) [21:53:41] 10Fundraising-Backlog: Reattempt failed Ingencio transactions from Oct, Nov, Dec 2023 - https://phabricator.wikimedia.org/T353677 (10EMartin) [21:54:06] 10Fundraising-Backlog: Reattempt failed Ingencio transactions from Oct, Nov, Dec 2023 - https://phabricator.wikimedia.org/T353677 (10EMartin) [22:17:48] anilk: - Natalie is asking me on slack about the sending of the EOY receipts - I guess we should have a phab for discussions around sending them at the start of Jan if we don't already -normally they tell us a date & then change it [22:19:49] we do eileen anilk let me find it [22:20:25] https://phabricator.wikimedia.org/T353206 i guess we can rename it to better fit [22:24:44] 10Fundraising-Backlog: Check that end of year email receipt templates are ready for January 2024 send - https://phabricator.wikimedia.org/T353206 (10Cstone) @MSuijkerbuijk_WMF looks like that cid got deduped to cid=34846954. The popup defaults to the previous year so right now it's showing 2022, if you go to the... [22:46:23] 10Fundraising Sprint Byzantine Empire Strikes Back, 10Fundraising Sprint CAPS LOCK CULTS, 10Fundraising Sprint Dampness, 10Fundraising-Backlog, 10Patch-For-Review: 16 Multilingual Thank You Emails - https://phabricator.wikimedia.org/T243098 (10CDenes_WMF) Hi Francesca, No problem. My colleague @Eileenmc... [23:09:19] tx cstone