[00:18:28] 10Fundraising Tech - Chaos Crew, 10FR-Tech-Analytics, 10FR-tech-data-integrity: Odd Refund Recording in Civi - https://phabricator.wikimedia.org/T351651 (10ERoden-WMF) That's great, thanks for the update @Cstone. [00:39:51] (03PS1) 10Wfan: Use ja_01A as default [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/981445 (https://phabricator.wikimedia.org/T329807) [00:41:52] (03CR) 10CI reject: [V: 04-1] Use ja_01A as default [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/981445 (https://phabricator.wikimedia.org/T329807) (owner: 10Wfan) [00:43:42] (03PS2) 10Wfan: Use ja_01A as default [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/981445 (https://phabricator.wikimedia.org/T329807) [00:45:23] (03CR) 10CI reject: [V: 04-1] Use ja_01A as default [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/981445 (https://phabricator.wikimedia.org/T329807) (owner: 10Wfan) [00:49:20] (03PS3) 10Wfan: Use ja_01A as default [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/981445 (https://phabricator.wikimedia.org/T329807) [01:03:43] (03CR) 10Wfan: Add a second redis instance for donor prefs queues (031 comment) [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/980961 (https://phabricator.wikimedia.org/T352928) (owner: 10Ejegg) [01:10:34] (03CR) 10Wfan: [C: 03+2] Add recurring-upgrade to queue consume script [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/981419 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [01:13:33] (03CR) 10Wfan: [C: 03+2] Recurring QC can read from recurring-upgrade queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/980967 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [01:14:56] (03CR) 10Wfan: [C: 03+2] Send recurring upgrade donations to new queue [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/980964 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [01:17:04] (03CR) 10Wfan: [C: 03+2] Add defaults for recurring-upgrade queue [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/980962 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [01:30:17] (03Merged) 10jenkins-bot: Recurring QC can read from recurring-upgrade queue [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/980967 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [01:30:19] (03Merged) 10jenkins-bot: Add defaults for recurring-upgrade queue [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/980962 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [01:48:26] (03PS6) 10Cstone: Audit: fill missing contribution_tracking rows [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/972922 (https://phabricator.wikimedia.org/T339182) (owner: 10Ejegg) [01:53:22] (03CR) 10Cstone: Audit: fill missing contribution_tracking rows (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/972922 (https://phabricator.wikimedia.org/T339182) (owner: 10Ejegg) [02:14:32] thanks for all the CR cstone and wfan! [02:15:02] hah mine was just a question although it otherwise looks good [02:15:49] looking [02:18:17] cstone oh, last was working just fine, but it would loop over the whole table to get there [02:18:51] so with the new sort it just gets the lasy one from the db [02:18:52] ah so it wasn't working like i though tit did [02:19:49] i think anything before execute() can affect the query [02:20:21] but after it would just be grabbing from the rrsult set [02:20:22] i guess i wast thinking some orm magic was doing the sort part to return last [02:21:20] nah, first and last are just convenience functions on the result set [02:29:31] (03CR) 10Cstone: [C: 03+2] "thanks, looks good!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/972922 (https://phabricator.wikimedia.org/T339182) (owner: 10Ejegg) [02:30:25] ejegg: on the auto rescue and to our hard check for two payments within 23 days, Evelyn came back and said that they don't want to move the cycle day so [02:30:43] either we just get rid of it completely? or trying to think of the least complicated way to check now [02:32:07] i guess we could just log it and make a graph or something too [02:32:47] shoot cstone i would prefer to only make the exception for known autorescue [02:33:35] but once its rescued i dont think theres any knowledge it was in that status besides in the log tables? [02:35:42] right, huh [02:36:25] let's see if we come up with anything better overnight [02:36:54] i gotta head out. see you tomorrow! [02:36:58] night! [02:46:09] (03Merged) 10jenkins-bot: Audit: fill missing contribution_tracking rows [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/972922 (https://phabricator.wikimedia.org/T339182) (owner: 10Ejegg) [03:38:26] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10FR-AutoTY-Email, 10FR-donorservices: EoY receipt email donation format issue 2023 - https://phabricator.wikimedia.org/T332232 (10Cstone) Okay I have deleted the 174 duplicate refund lines from the ones fixed in {T351822}. Still need to ch... [03:41:50] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10Recurring-Donations: Auto-Rescue Edge Cases - https://phabricator.wikimedia.org/T350501 (10Cstone) [04:13:04] (03PS2) 10Wangombe: Update reference to namespaced TranslateMetadata::class [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/980983 (https://phabricator.wikimedia.org/T348496) [07:30:51] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/982266 (owner: 10L10n-bot) [11:05:30] 10Fundraising-Backlog, 10acl*WMF-FR: "Extension Upgrades Pending" on Community CRM - https://phabricator.wikimedia.org/T353234 (10Qgil) [14:34:20] (03CR) 10Jgleeson: "I left a question inline related to how we're detecting the constraint scenario." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/982117 (https://phabricator.wikimedia.org/T352321) (owner: 10Ejegg) [14:36:19] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog: Remove the recurring queue threshold alerts - https://phabricator.wikimedia.org/T348533 (10Jgreen) >>! In T348533#9239869, @Ejegg wrote: >> @Elliott Eggleston what was the reason we set up this alert again? >... [14:53:24] 10Fundraising-Backlog: Segmentation - Sample data request - https://phabricator.wikimedia.org/T353264 (10AKanji-WMF) [14:56:58] 10Fundraising-Backlog: Segmentation - Sample data request - https://phabricator.wikimedia.org/T353264 (10AKanji-WMF) [14:58:59] 10Fundraising-Backlog: Segmentation - Sample data request - https://phabricator.wikimedia.org/T353264 (10AKanji-WMF) [15:18:15] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Successful Adyen transactions not making it into CiviCRM - https://phabricator.wikimedia.org/T345736 (10Damilare) @krobinson looks like those transactions have been imported through the audit imports last night. Looking to identify the cause for the delay... [15:30:31] (03CR) 10Ejegg: "Thanks jgleeson, I'll add a new PS with finer targeting" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/982117 (https://phabricator.wikimedia.org/T352321) (owner: 10Ejegg) [16:17:09] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog: Remove the recurring queue threshold alerts - https://phabricator.wikimedia.org/T348533 (10Ejegg) Sure, sounds good to me @Jgreen, then we can close this ticket. [17:13:25] (03PS14) 10Ejegg: Scripts for doing token migration in the db [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/965844 (https://phabricator.wikimedia.org/T344645) [17:17:19] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog: Measure donation processing rate (once we decide what that is) - https://phabricator.wikimedia.org/T342013 (10Jgreen) We scrape some metrics from logs that could be informative and/or useful for comparison re. this task: https://frmon.wikimedia... [17:30:01] (03PS4) 10Ejegg: Add a second redis instance for donor prefs queues [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/980961 (https://phabricator.wikimedia.org/T352928) [17:30:03] (03PS5) 10Ejegg: Network segmentation [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/980974 (https://phabricator.wikimedia.org/T352928) [17:30:05] (03PS3) 10Ejegg: Add recurring-upgrade to queue consume script [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/981419 (https://phabricator.wikimedia.org/T352907) [17:30:16] (03CR) 10Ejegg: "Thanks wfan! Fixed in PS4" [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/980961 (https://phabricator.wikimedia.org/T352928) (owner: 10Ejegg) [17:31:07] (03CR) 10Wfan: [C: 03+2] Add a second redis instance for donor prefs queues [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/980961 (https://phabricator.wikimedia.org/T352928) (owner: 10Ejegg) [17:32:13] !log payments-wiki upgraded from 1d24dc90 to c1181b95 [17:32:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:34:08] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.9; 2023-12-12): Enable variant with no matching gift field for GPay/ApplePay/Venmo - https://phabricator.wikimedia.org/T353064 (10Ejegg) OK, this has been deployed. See the form with variant=noEmployer. That should... [17:35:09] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/982442 [17:35:12] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/982442 (owner: 10Ejegg) [17:36:28] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/982442 (owner: 10Ejegg) [17:42:07] !log fundraising civicrm upgraded from 8c107215 to 834606ef [17:42:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:52:43] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Adyen, 10FR-Smashpig: Adyen SecondChargeback IPN causes refund queue failure - https://phabricator.wikimedia.org/T352035 (10jgleeson) a:03jgleeson [19:20:13] PROBLEM - check_puppetrun on frmon2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 4 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2002&service=check_puppetrun [19:20:17] PROBLEM - check_puppetrun on frmx2001 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 4 minutes ago with 29 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [19:21:13] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 5 minutes ago with 32 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [19:21:13] PROBLEM - check_puppetrun on frdb2002 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 5 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [19:21:15] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 32 seconds ago with 32 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [19:21:17] PROBLEM - check_puppetrun on frauth2002 is CRITICAL: CRITICAL: Puppet has 34 failures. Last run 5 minutes ago with 34 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [19:21:17] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 5 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [19:21:23] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 5 minutes ago with 35 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [19:22:13] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 6 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [19:22:13] PROBLEM - check_puppetrun on frbackup2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 6 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [19:22:13] PROBLEM - check_puppetrun on frdb2003 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 6 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_puppetrun [19:22:17] PROBLEM - check_puppetrun on frqueue2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 6 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [19:22:17] PROBLEM - check_puppetrun on frban2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 6 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [19:22:17] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 6 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [19:22:17] PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 6 minutes ago with 35 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [19:23:13] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 7 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [19:23:13] PROBLEM - check_puppetrun on frbast2002 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 6 minutes ago with 29 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2002&service=check_puppetrun [19:23:23] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 7 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [19:23:23] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 1 minute ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [19:23:49] ^^^ debugging. puppetserver didn't start it appears. [19:25:13] PROBLEM - check_puppetrun on frlog2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 9 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [19:25:13] PROBLEM - check_puppetrun on frmon2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 9 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2002&service=check_puppetrun [19:25:13] (03PS1) 10Eileen: Add dynamic preview to EOY email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/982455 [19:25:17] RECOVERY - check_puppetrun on frmx2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [19:26:11] ok they should start recovering. [19:26:13] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 4 minutes ago with 32 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [19:26:13] PROBLEM - check_puppetrun on frdb2002 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 10 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [19:26:13] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 5 minutes ago with 32 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [19:26:15] PROBLEM - check_puppetrun on frauth2002 is CRITICAL: CRITICAL: Puppet has 34 failures. Last run 4 minutes ago with 34 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [19:26:17] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 4 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [19:26:23] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 10 minutes ago with 35 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [19:26:50] cstone: this is what you were wanting https://gerrit.wikimedia.org/r/982455 [19:27:13] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 11 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [19:27:13] PROBLEM - check_puppetrun on frdb2003 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 11 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_puppetrun [19:27:13] PROBLEM - check_puppetrun on frbackup2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 4 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [19:27:13] PROBLEM - check_puppetrun on frban2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 11 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [19:27:13] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 11 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [19:27:15] PROBLEM - check_puppetrun on frqueue2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 11 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [19:27:15] PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 11 minutes ago with 35 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [19:28:13] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 12 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [19:28:13] PROBLEM - check_puppetrun on frbast2002 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 12 minutes ago with 29 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2002&service=check_puppetrun [19:28:23] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 12 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [19:28:23] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 6 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [19:29:41] hmm, unhappy puppets [19:29:46] ya [19:30:13] RECOVERY - check_puppetrun on frmon2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2002&service=check_puppetrun [19:30:13] PROBLEM - check_puppetrun on frlog2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 14 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [19:31:13] RECOVERY - check_puppetrun on pay-lvs2001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [19:31:13] RECOVERY - check_puppetrun on frdb2002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [19:31:15] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 10 minutes ago with 32 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [19:31:17] RECOVERY - check_puppetrun on frauth2002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [19:31:17] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 9 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [19:31:23] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [19:31:24] they should be recovering soon. puppetserver is back up. [19:31:47] just have to wait for the next runs to complete. [19:32:13] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [19:32:13] RECOVERY - check_puppetrun on frdb2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_puppetrun [19:32:13] RECOVERY - check_puppetrun on frbackup2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [19:32:17] RECOVERY - check_puppetrun on frban2001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [19:32:17] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [19:32:17] RECOVERY - check_puppetrun on frqueue2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [19:32:21] RECOVERY - check_puppetrun on frdata2001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [19:33:13] RECOVERY - check_puppetrun on frqueue2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [19:33:13] RECOVERY - check_puppetrun on frbast2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2002&service=check_puppetrun [19:33:17] RECOVERY - check_puppetrun on payments2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [19:33:17] RECOVERY - check_puppetrun on payments2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [19:33:32] debugging what happened, but the puppetserver tried an operation on start and it failed so it didn't continue. [19:35:13] RECOVERY - check_puppetrun on frlog2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [19:36:13] RECOVERY - check_puppetrun on pay-lvs2002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [19:36:17] RECOVERY - check_puppetrun on payments2002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [19:56:46] fr-tech: just a warning that we are going to reboot frpm2001 again to verify puppetserver behavior on restart. we may see some more errors but i'll hop on the service restart faster if it fails. [20:33:03] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Keep pending message when validation action is set to "Leave in console" - https://phabricator.wikimedia.org/T353292 (10Damilare) [20:35:14] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Smashpig, and 2 others: Cancel auto-rescue at Adyen when donor cancels recurring donation in Civi - https://phabricator.wikimedia.org/T351338 (10AnnWF) [20:36:33] 10Fundraising Sprint - X gon' donate to ya, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Smashpig, 10Recurring-Donations: save retry.rescueReference for future cancel action - https://phabricator.wikimedia.org/T353293 (10AnnWF) [20:39:55] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Keep pending message when validation action is set to "Leave in console" - https://phabricator.wikimedia.org/T353292 (10Damilare) [20:39:57] 10Fundraising-Backlog: Figure out what we want to do with audit data mismatches/missing data - https://phabricator.wikimedia.org/T340066 (10Damilare) [20:46:27] (03CR) 10Ejegg: [V: 03+2] Add a second redis instance for donor prefs queues [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/980961 (https://phabricator.wikimedia.org/T352928) (owner: 10Ejegg) [20:47:36] (03PS2) 10Ejegg: Send recurring upgrade donations to new queue [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/980964 (https://phabricator.wikimedia.org/T352907) [21:00:55] 10Fundraising-Backlog: Auto-Rescue - set next_sched_contribution_date when it goes into the auto rescue flow - https://phabricator.wikimedia.org/T353294 (10Cstone) [21:19:47] (03PS2) 10Wfan: Add Auto Rescue Cancel Api to adyen [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/982155 (https://phabricator.wikimedia.org/T351338) [21:23:00] (03PS3) 10Wfan: Add Auto Rescue Cancel Api to adyen [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/982155 (https://phabricator.wikimedia.org/T351338) [21:23:38] (03CR) 10CI reject: [V: 04-1] Add Auto Rescue Cancel Api to adyen [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/982155 (https://phabricator.wikimedia.org/T351338) (owner: 10Wfan) [21:24:28] (03PS4) 10Wfan: Add Auto Rescue Cancel Api to adyen [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/982155 (https://phabricator.wikimedia.org/T351338) [21:31:21] 10fundraising-tech-ops: Migrate fr-tech-ops documentation from Collab Wiki to Wikitech - https://phabricator.wikimedia.org/T342658 (10Jgreen) [21:43:24] 10Fundraising-Backlog, 10Documentation: fr-tech documentation needed for datacenter cutover procedure - https://phabricator.wikimedia.org/T266810 (10Jgreen) [21:50:55] (03CR) 10Ejegg: "resubmit" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/980964 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [22:32:40] (03CR) 10Ejegg: [C: 03+2] "Re-applying C+2 originally given by wfan" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/980964 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [22:42:23] (03Merged) 10jenkins-bot: Send recurring upgrade donations to new queue [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/980964 (https://phabricator.wikimedia.org/T352907) (owner: 10Ejegg) [23:07:15] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/982490 [23:07:26] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/982490 (owner: 10Ejegg) [23:08:35] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/982490 (owner: 10Ejegg) [23:56:55] !log donorwiki upgraded from f7407053 to bc49e5a6 [23:56:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log