[00:16:13] PROBLEM - check_puppetrun on frdb2002 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 13 seconds ago with 24 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [00:16:13] PROBLEM - check_puppetrun on pay-lvs2001 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 34 seconds ago with 20 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [00:16:15] PROBLEM - check_puppetrun on pay-lvs2002 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 22 seconds ago with 20 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [00:16:17] PROBLEM - check_puppetrun on frmon1001 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 11 minutes ago with 18 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon1001&service=check_puppetrun [00:16:17] PROBLEM - check_puppetrun on payments1006 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 11 minutes ago with 27 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_puppetrun [00:16:17] PROBLEM - check_puppetrun on payments2002 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 45 seconds ago with 27 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [00:16:23] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 59 seconds ago with 23 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [00:17:13] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 1 minute ago with 24 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [00:17:13] PROBLEM - check_puppetrun on frdb2003 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 1 minute ago with 24 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_puppetrun [00:17:15] PROBLEM - check_puppetrun on frqueue2002 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 1 minute ago with 18 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [00:17:17] PROBLEM - check_puppetrun on frmon2001 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 1 minute ago with 18 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2001&service=check_puppetrun [00:18:00] ^^^ these are known. should have been caught by downtime. [00:18:13] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 2 minutes ago with 18 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [00:18:17] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [00:18:17] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [00:18:52] ah. the downtime just expired. re-added. [00:19:01] sorry about the noise. [00:21:13] RECOVERY - check_puppetrun on frdb2002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [00:21:17] RECOVERY - check_puppetrun on frmon1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon1001&service=check_puppetrun [00:21:17] RECOVERY - check_puppetrun on payments1006 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_puppetrun [00:22:13] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [00:27:13] RECOVERY - check_puppetrun on frdb2003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_puppetrun [00:27:17] RECOVERY - check_puppetrun on frmon2001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2001&service=check_puppetrun [00:31:13] RECOVERY - check_puppetrun on pay-lvs2001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2001&service=check_puppetrun [00:31:23] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [00:32:17] RECOVERY - check_puppetrun on frqueue2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [00:33:13] RECOVERY - check_puppetrun on frqueue2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [00:33:23] RECOVERY - check_puppetrun on payments2001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_puppetrun [00:33:33] hmm, failstorm? [00:34:18] oh, back to normal now? [00:35:20] thanks for the code review wfan! [00:36:15] RECOVERY - check_puppetrun on pay-lvs2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs2002&service=check_puppetrun [00:36:17] RECOVERY - check_puppetrun on payments2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_puppetrun [00:38:02] np~ I just realized that I forget to renew my old car's registration and when I drove the old car back after the solar panel guys leave my drive way I find the sticker color is still 2022, and today is the last day for the final late payment, while still not able to pay online as over 75 days, wait for their call now 😅 [00:38:17] RECOVERY - check_puppetrun on payments2003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [00:38:25] ah dang, I hope you can get that in on time! [00:39:59] (03PS1) 10Ejegg: Update smash-pig dependency and own version [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883291 [00:40:02] (03CR) 10Ejegg: [C: 03+2] Update smash-pig dependency and own version [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883291 (owner: 10Ejegg) [00:43:38] (03Merged) 10jenkins-bot: Update smash-pig dependency and own version [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883291 (owner: 10Ejegg) [00:51:48] (03PS2) 10Ejegg: New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) [00:53:13] fr-tech: we are in a steady state for the puppetmaster swap. please use frpm1002 for anything you used to use frpm1001 for and don't hesitate to reach out if anything goes awry. [00:53:52] eileen: i will have to do a run to pick up the dog and possibly a run to get a kid but should be available to do the civi upgrade in a little if you are still game for it. [01:04:09] Ha, turned out I paid just they mailed the sticker to old address, so they will mail a new one to me, that's an awesome result 🤩 [01:05:07] nice, wfan! [01:06:13] (03CR) 10CI reject: [V: 04-1] New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [01:08:12] hah, the kid has a new all-purpose answer. Any question with 'when' gets 'dos semanas', i.e. two weeks [01:09:16] lol and my new sticker will be arrived in 2 weeks [01:09:23] :) [01:17:15] dwisehaupt: yep [01:20:45] dos semanas and not quince días? [01:26:21] yeah that [01:51:57] (03PS3) 10Ejegg: New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) [01:53:30] (03CR) 10Wfan: "Since there is no documentation from api https://epayments-api.developer-ingenico.com/s2sapi/v1/en_US/php/hostedcheckouts/get.html?payment" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883248 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [01:54:06] (03CR) 10Wfan: Split full_name for use in minfraud queries (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883248 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [01:59:43] eileen: i'm going to have to do the kid pickup. so i'll be available to start the upgrade in around 75 mins or so. [01:59:46] https://www.timeanddate.com/worldclock/meetingdetails.html?year=2023&month=1&day=25&hour=3&min=15&sec=0&p1=224&p2=22 [02:02:58] (03PS4) 10Ejegg: Ingenico: get name from iframe, not our field [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/882684 (https://phabricator.wikimedia.org/T312877) [02:03:12] (03PS2) 10Ejegg: Split full_name for use in minfraud queries [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883248 (https://phabricator.wikimedia.org/T312877) [02:06:13] (03CR) 10CI reject: [V: 04-1] New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [02:08:39] (03CR) 10Ejegg: "Thanks for the review, and good catch on that copypasta!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883248 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [02:09:05] cool cool [02:13:23] (03PS4) 10Ejegg: New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) [02:26:48] (03CR) 10Ejegg: "This is looking pretty good! Just a couple things I think we can push down to the SmashPig layer." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) (owner: 10Damilare Adedoyin) [02:33:45] (03CR) 10Ejegg: "So I'm not sure now that we should have this configuration up at the DonationInterface layer." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/874932 (https://phabricator.wikimedia.org/T324276) (owner: 10Wfan) [02:37:44] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/883295 [02:38:33] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/883295 (owner: 10Eileen) [02:39:35] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fr-donorservices: Add check to pending transaction resolver to not resolve more than one donation per donor per run - https://phabricator.wikimedia.org/T326361 (10Eileenmcnaughton) @Ejegg I am including a patch from this in deploy - does that mean thi... [02:59:51] eileen: I think that's OK to go out! [03:00:01] It's passing tests, anyway :) [03:00:17] cool - did I get it right on the other channel? [03:05:33] !log config revision changed from 3f641fce to dc0a0d3a [03:05:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:27:23] !log civicrm upgraded from f6093fb2 to 9197ca29 [03:27:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:32:24] eileen: ejegg: heyyy back at the keyboard, suggestions for stuff to work on? [03:32:44] AndyRussG: I don't have any review ready - just updating civi [03:33:06] okiii have fun :) lmk if I can help in any way [03:37:17] ahh gonna test out mw 1.39 [03:37:50] (03PS1) 10Eileen: Triggers update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883296 [03:44:27] (03PS2) 10Eileen: Triggers update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883296 [03:47:54] (03CR) 10Dwisehaupt: [C: 03+2] "Much nicer. Shipit." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883296 (owner: 10Eileen) [03:56:45] 10Fundraising Sprint Bridge over troubled Wifi, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: Upgrade Civi to latest point version now BE is over - https://phabricator.wikimedia.org/T326272 (10Eileenmcnaughton) I hit one issue during upgrade https://lab.civicrm.org/dev/core/-/... [03:59:53] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/883297 [04:00:02] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/883297 (owner: 10Eileen) [04:01:53] !log civicrm upgraded from 9197ca29 to 3e6b21b6 [04:01:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:10:05] !log config revision changed from dc0a0d3a to 089d0acb [04:10:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:33:30] 10Fundraising-Backlog: Update config for Payments Staging - https://phabricator.wikimedia.org/T327855 (10AndyRussG) [06:48:47] fr-tech anyone around? [06:51:14] I shouldn't be, but I am [06:52:22] Kind of [06:54:22] XenoRyet: ohh heyy thanks for the reply [06:54:31] hope everything's ok! [06:54:48] would you like to check a UBN but not-very-grave localsettings change? [06:54:53] no worries if not [06:55:03] or I could just explain it to you and if you're good I'll just push it out [06:55:15] hope Dylan's feeling better! [06:55:29] Yea, little Dylan had a good day today. No fever, so she can probably go to school tomorrow.\ [06:55:49] ah great! [06:56:35] the change is on frpm1002 (the new puppet master). Just finishing the task explaining it [06:56:56] tl;dr we forgot to push out a config change and GPay is broken, but for some reason it's failing silently [06:58:32] I'm not really set to do code review of any kind, but if it's broken and you think you have a fix go ahead and push it out. Can't get more broke than failing silently, right? [06:59:47] 10Fundraising Tech - Chaos Crew: Fix GPay issue due to missing config change - https://phabricator.wikimedia.org/T327857 (10AndyRussG) p:05Triage→03Unbreak! [07:00:05] yeah oki thanks! [07:00:07] https://phabricator.wikimedia.org/T327857 [07:01:14] the diff on localsettings is to add this key: 'GoogleScript' => 'https://pay.google.com/gp/p/js/pay.js', [07:08:15] !log updated payments (config only) revision 15395d05, config 418160e9 [07:08:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:08:43] XenoRyet: fixed! thanks much! [07:08:57] No worries. [07:09:02] Bed time now ;-) [07:17:08] PROBLEM - check_log_messages on frav1002 is CRITICAL: reading conf /etc/check_log_messages.conf https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [07:18:25] 'night! :) [07:19:41] 10Fundraising Tech - Chaos Crew: Fix GPay issue due to missing config change - https://phabricator.wikimedia.org/T327857 (10AndyRussG) [07:20:26] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Fix GPay issue due to missing config change - https://phabricator.wikimedia.org/T327857 (10AndyRussG) [07:22:08] RECOVERY - check_log_messages on frav1002 is OK: reading conf /etc/check_log_messages.conf https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [07:24:31] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: GPay issue due to missing config update [fixed] - https://phabricator.wikimedia.org/T327857 (10AndyRussG) [07:32:10] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: GPay issue due to missing config update [fixed] - https://phabricator.wikimedia.org/T327857 (10AndyRussG) [07:33:19] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/883344 (owner: 10L10n-bot) [07:39:13] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: GPay issue due to missing config update [fixed] - https://phabricator.wikimedia.org/T327857 (10AndyRussG) [07:46:41] (03CR) 10Nikerabbit: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/883344 (owner: 10L10n-bot) [08:22:31] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: GPay issue due to missing config update [fixed] - https://phabricator.wikimedia.org/T327857 (10AndyRussG) [10:11:13] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10jgleeson) @Eileenmcnaughton I can't see any timeout controls in the Imap class you link to? [11:25:30] (03CR) 10Jgleeson: "Thanks! Working well for me. I've left a few comments inline." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [12:51:49] (03CR) 10Jgleeson: New name flow for Ingenico pending resolver (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [13:14:14] (03CR) 10Jgleeson: "Thanks for the reivew! I've added the mapping suggested in the next PS" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/881700 (https://phabricator.wikimedia.org/T324281) (owner: 10Jgleeson) [13:14:45] (03PS13) 10Jgleeson: Add dlocal API capturePayment call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/881700 (https://phabricator.wikimedia.org/T324281) [13:16:48] (03PS14) 10Jgleeson: Add dlocal API capturePayment call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/881700 (https://phabricator.wikimedia.org/T324281) [13:17:49] (03PS5) 10Jgleeson: Add dlocal approvePayment call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/883268 (https://phabricator.wikimedia.org/T324281) [13:49:16] (03PS14) 10Damilare Adedoyin: Handle card submission in DLocal in DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) [13:49:33] (03CR) 10CI reject: [V: 04-1] Handle card submission in DLocal in DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) (owner: 10Damilare Adedoyin) [13:51:15] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 18 seconds ago with 5 failures. Failed resources (up to 3 shown): Package[systemd-timesyncd],Package[libdbd-mariadb-perl],Package[nginx],Package[nginx-full] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [13:56:17] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 36 seconds ago with 1 failures. Failed resources (up to 3 shown): Mount[/srv/archive/banner_logs] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [14:01:15] PROBLEM - check_ipsec on fran1001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: frban1001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_ipsec [14:01:15] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/srv/archive/banner_logs] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [14:02:15] PROBLEM - check_ipsec on frban1001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran1001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_ipsec [14:06:15] PROBLEM - check_ipsec on fran1001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: frban1001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_ipsec [14:06:15] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/srv/archive/banner_logs] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [14:07:15] PROBLEM - check_ipsec on frban1001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran1001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_ipsec [14:12:15] RECOVERY - check_ipsec on frban1001 is OK: Strongswan OK - 1 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_ipsec [14:16:11] RECOVERY - check_ipsec on fran1001 is OK: Strongswan OK - 1 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_ipsec [14:16:15] RECOVERY - check_puppetrun on fran1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [14:40:42] 10fundraising-tech-ops, 10FR-Tech-Analytics: Upgrade Fundraising Superset to 1.5.3 - https://phabricator.wikimedia.org/T311540 (10Jgreen) 05Open→03Resolved [15:02:14] (03PS15) 10Damilare Adedoyin: Handle card submission in DLocal in DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) [15:03:50] (03CR) 10CI reject: [V: 04-1] Handle card submission in DLocal in DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) (owner: 10Damilare Adedoyin) [15:16:01] (03PS16) 10Damilare Adedoyin: Handle card submission in DLocal in DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) [15:17:38] (03CR) 10CI reject: [V: 04-1] Handle card submission in DLocal in DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) (owner: 10Damilare Adedoyin) [15:25:35] (03PS17) 10Damilare Adedoyin: Handle card submission in DLocal in DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) [15:27:02] (03CR) 10CI reject: [V: 04-1] Handle card submission in DLocal in DonationInterface [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) (owner: 10Damilare Adedoyin) [15:42:11] PROBLEM - check_ipsec on frban2001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran2001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_ipsec [15:44:51] (03PS1) 10Damilare Adedoyin: Remove Payment_method and Payment_submethod from required fields in Dlocal createPayment [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/883594 (https://phabricator.wikimedia.org/T324279) [15:47:09] (03CR) 10Ejegg: [C: 03+1] "Looks good, just one question" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/883594 (https://phabricator.wikimedia.org/T324279) (owner: 10Damilare Adedoyin) [15:47:11] PROBLEM - check_ipsec on frban2001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran2001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_ipsec [15:48:09] PROBLEM - Host fran1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:49:31] PROBLEM - Host frdata1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:49:31] PROBLEM - Host frlog1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:49:41] PROBLEM - Host frauth1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:49:42] PROBLEM - Host frbast1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:49:45] PROBLEM - Host frdev1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:49:46] PROBLEM - Host frdev1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:49:46] PROBLEM - Host frdb1005 is DOWN: PING CRITICAL - Packet loss = 100% [15:49:46] PROBLEM - Host frpig1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:01] PROBLEM - Host payments1008 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:19] PROBLEM - Host frban1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:39] PROBLEM - Host payments1005 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:51] PROBLEM - Host frav1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:51] PROBLEM - Host frmx1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:52] PROBLEM - Host frmon1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:52] PROBLEM - Host frnetmon1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:52] PROBLEM - Host frdb1004 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:52] PROBLEM - Host frqueue1004 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:53] PROBLEM - Host frpm1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:53] PROBLEM - Host pay-lvs1003 is DOWN: PING CRITICAL - Packet loss = 100% [15:50:53] PROBLEM - Host frdb1003 is DOWN: PING CRITICAL - Packet loss = 100% [15:51:19] PROBLEM - Host frpm1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:51:27] PROBLEM - Host frqueue1003 is DOWN: PING CRITICAL - Packet loss = 100% [15:51:55] PROBLEM - Host civi1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:52:11] PROBLEM - check_ipsec on frban2001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran2001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_ipsec [15:53:13] PROBLEM - check_redis on frqueue2001 is CRITICAL: CRITICAL: replication_delay is 338 300 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 5 keys, up 22 days 1 hours - replication_delay is 338, memory use is 1.86M (peak 7.95M, 0.07% of max, fragmentation 3.32%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_redis [15:53:13] PROBLEM - check_mysql on payments2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_mysql [15:53:14] PROBLEM - check_redis_donor_prefs on frqueue2001 is CRITICAL: CRITICAL: replication_delay is 345 300 - REDIS 5.0.14 on 127.0.0.1:6380 has 0 databases (), up 22 days 1 hours - replication_delay is 345, memory use is 1.84M (peak 1.88M, 0.07% of max, fragmentation 3.29%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_redis_donor_prefs [15:53:14] PROBLEM - check_mysql on payments2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_mysql [15:53:37] 10Fundraising-Backlog, 10Python3-Porting: modernize DjangoBannerStats to python3 - https://phabricator.wikimedia.org/T301905 (10greg) 07:49:59 fr-tech is there anyone who can look at https://phabricator.wikimedia.org/T301905? The urgency has just gone up since we're upgrading the analytics serve... [15:56:11] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [15:56:13] PROBLEM - check_mysql on payments2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_mysql [15:57:09] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [15:57:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [15:57:09] PROBLEM - check_ipsec on frban2001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran2001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_ipsec [15:57:17] PROBLEM - check_mysql on frdata2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_mysql [15:57:19] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 582 300 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 5 keys, up 1 days 15 hours - replication_delay is 582, memory use is 1.87M (peak 3.71M, 0.09% of max, fragmentation 3.99%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [15:57:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 588 300 - REDIS 6.0.16 on 127.0.0.1:6380 has 0 databases (), up 1 days 15 hours - replication_delay is 588, memory use is 1.85M (peak 1.95M, 0.08% of max, fragmentation 3.46%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [15:58:11] RECOVERY - Host payments1005 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [15:58:13] PROBLEM - check_redis on frqueue2001 is CRITICAL: CRITICAL: replication_delay is 639 300 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 5 keys, up 22 days 1 hours - replication_delay is 639, memory use is 1.86M (peak 7.95M, 0.07% of max, fragmentation 3.32%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_redis [15:58:13] PROBLEM - check_mysql on payments2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_mysql [15:58:13] PROBLEM - check_mysql on payments2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_mysql [15:58:14] PROBLEM - check_redis_donor_prefs on frqueue2001 is CRITICAL: CRITICAL: replication_delay is 645 300 - REDIS 5.0.14 on 127.0.0.1:6380 has 0 databases (), up 22 days 1 hours - replication_delay is 645, memory use is 1.84M (peak 1.88M, 0.07% of max, fragmentation 3.29%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_redis_donor_prefs [15:58:15] RECOVERY - Host frnetmon1001 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [15:58:15] RECOVERY - Host frpig1001 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [15:58:15] RECOVERY - Host frpm1002 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [15:58:16] RECOVERY - Host frqueue1003 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [15:58:16] RECOVERY - Host frlog1002 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [15:58:17] RECOVERY - Host frdb1004 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [16:00:11] RECOVERY - Host frdata1002 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms [16:00:13] RECOVERY - Host payments1008 is UP: PING OK - Packet loss = 0%, RTA = 1.33 ms [16:00:13] RECOVERY - Host frdev1001 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [16:00:13] RECOVERY - Host frdb1005 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [16:00:14] RECOVERY - Host fran1001 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [16:00:15] RECOVERY - Host civi1001 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [16:00:41] RECOVERY - Host frauth1002 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [16:00:41] RECOVERY - Host frbast1001 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [16:00:47] RECOVERY - Host frdev1002 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [16:01:01] RECOVERY - Host frban1001 is UP: PING OK - Packet loss = 0%, RTA = 1.33 ms [16:01:11] RECOVERY - Host frpm1001 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [16:01:13] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 56359 Threads: 11 Questions: 1194376 Slow queries: 44 Opens: 2267 Flush tables: 1 Open tables: 1083 Queries per second avg: 21.192 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [16:01:13] RECOVERY - check_mysql on payments2002 is OK: Uptime: 55607 Threads: 5 Questions: 92839 Slow queries: 0 Opens: 96 Open tables: 90 Queries per second avg: 1.669 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_mysql [16:01:15] RECOVERY - Host frmon1001 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [16:01:37] RECOVERY - Host frav1002 is UP: PING OK - Packet loss = 0%, RTA = 0.56 ms [16:01:38] RECOVERY - Host frmx1001 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [16:01:39] RECOVERY - Host frdb1003 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [16:01:39] RECOVERY - Host frqueue1004 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [16:01:40] RECOVERY - Host pay-lvs1003 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [16:02:09] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 56574 Threads: 10 Questions: 1197419 Slow queries: 43 Opens: 2353 Flush tables: 1 Open tables: 1170 Queries per second avg: 21.165 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [16:02:09] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 56283 Threads: 4 Questions: 9144315 Slow queries: 26 Opens: 2973 Open tables: 1579 Queries per second avg: 162.470 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [16:02:15] RECOVERY - check_mysql on frdata2001 is OK: Uptime: 1905347 Threads: 9 Questions: 490799 Slow queries: 0 Opens: 93 Flush tables: 1 Open tables: 87 Queries per second avg: 0.257 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_mysql [16:02:19] RECOVERY - check_redis on frqueue2002 is OK: OK: REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 4 keys, up 1 days 15 hours - replication_delay is 2, memory use is 1.87M (peak 3.71M, 0.09% of max, fragmentation 3.98%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [16:02:19] RECOVERY - check_redis_donor_prefs on frqueue2002 is OK: OK: REDIS 6.0.16 on 127.0.0.1:6380 has 0 databases (), up 1 days 15 hours - replication_delay is 8, memory use is 1.85M (peak 1.95M, 0.08% of max, fragmentation 3.56%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [16:02:56] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10SRE, 10netops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10Papaul) [16:03:15] RECOVERY - check_redis on frqueue2001 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 5 keys, up 22 days 1 hours - replication_delay is 6, memory use is 1.87M (peak 7.95M, 0.07% of max, fragmentation 3.33%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_redis [16:03:15] RECOVERY - check_redis_donor_prefs on frqueue2001 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6380 has 0 databases (), up 22 days 1 hours - replication_delay is 3, memory use is 1.84M (peak 1.88M, 0.07% of max, fragmentation 3.29%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_redis_donor_prefs [16:03:16] RECOVERY - check_mysql on payments2001 is OK: Uptime: 55810 Threads: 5 Questions: 93107 Slow queries: 0 Opens: 98 Open tables: 92 Queries per second avg: 1.668 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_mysql [16:03:16] RECOVERY - check_mysql on payments2003 is OK: Uptime: 55731 Threads: 5 Questions: 91915 Slow queries: 0 Opens: 96 Open tables: 90 Queries per second avg: 1.649 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_mysql [16:03:40] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10SRE, 10netops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10Papaul) 05Open→03Resolved This is complete. [16:06:48] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10SRE, 10netops: Set consistent MTUs - https://phabricator.wikimedia.org/T315838 (10ayounsi) 05Open→03Resolved All done! [16:47:11] RECOVERY - check_ipsec on frban2001 is OK: Strongswan OK - 1 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_ipsec [17:10:58] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Update SmashPig currency rates - https://phabricator.wikimedia.org/T326671 (10HNordeenWMF) thank you @Ejegg ! [17:33:51] fr-tech anilk here's the alternate, bigger task for getting rid of a big chunk of the data from pgheres/DjangoBannerStats: https://phabricator.wikimedia.org/T296221 [17:34:46] sadly it wouldn't replace everything in pgheres/DjangoBannerStats, though. ^ the proposal in that task would only replace the pgheres table that holds data about banner impressions [17:36:12] and there other tables there, too, though I'm not sure how much, if at all, they're still used. in any case we could still revive FRUEC (the replacement for DjangoBannerStats that was shelved) to handle those remaining non-banner-impression tables [17:36:37] anilk: the analytics stuff isn't currently on the fr-tech architecture diagram as we've don't typically update those systems across our team. we could make an analyics specific diagram but it would probably be best for the analytics folks to author that as they will likely have the best "vision" of how their architecture exists alongside ours [17:36:56] jgleeson: +1 ^ [17:37:24] the systems the analytics folks use primarily interacts with us by interacting with databases we write to and I think they might pull from other sources [17:37:41] anilk: just to add to what jgleeson said, the DjangoBannerStats doesn't really interact with hardly anything in our stack [17:37:47] jgleeson: jinx ;p [17:38:04] :) [17:38:05] well it interacts a bit [17:38:21] it'd be a mostly isolated branch of stuff on that diagram [17:39:06] for e-mail clicks it gets data from the landing pages on Donate wiki [17:43:37] 10Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for ehughes - https://phabricator.wikimedia.org/T327699 (10Dwisehaupt) Certificate renewed and sent via email. Password sent via SMS. [17:55:51] (03PS6) 10Jgleeson: Add dlocal approvePayment call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/883268 (https://phabricator.wikimedia.org/T324281) [17:58:38] (03PS7) 10Jgleeson: Add dlocal approvePayment call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/883268 (https://phabricator.wikimedia.org/T324281) [18:12:15] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1004:down https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [18:17:15] RECOVERY - check_kafkatee on frban1001 is OK: OK: brokers:9 topics:1 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [18:37:24] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1005:down https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [18:41:51] (03PS5) 10Ejegg: New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) [18:42:13] RECOVERY - check_kafkatee on frban1001 is OK: OK: brokers:9 topics:1 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [18:45:02] (03PS6) 10Ejegg: New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) [18:47:03] (03CR) 10Ejegg: "Thanks for the code review! I've addressed most of your concerns in the next coupld of patch sets. I agree that this is getting a bit larg" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [18:59:27] Thanks AndyRussG and jgleeson. Silly question, but for my own understanding: Central Notice queues and deploys banners, but can measure impressions, right? Are there broader wikipedia site analytics that banner impressions could just be part of/pulled from? [19:02:13] PROBLEM - check_kafkatee on frban1001 is CRITICAL: CRITICAL: kafka-jumbo1006:down https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [19:07:06] anilk: yes exactly. in fact that's the "Druid banner_activity_minutely" dataset mentioned in T296221 [19:07:07] T296221: Investigate a job to copy data from Druid banner_activity_minutely to a database on the FR cluster - https://phabricator.wikimedia.org/T296221 [19:07:13] RECOVERY - check_kafkatee on frban1001 is OK: OK: brokers:9 topics:1 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [19:07:38] the main analytics cluster is massive due to the huge volume of data it has to process [19:08:02] and Druid is one of the systems they have for running fast queries on that huge volume of data [19:08:55] and so that Druid system on the main cluster actually has a special dataset for banner impression data, which works really well, can be used in dashboards, etc. like other stuff on the main analytics cluster [19:09:13] it's orders of magnitude superior to pgheres [19:10:03] the only advantage pgheres has is that it's on our cluster so Advancement Analytics can run queries that join it to the Advancement-specific data that we don't allow outside the Fundraising cluster for privace reasons [19:10:40] sorry Advancement and Fundraising mostly interchangeable in my brain, and in this particular case too I guess [19:14:26] so not a silly question :) indeed that task is exactly about pulling from that better source instead [19:16:04] though to be very specific, it's still coming from CentralNotice at the source. The pipeline is convoluted, but in a nutshell, CentralNotice is responsible for sending the initial data from the user's browser, then it goes to various systems on the main analytics cluster, and is processed there. for pgheres, we take data from the main analytics cluster, but before it's hardly [19:16:06] processed at all, and then sample it to put it in pgheres [19:16:45] and that taking it from Druid would involve taking after the main Analytics cluster has done more stuff to it, which would be better, since we wouldn't have to sample it [19:17:07] happy to walk u through all these crazy details anytime ofc [19:19:27] (03PS7) 10Ejegg: New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) [19:22:15] Thanks for the review wfan - I fixed that typo you found, and left a long message on the patch about why I thought the simple split was the appropriate solution there [19:22:21] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/883248 [19:32:33] (03CR) 10Wfan: [C: 03+2] Split full_name for use in minfraud queries (032 comments) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883248 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [19:32:56] thanks wfan! Have you had a chance to look at the parent patch? [19:33:10] Looking [19:33:41] (03PS1) 10Ejegg: Rename 'token' param in getLatestPaymentStatus [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/883651 (https://phabricator.wikimedia.org/T324642) [19:45:19] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10Eileenmcnaughton) @jgleeson - no they feel out during upgrade in favour of https://phabricator.wikimedia.org/T327225 [19:45:49] really really appreciate the context/explanations AndyRussG! [19:46:16] :) [19:46:42] anilk: the data stuff is fun! you could get access to the main cluster analytics dashboards I imagine, if you're interested [19:48:03] 10Fundraising-Backlog: Unhurt our brains - activity_date_time - https://phabricator.wikimedia.org/T326606 (10Eileenmcnaughton) [19:49:36] (03PS16) 10Jgleeson: Add dlocal API capturePayment call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/881700 (https://phabricator.wikimedia.org/T324281) [19:50:04] (03PS17) 10Jgleeson: Add dlocal API capturePayment call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/881700 (https://phabricator.wikimedia.org/T324281) [19:50:36] (03PS10) 10Jgleeson: Add dlocal approvePayment call [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/883268 (https://phabricator.wikimedia.org/T324281) [19:54:10] PROBLEM - Host frdb1005 is DOWN: PING CRITICAL - Packet loss = 100% [19:55:14] RECOVERY - Host frdb1005 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [19:55:26] 10Fundraising Tech - Chaos Crew: Fail Mail (civi1001) run-job: Fetch CiviMail Bounces failed with code - https://phabricator.wikimedia.org/T323057 (10jgleeson) Is it too late to say, I'm not in favour of the new one :) [19:56:03] (03CR) 10Wfan: New name flow for Ingenico pending resolver (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [19:59:17] (03PS2) 10Ejegg: Rename 'token' params in PayPal API calls [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/883651 (https://phabricator.wikimedia.org/T324642) [20:02:05] (03CR) 10Ejegg: New name flow for Ingenico pending resolver (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [20:03:13] (03CR) 10Wfan: [C: 03+2] "tested and get the log with "cardholderName":"xxxxx" also again, there is no restriction for this fullname from iframe, so we can not full" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/882684 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [20:04:20] eileen: [20:04:23] your meeting [20:04:27] opps [20:04:33] oops also! [20:04:33] :> [20:08:27] (03Merged) 10jenkins-bot: Ingenico: get name from iframe, not our field [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/882684 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [20:08:32] (03Merged) 10jenkins-bot: Split full_name for use in minfraud queries [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883248 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [20:08:47] thanks for the review wfan! [20:12:38] np~ but based on the test, the full name part from ingenico is not limited to three parts, or one string without space, as long we we have more than 2 chars it will pass, so hope most ppl enter the card holder name in a good format~ [20:17:36] Jeff_Green: dwisehaupt are you all set for civi-scaling [20:17:44] in about an hour [20:17:46] damilare: I can see you pushed up a new patch after the first round of feedback on this patch but could you also mark them as resolved on this list? https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/881719/ it just makes it a bit easier to pick review back up. thanks in advance! [20:18:03] eileen: yup [20:18:33] scale mount civi! [20:19:21] (03CR) 10Damilare Adedoyin: "Thanks jgleeson, done now" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/881719 (https://phabricator.wikimedia.org/T324290) (owner: 10Damilare Adedoyin) [20:28:31] 10Fundraising-Backlog, 10Python3-Porting: modernize DjangoBannerStats to python3 - https://phabricator.wikimedia.org/T301905 (10Jgreen) For what it's worth, I did a little poking at this on a virtualbox bullseye box and got LoadLPImpressions to stop faceplanting on startup by adjusting several minorish things:... [20:38:11] thanks dwisehaupt [20:38:13] oops [20:38:21] thanks damilare* [20:38:22] :) [20:39:45] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Add gateway label to 'IP repeat count' grafana graph - https://phabricator.wikimedia.org/T327958 (10jgleeson) [20:48:45] (03PS1) 10Eileen: Port upstream version of imap change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883656 (https://phabricator.wikimedia.org/T327225) [20:48:55] jgleeson: ^^ [20:49:01] !log updated employers.csv on paymentswiki [20:49:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:50:20] 10Fundraising-Backlog, 10fr-matching-gifts, 10Epic: Perform CPR on Matching Gifts process. - https://phabricator.wikimedia.org/T273551 (10Cstone) [20:50:22] 10Fundraising-Backlog, 10fr-matching-gifts, 10Documentation: Document matching gifts employer data deployment process - https://phabricator.wikimedia.org/T273559 (10Cstone) https://wikitech.wikimedia.org/wiki/Fundraising/Cluster/Deployments#Matching_gifts_employers_list [20:50:29] 10Fundraising-Backlog, 10fr-matching-gifts, 10Documentation: Document matching gifts employer data deployment process - https://phabricator.wikimedia.org/T273559 (10Cstone) 05Open→03Resolved a:03Cstone [21:24:32] eileen I will be a few minutes late (max 10) for the meeting, I've gotta drop my son off at work [21:35:09] checkin eileen [21:47:37] hmm eileen the local civicrm unit tests seem to be taking longer than I remember [21:47:46] how long [21:48:17] snail pace atm [21:49:39] I'd say it's taken about 5 minutes to run the first 20% [21:50:05] .............civicrm.wmf.WARNING: CVV score mismatch for order_id order-1905461516. Front end score 80.0, pending resolver score 50. Please check that cvv_map settings are consistent. [21:50:07] .................................................. 63 / 600 ( 10%) [21:50:09] .....................................F......................... 126 / 600 ( 21%) [21:50:47] I'll check if debugging autostart is enabled [21:51:19] it was, I'll disable it and start again [21:51:25] * jgleeson crosses fingers [21:51:26] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Try creating a new activity_type_id for Thank yous & see if the UI is still fine - if so switch - https://phabricator.wikimedia.org/T327963 (10Eileenmcnaughton) [21:52:38] Civi\Wmf\Merge [21:52:40] ✔ Merge hook with data set #0 [21:52:42] ✔ Merge hook with data set #1 [21:52:44] ✔ Merge email non primary with data set #0 [21:52:46] those are moving slow [21:54:54] taking the dog who just woke up out for a walk. back in a few. [21:54:55] ok they take 7m30s on CI so I guess it's normal [21:55:05] according to https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/9179/console [21:55:14] I feel like they used to be quicker than that [21:55:47] the batch merge tests seems to be a bottleneck for me [22:02:53] (03CR) 10Jgleeson: [C: 03+2] "LGTM! I can see this new setting gets mapped to the underlying component and tests pass on CI. Thanks" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883656 (https://phabricator.wikimedia.org/T327225) (owner: 10Eileen) [22:07:22] 10Fundraising Tech - Chaos Crew, 10MW-1.40-notes (1.40.0-wmf.21; 2023-01-30), 10Patch-For-Review: Name changes required for Ingenico 3DS2.0 - https://phabricator.wikimedia.org/T312877 (10Ejegg) [22:08:12] wfan: I'm feeling like https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/883252/ is in a +2able state. how about you? [22:08:48] 10Fundraising Sprint Amazing grep, 10Fundraising Sprint Bridge over troubled Wifi, 10Fundraising-Backlog, 10FR-Japan, and 2 others: Japan Form Variations for Testing for Q3 - https://phabricator.wikimedia.org/T322793 (10Ejegg) Pulling back into 'Doing' because it sounds like people really do want to get th... [22:09:10] ok, I just saw the reply from elliott, I am good with plus 2 then~ [22:10:19] awesome [22:10:37] (03CR) 10Jgleeson: [C: 03+2] "This LGTM and wfan! thanks" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [22:13:20] 10Fundraising Sprint Bridge over troubled Wifi, 10Fundraising-Backlog: change default to Adyen for Japan - https://phabricator.wikimedia.org/T326676 (10EMartin) 05Open→03Resolved a:03EMartin [22:13:59] yay, thanks wfan and jgleeson! [22:14:57] ok, so... that pending txn resolver patch SHOULD be backwards compatible with the current way Ingenico works [22:17:21] (03Merged) 10jenkins-bot: Port upstream version of imap change [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883656 (https://phabricator.wikimedia.org/T327225) (owner: 10Eileen) [22:20:31] have a good evening fr-tech o/ [22:25:13] (03Merged) 10jenkins-bot: New name flow for Ingenico pending resolver [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883252 (https://phabricator.wikimedia.org/T312877) (owner: 10Ejegg) [22:29:10] k, lemme prep a deploy [22:29:29] cstone: that will also bring in new audit code [22:29:59] ok ejegg [22:32:18] (03PS1) 10Ejegg: Put resolvable methods in one place [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883669 [22:34:58] (03PS1) 10Ejegg: Check for success on getStatus call [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883670 [22:37:50] (03PS1) 10Ejegg: Update DonationInterface and SmashPig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/883671 [22:38:03] (03CR) 10Ejegg: [C: 03+2] Update DonationInterface and SmashPig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/883671 (owner: 10Ejegg) [22:38:52] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/883672 [22:38:55] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/883672 (owner: 10Ejegg) [22:40:19] dwisehaupt: so I'll be free after 5PM PST for the payments-wiki upgrade. Is that going to be OK? [22:46:53] ejegg: yeah that is fine. i should be home around then after dropping $kid off at their DJ gig. [22:51:46] (03Merged) 10jenkins-bot: Update DonationInterface and SmashPig [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/883671 (owner: 10Ejegg) [23:04:35] (03PS1) 10Ejegg: Send phonetic name fields through on queue messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883673 (https://phabricator.wikimedia.org/T322793) [23:06:32] (03CR) 10CI reject: [V: 04-1] Send phonetic name fields through on queue messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883673 (https://phabricator.wikimedia.org/T322793) (owner: 10Ejegg) [23:09:05] (03PS1) 10Eileen: [WIP] exception for civiimport [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883674 [23:13:12] (03PS2) 10Eileen: [WIP] exception for civiimport [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/883674 [23:13:17] (03PS2) 10Ejegg: Send phonetic name fields through on queue messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883673 (https://phabricator.wikimedia.org/T322793) [23:14:47] (03CR) 10CI reject: [V: 04-1] Send phonetic name fields through on queue messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883673 (https://phabricator.wikimedia.org/T322793) (owner: 10Ejegg) [23:16:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [23:17:41] (03PS3) 10Ejegg: Send phonetic name fields through on queue messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883673 (https://phabricator.wikimedia.org/T322793) [23:19:11] (03CR) 10CI reject: [V: 04-1] Send phonetic name fields through on queue messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883673 (https://phabricator.wikimedia.org/T322793) (owner: 10Ejegg) [23:21:05] (03PS4) 10Ejegg: Send phonetic name fields through on queue messages [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/883673 (https://phabricator.wikimedia.org/T322793) [23:21:16] PROBLEM - check_mysql on frdb1006 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [23:26:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [23:31:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [23:36:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: Cant connect to local MySQL server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql