[00:26:22] 06Fundraising-Backlog, 07fr-combowiki: Employer field in combowiki - https://phabricator.wikimedia.org/T433142#12193822 (10Cstone) [00:26:52] 03Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 07fr-combowiki: Employer field in combowiki - https://phabricator.wikimedia.org/T433142#12193823 (10Cstone) [00:34:00] (03PS2) 10Lars SG: 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) [00:36:45] (03PS1) 10Lars SG: Fix for object no longer allowed as api param in 6.18 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322243 (https://phabricator.wikimedia.org/T434264) [00:40:04] 03Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 07fr-combowiki: Employer field in combowiki - https://phabricator.wikimedia.org/T433142#12193830 (10AnnWF) a:03AnnWF [00:53:57] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donor-portal, 06FR-donorrelations: Donor portal dedupe logic issue - https://phabricator.wikimedia.org/T433723#12193833 (10Lars) The only work to do here is to block donate@wikimedia.org and other similar addresses (hello@, etc). [[ https://wikim... [00:57:11] (03PS2) 10Lars SG: Fix for object no longer allowed as api param in 6.18 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322243 (https://phabricator.wikimedia.org/T434264) [01:03:58] (03CR) 10CI reject: [V:04-1] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [01:21:24] (03PS3) 10Lars SG: Fix for object no longer allowed as api param in 6.18 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322243 (https://phabricator.wikimedia.org/T434264) [01:31:43] (03PS1) 10Lars SG: Update wikimedia/omnimail-silverpop to latest [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322263 (https://phabricator.wikimedia.org/T434264) [01:34:32] (03PS3) 10Lars SG: 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) [01:34:52] (03Abandoned) 10Lars SG: Update wikimedia/omnimail-silverpop to latest [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322263 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [01:52:13] (03CR) 10Eileen: [C:03+2] Fix for object no longer allowed as api param in 6.18 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322243 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [01:52:48] (03CR) 10CI reject: [V:04-1] Fix for object no longer allowed as api param in 6.18 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322243 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [02:01:05] (03CR) 10Eileen: [C:03+2] "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322243 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [02:05:54] (03CR) 10CI reject: [V:04-1] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [02:08:13] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donor-portal, 06FR-donorrelations: Donor portal dedupe logic issue - https://phabricator.wikimedia.org/T433723#12193858 (10MBeat33) 05Open→03Resolved a:03MBeat33 Thank you, Lars for confirming the links will have the same form. I bloc... [02:10:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [02:16:36] (03Merged) 10jenkins-bot: Fix for object no longer allowed as api param in 6.18 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322243 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [02:20:42] (03PS4) 10Lars SG: 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) [02:24:43] FIRING: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:25:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.5] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [02:27:10] (03CR) 10Eileen: [C:03+2] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [02:42:52] (03CR) 10CI reject: [V:04-1] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [03:19:42] (03CR) 10Ejegg: [C:03+2] "This does the job! I might clean up those references to 'Fredge' as that refers to a separate database, but that's just cosmetic." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321577 (https://phabricator.wikimedia.org/T430711) (owner: 10Damilare Adedoyin) [03:20:18] (03PS4) 10Damilare Adedoyin: Payments attempt queue consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321577 (https://phabricator.wikimedia.org/T430711) [03:20:28] (03CR) 10Ejegg: [C:03+2] Payments attempt queue consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321577 (https://phabricator.wikimedia.org/T430711) (owner: 10Damilare Adedoyin) [03:27:19] (03CR) 10Cstone: [C:03+2] "thanks!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320190 (owner: 10Damilare Adedoyin) [03:27:41] 03Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 07Fundraising-ComboWiki-Dependency, 03Civi RoadMap FY 26-27, and 2 others: Use GatewayRouter to select the payment gateway in ComboWiki - https://phabricator.wikimedia.org/T430821#12193917 (10Ejegg) Moving based on its position in last sprint.... [03:28:50] (03CR) 10Eileen: [C:03+2] "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [03:31:31] (03Merged) 10jenkins-bot: Add cleanup for Apple Pay on close [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320190 (owner: 10Damilare Adedoyin) [03:45:51] (03CR) 10CI reject: [V:04-1] Payments attempt queue consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321577 (https://phabricator.wikimedia.org/T430711) (owner: 10Damilare Adedoyin) [03:49:09] (03PS5) 10Ejegg: Payments attempt queue consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321577 (https://phabricator.wikimedia.org/T430711) (owner: 10Damilare Adedoyin) [03:49:12] (03CR) 10Ejegg: [C:03+2] Payments attempt queue consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321577 (https://phabricator.wikimedia.org/T430711) (owner: 10Damilare Adedoyin) [03:49:51] (03CR) 10Ejegg: [C:03+2] "PS5 just updates the SmashPig version in composer.json to get the new queue definitions." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321577 (https://phabricator.wikimedia.org/T430711) (owner: 10Damilare Adedoyin) [03:52:10] (03CR) 10CI reject: [V:04-1] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [04:20:25] huh, the omniobjects aren't getting the mock guzzle? [04:34:17] (03Merged) 10jenkins-bot: Payments attempt queue consume [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321577 (https://phabricator.wikimedia.org/T430711) (owner: 10Damilare Adedoyin) [04:36:56] ack, it's super late here [04:37:10] have a good weekend, all [04:50:29] (03PS5) 10Lars SG: 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) [04:50:46] (03CR) 10CI reject: [V:04-1] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [04:56:52] (03PS7) 10Lars SG: 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) [04:57:14] (03CR) 10CI reject: [V:04-1] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [05:10:43] (03PS8) 10Lars SG: 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) [05:17:43] (03PS1) 10Eileen: Trustly audit: use UUID consistently for reversal gateway_txn_id [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1322506 (https://phabricator.wikimedia.org/T433919) [05:27:17] 06Fundraising-Backlog, 10FR-WMF-Audit: Adyen - new unknown payment method 'accel' - https://phabricator.wikimedia.org/T434269 (10Eileenmcnaughton) 03NEW [05:43:14] (03CR) 10CI reject: [V:04-1] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [05:43:26] (03PS2) 10Eileen: Trustly audit: use UUID consistently for reversal gateway_txn_id [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1322506 (https://phabricator.wikimedia.org/T433919) [05:44:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [05:44:40] (03CR) 10Lars SG: [C:03+2] Trustly audit: use UUID consistently for reversal gateway_txn_id [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1322506 (https://phabricator.wikimedia.org/T433919) (owner: 10Eileen) [05:45:40] (03Merged) 10jenkins-bot: Trustly audit: use UUID consistently for reversal gateway_txn_id [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1322506 (https://phabricator.wikimedia.org/T433919) (owner: 10Eileen) [05:45:58] (03CR) 10Lars SG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [05:49:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [06:08:28] FIRING: OOM: OOM killer active on fransw1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [06:13:28] RESOLVED: OOM: OOM killer active on fransw1002.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [06:24:43] FIRING: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:29:43] RESOLVED: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:09:29] (03PS2) 10Dmiranda: Added clarifying language to email preference center form. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322063 (https://phabricator.wikimedia.org/T433679) [07:11:28] FIRING: OOM: OOM killer active on fransw1001.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [07:16:28] RESOLVED: OOM: OOM killer active on fransw1001.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [07:16:34] 03Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences, and 2 others: Additions to Email Preference Centre - https://phabricator.wikimedia.org/T433679#12194050 (10dmiranda) 05In progress→03Resolved Thanks for the review @Lars, the temp... [07:16:35] (03CR) 10Dmiranda: Added clarifying language to email preference center form. (032 comments) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322063 (https://phabricator.wikimedia.org/T433679) (owner: 10Dmiranda) [07:53:29] (03CR) 10Umherirrender: [C:03+2] build: Updating dependencies [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1322105 (owner: 10Libraryupgrader) [08:16:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [09:16:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [10:43:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.4] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [11:33:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.7] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [12:34:47] (03PS5) 10Damilare Adedoyin: Add i18n locales to combowiki skeletons [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1321614 (https://phabricator.wikimedia.org/T433139) [15:01:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Fix omnimail so CI tests pass - https://phabricator.wikimedia.org/T434308 (10Lars) 03NEW [15:03:22] (03CR) 10Lars SG: [C:03+2] Added clarifying language to email preference center form. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322063 (https://phabricator.wikimedia.org/T433679) (owner: 10Dmiranda) [15:04:09] 03Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences, and 2 others: Additions to Email Preference Centre - https://phabricator.wikimedia.org/T433679#12195567 (10Lars) 05Resolved→03Open Looks good, thanks @dmiranda. [15:05:05] 03Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences, and 3 others: Additions to Email Preference Center - https://phabricator.wikimedia.org/T433679#12195570 (10Aklapper) [15:11:09] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12195583 (10hashar) i think @mhurd and @peter noticed the MediaW... [15:15:39] (03Merged) 10jenkins-bot: Added clarifying language to email preference center form. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322063 (https://phabricator.wikimedia.org/T433679) (owner: 10Dmiranda) [15:20:10] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Fix omnimail so CI tests pass - https://phabricator.wikimedia.org/T434308#12195604 (10Lars) Alternatively we could just fork the repo and drop mailgun support entirely, since that's what is causing the composer issue. H... [15:22:37] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12195624 (10hashar) And the changes we made to the `docker-regis... [15:30:41] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12195653 (10hashar) And from a Quibble / MediaWiki job, the dail... [15:36:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12195674 (10Lars) Thanks @hashar. I did some work to speed up ou... [15:42:42] larssandergreen: good morning! Greg reached out about a task pointing CiviCRM builds are slower than they used to ( https://phabricator.wikimedia.org/T434024 ) [15:42:57] I'll see what I can debug tonight :] I doubt it is caused by civicrm [15:43:19] one sure thing, if that is WAY slower AND cause tests to fail and therefore developers to be affected, it is not a good thing :] [15:57:21] thanks hashar, I replied on the phab, but agree with all that you've laid out there, makes sense with what we are seeing [15:57:45] yeah so I don't think it is an issue with CiviCRM [15:57:51] or the image used to run the suite [15:58:08] what it is, I have absolutely no clue. But that sounds like the kind of jigsaw puzzle I like to solve [15:59:47] 06Fundraising-Backlog, 10FR-Donation-Processing: Chariot data issue - DAFpay payment method - https://phabricator.wikimedia.org/T434317 (10MDemosWMF) 03NEW [16:02:06] hashar: that's great to hear! For me, the fact that restoring the DB snapshot in our build process sometimes takes seconds and sometimes takes minutes feels like a clue. There must be some kind of underlying infra difference there. [16:10:09] (03PS1) 10Lars SG: 6.18: remove files deleted upstream [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322796 (https://phabricator.wikimedia.org/T434264) [16:20:17] (03CR) 10Wfan: "Translation looks good, just remove the .text() since vue renders strings directly." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1321614 (https://phabricator.wikimedia.org/T433139) (owner: 10Damilare Adedoyin) [16:39:13] (03PS2) 10Lars SG: 6.18: remove files deleted upstream [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322796 (https://phabricator.wikimedia.org/T434264) [16:55:56] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Audience-Dev, 07fr-current-sprint: Add recurring donors whose recurring donations don't yet have a donation attached to recurring segments & statuses - https://phabricator.wikimedia.org/T410432#12196062 (10Lars) 05Open→03Resolved a:03Lar... [16:58:59] larssandergreen: possibly that is due to IO being starved/throttled somehow [16:59:11] I will find it eventuually, but surely not tonight or over the week-end [17:00:11] yes, it does seem like an IO issue. Great that you're on it, definitely doesn't need to be solved immediately. [17:00:57] * hashar nods [17:01:17] whenever I make progress, I'll update the phab task [17:07:18] 03Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 07fr-combowiki, 13Patch-For-Review: Explain text on combowiki (text and links at the bottom of the payments form) - https://phabricator.wikimedia.org/T433144#12196085 (10AnnWF) [17:29:00] (03CR) 10Wfan: [C:03+2] "Looks good, I will do a refactor later for the html cleanup, but let's move this first" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1321614 (https://phabricator.wikimedia.org/T433139) (owner: 10Damilare Adedoyin) [17:31:47] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12196149 (10Andrew) a minor thing that I did on the 24th: https... [17:31:59] (03Merged) 10jenkins-bot: Add i18n locales to combowiki skeletons [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1321614 (https://phabricator.wikimedia.org/T433139) (owner: 10Damilare Adedoyin) [17:34:27] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 07fr-acoustic, 07FR-Audience-Dev: Populating both_funds_latest_donation_source field - https://phabricator.wikimedia.org/T379700#12196155 (10Lars) This backfill to Acoustic has been done, however there are still [[ https:... [17:37:05] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12196170 (10hashar) I went to look at [[ https://grafana.wikimed... [17:45:41] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12196202 (10Lars) @Andrew Interesting, our first timeout was abo... [17:59:45] (03PS6) 10Wfan: Add ach when recurring in combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322120 (https://phabricator.wikimedia.org/T433655) [18:10:29] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Merging contacts can create multiple segment/status change rows in wmf_donor_history - https://phabricator.wikimedia.org/T431741#12196261 (10CHudson-WMF) Adding this as another strange outcome of merges in case it hasn't been seen yet. Contact #63273950 h... [18:13:09] (03PS7) 10Wfan: Add ach when recurring in combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322120 (https://phabricator.wikimedia.org/T433655) [18:26:58] (03PS8) 10Wfan: Add ach when recurring in combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322120 (https://phabricator.wikimedia.org/T433655) [18:36:21] (03PS9) 10Wfan: Add ach when recurring in combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322120 (https://phabricator.wikimedia.org/T433655) [19:12:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, and 2 others: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334 (10Andrew) 03NEW [19:15:34] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, and 2 others: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334#12196432 (10Andrew) p:05Triage→03High a:03Andrew This started happenin... [19:31:58] (03CR) 10Cstone: [C:03+2] "Thanks Wenjun! Yeah we can handle a better ... design later but that's great for now" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322120 (https://phabricator.wikimedia.org/T433655) (owner: 10Wfan) [19:39:49] (03Merged) 10jenkins-bot: Add ach when recurring in combo wiki [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1322120 (https://phabricator.wikimedia.org/T433655) (owner: 10Wfan) [20:01:58] (03PS7) 10Cstone: WIP: Google pay again ontop of latest refactor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1319893 (https://phabricator.wikimedia.org/T430749) [20:06:46] (03CR) 10CI reject: [V:04-1] WIP: Google pay again ontop of latest refactor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1319893 (https://phabricator.wikimedia.org/T430749) (owner: 10Cstone) [20:40:19] (03PS8) 10Cstone: Google pay again ontop of latest refactor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1319893 (https://phabricator.wikimedia.org/T430749) [20:44:26] (03CR) 10CI reject: [V:04-1] Google pay again ontop of latest refactor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1319893 (https://phabricator.wikimedia.org/T430749) (owner: 10Cstone) [20:45:40] (03PS9) 10Cstone: Google pay again ontop of latest refactor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1319893 (https://phabricator.wikimedia.org/T430749) [21:10:55] larssandergreen: if you are still around. It seems the slow build time was due to a disk being too slow and that has been resolved by andrewbogott \o/ So tentatively your civicrm build should be fast again now [21:11:13] if you confirm on https://phabricator.wikimedia.org/T434024 that'll be nice [21:11:39] (03CR) 10Lars SG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321676 (https://phabricator.wikimedia.org/T432091) (owner: 10Lars SG) [21:12:30] thanks hashar, that's great news. I saw those charts you posted, which looked pretty brutal. I'll kick off a few runs and report back on the phab shortly [21:12:58] (03CR) 10Lars SG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [21:13:05] (03CR) 10Lars SG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322796 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [21:15:14] (03CR) 10Wfan: "This looks good, same problem we have to wait pretty a while to see success page haha, let's fix later" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1319893 (https://phabricator.wikimedia.org/T430749) (owner: 10Cstone) [21:18:21] larssandergreen: awesome thanks! [21:19:17] larssandergreen: and feel free to close https://phabricator.wikimedia.org/T434024 if it is fixd [21:19:35] the follows up will be handled in the subtask https://phabricator.wikimedia.org/T434334 [21:40:31] (03CR) 10Wfan: [C:03+2] Google pay again ontop of latest refactor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1319893 (https://phabricator.wikimedia.org/T430749) (owner: 10Cstone) [21:45:16] (03Merged) 10jenkins-bot: Google pay again ontop of latest refactor [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1319893 (https://phabricator.wikimedia.org/T430749) (owner: 10Cstone) [22:08:35] (03PS3) 10Wfan: add some bottom text and links for combo wiki form as component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320219 (https://phabricator.wikimedia.org/T433144) [22:08:44] (03CR) 10CI reject: [V:04-1] add some bottom text and links for combo wiki form as component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320219 (https://phabricator.wikimedia.org/T433144) (owner: 10Wfan) [22:14:41] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12196806 (10Lars) Thanks @Andrew and @hashar. I //think// it's b... [22:18:12] (03CR) 10Eileen: [C:03+2] 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [22:18:31] (03CR) 10Eileen: [C:03+2] 6.18: remove files deleted upstream [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322796 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [22:18:51] (03PS4) 10Wfan: add some bottom text and links for combo wiki form as component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320219 (https://phabricator.wikimedia.org/T433144) [22:20:52] (03CR) 10CI reject: [V:04-1] add some bottom text and links for combo wiki form as component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320219 (https://phabricator.wikimedia.org/T433144) (owner: 10Wfan) [22:22:29] (03PS5) 10Wfan: add some bottom text and links for combo wiki form as component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320219 (https://phabricator.wikimedia.org/T433144) [22:37:07] (03Merged) 10jenkins-bot: 6.18 upgrade [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322202 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [22:37:11] (03PS2) 10Eileen: Trustly audit parse - handle all R codes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1320510 (https://phabricator.wikimedia.org/T433921) [22:38:24] (03Merged) 10jenkins-bot: 6.18: remove files deleted upstream [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322796 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [22:41:57] (03PS1) 10Lars SG: Bump smarty to 5.8.4. for today's security advisories [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322980 (https://phabricator.wikimedia.org/T434264) [22:43:29] (03PS6) 10Wfan: add some bottom text and links for combo wiki form as component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320219 (https://phabricator.wikimedia.org/T433144) [22:49:47] (03CR) 10Eileen: [C:03+2] Prevent retry of single statement when within db transaction [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321676 (https://phabricator.wikimedia.org/T432091) (owner: 10Lars SG) [22:50:01] (03CR) 10Eileen: [C:03+2] Bump smarty to 5.8.4. for today's security advisories [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322980 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [22:54:14] (03PS7) 10Wfan: Add some bottom text and links for combo wiki form as component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1320219 (https://phabricator.wikimedia.org/T433144) [23:15:51] (03Merged) 10jenkins-bot: Prevent retry of single statement when within db transaction [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1321676 (https://phabricator.wikimedia.org/T432091) (owner: 10Lars SG) [23:15:54] (03Merged) 10jenkins-bot: Bump smarty to 5.8.4. for today's security advisories [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1322980 (https://phabricator.wikimedia.org/T434264) (owner: 10Lars SG) [23:19:15] (03PS3) 10Eileen: Trustly audit parse - handle all R codes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1320510 (https://phabricator.wikimedia.org/T433921) [23:19:15] (03PS1) 10Eileen: Trustly audit: use the main trustly transaction_id for the refund_id [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1322987 (https://phabricator.wikimedia.org/T433919) [23:24:43] FIRING: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:27:22] civi disk space warning? ^ [23:39:59] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-CiviCRM-Searchkit, 06FR-donorrelations: SearchKit for Merged Contacts and Potential Duplicates by Donor Segment - https://phabricator.wikimedia.org/T418014#12196896 (10Lars) Hi @SHust, while the full ask here will have to wait until later, I saw... [23:53:21] FIRING: [4x] BirdServiceDown: bird service down on pay-lb1001.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DBirdServiceDown [23:53:47] FIRING: [2x] HAProxyBackendDown: HAProxy backend payments_backend on pay-lb1002.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendDown [23:58:21] RESOLVED: [4x] BirdServiceDown: bird service down on pay-lb1001.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DBirdServiceDown [23:58:47] RESOLVED: [4x] HAProxyBackendDown: HAProxy backend payments_backend on pay-lb1001.frack.eqiad.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendDown [23:59:14] yeah. checking on pay-lb. that's a bit concerning