[03:18:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [03:28:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [05:50:28] FIRING: OOM: OOM killer active on fransw1001.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [05:55:28] RESOLVED: OOM: OOM killer active on fransw1001.frack.eqiad.wmnet:9100 - https://alerts.wikimedia.org/?q=alertname%3DOOM [06:36:22] (03PS3) 10Dmiranda: Cover the hidden switch to annual link in cancel confirmation [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1313269 (https://phabricator.wikimedia.org/T429516) [06:49:23] 06Fundraising-Backlog, 10FR-Donor-portal: Investigate updating jsdom to v30 - https://phabricator.wikimedia.org/T433465 (10dmiranda) 03NEW [06:50:25] (03CR) 10Dmiranda: Updated router.js test coverage (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1310693 (https://phabricator.wikimedia.org/T429516) (owner: 10Dmiranda) [08:59:28] FIRING: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:15:19] 10Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 07fr-combowiki: Switch text to current translateable strings or create new ones if needed - https://phabricator.wikimedia.org/T433139#12166734 (10Damilare) a:03Damilare [12:59:28] FIRING: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:24:24] 06Fundraising-Backlog, 07FR-Audience-Dev: [low priority] Investigate conditional Google's script addition for Google Ads Grant (fundraising) - https://phabricator.wikimedia.org/T374079#12167450 (10MSuijkerbuijk_WMF) Hi @Damilare Coming back to this task to re-prioritise it as we plan to shortly restart online... [14:36:05] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07fr-combowiki: Extract GatewayChooser logic from special page - https://phabricator.wikimedia.org/T427326#12167512 (10Ejegg) 05Open→03Resolved a:03Ejegg Work done against {T430821} [15:19:59] (03Abandoned) 10Ejegg: WIP replace unmaintained NameParser with fork [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1306973 (https://phabricator.wikimedia.org/T425578) (owner: 10Ejegg) [15:39:35] (03PS6) 10Vaughn Walters: Add seedTestData.php maintenance script for selenium seeding [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1312579 (https://phabricator.wikimedia.org/T429531) [15:45:06] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donor-portal, and 4 others: Paused donations show as active in donor portal after clicking Manage - https://phabricator.wikimedia.org/T430700#12167887 (10Priyankar22) [16:44:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-payments-general, 10FR-Smashpig: Failing Trustly recurring charges with No Institution ID configured - https://phabricator.wikimedia.org/T430789#12168337 (10MBeat33) Marc at Trustly confirmed Monday, "the change has been finalized. No users will... [16:59:28] FIRING: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:23:27] damilare: OK, i've got the config patch and the ml patch locally and I've restarted the container [17:23:31] but I just get ImportError: cannot import name 'app' from 'fraud_service' (/srv/fundraising-ml/src/fraud_service/__init__.py) [17:24:11] interesting, do you still have the pth file in your venv? [17:25:21] let's see [17:25:58] yep, it's there [17:26:05] /srv/fundraising-ml/src/ [17:26:14] (contents of file) [17:26:41] ok let me pull that down again [17:28:16] oh wait that should be from fraud_service.fraud_service import app, shouldn't it? [17:28:19] let me try [17:28:38] I just did with the content of the MR and patch and it works for me [17:28:41] yes, that works! [17:28:45] oh weird [17:28:48] I had to do setup sh [17:29:06] hmmm [17:29:30] shouldn't it be fraud_service.fraud_service though? [17:29:48] if the PYTHONPATH root is /srv/fundraising-ml/src [17:30:04] then you have to specify from dir.file import var [17:31:14] I remember changing that because it was failing on prod and locally for me [17:31:23] I'd try that [17:31:39] oh i see the apache config tries to add an additional pythonpath root /srv/fundraising-ml/srv/fraud_service [17:31:45] that seems confusing [17:31:54] and seems not to be working for me [17:32:11] let's see if we can make it work with just the one pythonpath root on prod [17:33:19] ok, the prod venv does have the /srv/ml/src pointer as a pth file [17:34:24] (03PS4) 10Zaidusyy: Remove processor_form and IngenicoFormVariant [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1306753 (https://phabricator.wikimedia.org/T430678) [17:37:32] ehh [17:41:08] ok looks like I don't have the path file anymore, ejegg where was that placed again? [17:42:49] if so then that'd explain why the import is just fraud_service, because the path set in apache is in effect [17:45:24] the pth file is in /srv/ml-venv/lib/python3.11/site-packages/ [17:45:43] yh I just confirmed, I don't have it [17:46:59] the wsgi process is using the python specified in apache then [17:51:15] ok, let's make sure we're adding that in the image [17:54:17] it is in the latest image build MR [17:54:28] but that's in draft status now so hasn't been published [17:54:53] do you have this in dev-images? https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/109 [17:56:12] i just wish I could figure out why my local apache / wsgi doesn't seem to apply that additional python-path [18:20:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donor-portal: Donor Portal - Investigate Why Some Donors Didn't Receive Link/Explore Monitoring Options - https://phabricator.wikimedia.org/T433521#12168788 (10Ejegg) [18:51:35] 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#12168893 (10Dwisehaupt) [18:52:40] (03CR) 10Laurabarluzzi: [C:03+2] Change card color and remove pause button when donation is paused [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1318722 (https://phabricator.wikimedia.org/T430700) (owner: 10Damilare Adedoyin) [18:52:51] (03CR) 10CI reject: [V:04-1] Change card color and remove pause button when donation is paused [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1318722 (https://phabricator.wikimedia.org/T430700) (owner: 10Damilare Adedoyin) [19:50:31] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07fr-combowiki: For ComboWiki, add a helper class to integrate data from get, post, and session - https://phabricator.wikimedia.org/T433537 (10Ejegg) 03NEW [19:50:48] 10Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 07fr-combowiki: Populate fields from URL/Session values in ComboWiki - https://phabricator.wikimedia.org/T432271#12169101 (10Ejegg) [19:50:49] 06Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 07fr-combowiki: For ComboWiki, add a helper class to integrate data from get, post, and session - https://phabricator.wikimedia.org/T433537#12169100 (10Ejegg) [20:01:07] I had to take a break to get dinner and a light nap [20:01:14] ejegg I rebuilt the image on your gitlab mr and I can see the .pth file, but it still works with the same config. [20:01:22] 10Fundraising Sprint: Notorious B.E.D., 06Fundraising-Backlog, 07fr-combowiki: Send intial contribution tracking row from ComboWiki - https://phabricator.wikimedia.org/T433539 (10Ejegg) 03NEW [20:16:18] (03PS1) 10Ejegg: Delete privatebin config [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1319164 [20:23:16] damilare: i am so confused [20:23:35] i see it working in prod [20:24:05] (03PS5) 10Damilare Adedoyin: Change card color and remove pause button when donation is paused [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1318722 (https://phabricator.wikimedia.org/T430700) [20:29:11] ahhh it works locally now [20:29:21] well, I'm not going to touch anything more :) [20:29:45] did you have to change anything to get it to work ejegg ? [20:29:48] (03PS4) 10Damilare Adedoyin: Set ML Apache to match production [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1307467 [20:29:50] (03CR) 10Ejegg: [C:03+2] Set ML Apache to match production [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1307467 (owner: 10Damilare Adedoyin) [20:29:52] (03CR) 10Ejegg: [V:03+2 C:03+2] Set ML Apache to match production [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1307467 (owner: 10Damilare Adedoyin) [20:30:03] thanks! [20:30:55] thank you! [20:31:05] damilare: I just did another rebuild [20:31:18] not sure why the previous one didn't make it work [20:31:41] oh I just noticed the MR was a commit behind main [20:31:47] but I guess it rebases on merge [20:31:56] and it was a pretty trivial one that it was behind [20:32:19] ok, I'm going to move my MR on the image out of draft then [20:33:15] oh i need to rebase [20:33:27] ejegg: eileen: frdev1002 trixie update in vb worked out fine. only hitch now is related to the ml-venv but that shouldn't be an issue in practice. [20:34:05] dwisehaupt: oh, is the ml project on fr-dev as well as frproto ? [20:37:04] yeah. i think it was there as testing. [20:38:19] it doesn't get in the way of puppet runs since the error only triggers on a rebuild of the venv in trixie. [20:38:32] oh interesting [20:39:08] our local version is working under trixie, but using an annoying pyenv thing to get an older python version [20:39:11] https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/109 [20:39:20] damilare: also if you want to approve ^^^ [20:40:10] ah. yeah, this is around python 3.13 and numpy calling pkgutil.ImpImporter which has issues. [20:40:26] related to numpy-1.23.5 [20:41:01] oh yeah, that sounds familiar [20:41:42] i don't think it's anything to get in the way of upgrading if we need since we have a separate box/role to handle that testing now. [20:42:23] ooh, kserve 0.18 is released! I think that actually runs on python 3.13 [20:42:49] damilare: ^^^ now that we finally have it running, shall we break it all over again with a kserve / numpy upgrade ? [20:59:28] FIRING: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:25:40] 😂 perhaps we could after we complete the connection between DI and the service [23:54:28] RESOLVED: DiskSpace: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace