[06:47:28] hello hello [06:47:43] so I tried to create the wikilabels instance [06:47:57] there is a puppet role (good), but it references https://github.com/wikimedia/wikilabels-wikimedia-config that doesn't exists anymore [06:48:00] sigh [06:48:41] there is https://github.com/wikimedia/wikilabels-wmflabs-deploy but not sure if it is the same [06:49:15] starting the other instance so we can check [07:05:14] ah ok so there is a fabric file in the deploy repo [07:23:09] mmm but it seems in need of that config deploy repo that doesn't exists anymore [07:23:23] kevinbazira: o/ [07:23:34] so I brought up the old VM, https://labels.wmflabs.org/ is working now if you want to do some tests [07:23:47] we'll be able to keep it up for a bit I am afraid, I am working on the new VM [07:25:17] ok I was able to connect to the 02 instance, the wikilabels-wmflabs-deploy repo is the new wikilabels-wikimedia-config [07:25:20] will update puppet [07:28:39] basically https://gerrit.wikimedia.org/r/c/operations/puppet/+/832151 [07:30:37] waiting for the change to propagate [07:31:01] then it seems sufficient to run fab initialize etc.. from my laptop, to bootstrap the new vm [07:48:23] so as anticipated there is some mess with python deps [07:48:43] the list of dependencies is not big, so we can probably manage [07:49:20] kevinbazira: o/ [07:57:20] elukey o/ [07:57:54] thank you for bringing up the old vm. let me run the training pipeline now ... [07:58:02] will keep you posted [08:00:13] ack! I managed to make the buster VM to run [08:00:23] so in a while I'll try to switch the endpoint [08:00:27] I'll warn you beforehand [08:12:21] created https://github.com/wikimedia/wikilabels/pull/266 [08:19:28] great. the training pipeline is running ... [09:43:11] kevinbazira: I have the new host ready, ok if I switch to the labels endpoint to it? [09:43:28] if possible we'll need to test the new endpoint with the training pipeline, just to know that we are good [09:44:03] yes please. the training is done. now going to test the model to see whether performance has improved. [09:46:38] ok done, but we need to wait ~5 mins for the DNS change to be effective [10:06:06] kevinbazira: when you have time to re-test the wikilabels endpoint please go ahead :) [10:06:13] the webpage now points to the new instance [10:06:36] klausman: o/ I copied a tar with the content of the old wikilabels stretch vm's home dirs to my home dir on the new instance [10:06:48] (checked the sha etc.. so the backup looks good) [10:06:56] afaics nothing really worth it, but saved nonetheless [10:07:07] lemme know if it is ok for you to delete permanently the old instance [10:07:41] Did you also tar up /etc and potentially /var? Or is it likely nothing of value, either? [10:07:52] elukey earlier on this endpoint was working but now it's returning "Internal Server Error" https://labels.wmflabs.org/campaigns/fawiki/70/ [10:07:59] nah I think it doesn't really matter (/etc /var I mena) [10:08:17] Ok, then I'd delete the old instance once the URL kevin mentioned works [10:08:20] kevinbazira: ah the main page works, not the fawiki one sigh [10:09:17] psycopg2.OperationalError: fe_sendauth: no password supplied [10:09:18] mmmm [10:10:19] Maybe an env var? [10:10:38] Mh, but that would be part of puppet. [10:13:13] it is a part of the config probably not puppetized, like the oauth tokens [10:13:16] I doubt psycopg changed how to supply the password. [10:13:26] since puppet private for cloud vps is "public" [10:13:59] Ah, right, I was wondering if it might be in the real private repo somewhere [10:14:36] The old host still lives, so maybe the password can be found there. [10:15:58] now uwsgi refuses to work with the oauth config that I thought was working, so I'll work a bit more on it [10:19:40] ok it is back now [10:20:46] ahh there is a :98-database.yaml file [10:21:37] kevinbazira: now it works :) [10:21:42] (the fawiki link I mean) [10:21:47] checking ... [10:21:51] neat! [10:22:05] We should probably keep a copy of the non-puppet config "somewhere" [10:22:14] Or t least the credentials used [10:22:23] yes, elukey. it's now working. thanks! [10:23:15] super [10:23:21] klausman: definitely yes [10:24:09] going to have lunch, ttl! [11:59:36] also lunch [13:28:22] 10Machine-Learning-Team, 10ORES, 10MediaWiki-Core-Preferences, 10Moderator-Tools-Team (Kanban): 'Highlight likely problem edits' preference doesn't work in mobile web - https://phabricator.wikimedia.org/T314026 (10eigyan) Note: When installing the ORES extension make sure your ORES LocalSettings configurat... [13:53:43] elukey: hello! thank you for the wikilabels work [13:53:53] i'm wondering why T312564 and T307389 are separate tasks [14:00:23] taavi: o/ sorry for the lag, we'll get to the databases too I promise :) [14:17:57] aiko: https://github.com/faust-streaming/faust looks interesting too [14:21:12] 10Machine-Learning-Team, 10Data Engineering Planning, 10Research: Proposal: deprecate the mediawiki.revision-score stream in favour of more streams like mediawiki-revision-score- - https://phabricator.wikimedia.org/T317768 (10elukey) [14:21:18] created --^ [14:22:15] 10Machine-Learning-Team, 10Data Engineering Planning, 10Research: Proposal: deprecate the mediawiki.revision-score stream in favour of more streams like mediawiki-revision-score- - https://phabricator.wikimedia.org/T317768 (10elukey) [14:23:09] 10Machine-Learning-Team, 10Data Engineering Planning, 10Research: Proposal: deprecate the mediawiki.revision-score stream in favour of more streams like mediawiki-revision-score- - https://phabricator.wikimedia.org/T317768 (10elukey) [14:26:54] posted the link on the slack data-engineering channel, let's see what responses we get [15:49:58] going afk for a run folks, ttl! [16:42:59] results of load test for outlink model: 48 rps and 867ms p99 latency on codfw. 68 rps and 138ms p99 lantency on staging [16:56:56] 60 rps and 1.1s p99 latency on eqiad [17:03:42] but when I tested the model with feature-str (without calling mw api), the result became 115 rps/422ms on codfw, 145 rps/68ms on staging, and 233 rps/638ms on eqiad [17:07:51] eqiad is better than staging [17:19:01] 10Machine-Learning-Team, 10Data Engineering Planning, 10Research: Proposal: deprecate the mediawiki.revision-score stream in favour of more streams like mediawiki-revision-score- - https://phabricator.wikimedia.org/T317768 (10Milimetric) To me, this approach makes a lot of sense. Anyone that needs so... [18:25:13] 10Machine-Learning-Team, 10Data Engineering Planning, 10Research: Proposal: deprecate the mediawiki.revision-score stream in favour of more streams like mediawiki-revision-score- - https://phabricator.wikimedia.org/T317768 (10JAllemandou) +1 for me as well - I see only benefits in this move except for...