[07:18:31] <elukey>	 hello folks
[07:40:58] <elukey>	 the more I think about Cassandra for the ml-cache nodes the more I think that it will be a good solution for us
[07:41:57] <elukey>	 the main questio mark is latency
[07:42:15] <elukey>	 even if I think that a properly designed keyspace (well partitioned etc..) in Cassandra is fast enough
[07:56:14] <elukey>	 maybe a little less performant than Redis
[07:56:26] <elukey>	 but way more flexible (maintenance, availability, etc..)
[07:56:41] <elukey>	 we should involve Data Engineering for sure in the design
[08:19:49] <wikibugs>	 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Send score to eventgate when requested - https://phabricator.wikimedia.org/T301878 (10elukey) >>! In T301878#7741542, @Ottomata wrote: > I see!  So you'd kinda be using ChangeProp as the Knative eventing stand in for now :)   Exactly yes, after the...
[10:29:51] <aiko>	 good morning :)
[11:34:07] <klausman>	 \o
[11:35:36] <klausman>	 elukey: giving the VM creation for staging etcd's another try in a moment
[11:39:16] <elukey>	 ack!
[11:39:23] <elukey>	 I was in a meeting sorry, going afk for lunch now :)
[11:39:26] <elukey>	 hello aiko!
[11:44:22] <klausman>	 Buon appetito :)
[12:38:24] <klausman>	 elukey: ping when you're back. This VM creation is taking foever :-/
[12:50:40] <klausman>	 unping.
[13:26:07] <wikibugs>	 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Send score to eventgate when requested - https://phabricator.wikimedia.org/T301878 (10Ottomata) >  Does it make sense? Ya that makes sense!  So the switchover would just be a changeprop deployment that would change the consumer's behavoir when it r...
[14:12:50] <elukey>	 klausman: o/
[14:12:59] <elukey>	 yeah it takes a long time :(
[14:13:02] <elukey>	 did it work at the end?
[14:14:22] <klausman>	 Still waiting
[14:14:35] <klausman>	 Moritz is doing other maintenance that slows everything in codfw_A down
[14:14:47] <klausman>	 And my attempts for the other VMs are failing
[14:15:27] <elukey>	 makes sense yes
[14:16:51] <elukey>	 what kind of errors are you getting?
[14:17:19] <klausman>	 Ganeti doesn't see the added DNS entries
[14:32:24] <chrisalbon>	 Morning all
[14:34:00] <elukey>	 morning!
[14:35:06] <klausman>	 \o
[14:47:17] <wikibugs>	 (03CR) 10Elukey: [V: 03+2 C: 03+2] Updates frozen-requirements after wheel cleanup. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/762901 (https://phabricator.wikimedia.org/T300195) (owner: 10Halfak)
[14:47:47] <elukey>	 ok I think that we can try to deploy ORES
[14:47:55] <elukey>	 any thoughts?
[14:52:56] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Support (or not) the ORES augmented feature output in liftwing - https://phabricator.wikimedia.org/T301766 (10achou) Hi,   I did some research on how ORES does this, in [[ https://github.com/wikimedia/ores/blob/master/ores/scoring_context.py#L67 | ores/scori...
[14:56:50] <aiko>	 elukey: o/ let's do it!
[14:58:18] <elukey>	 aiko: sure!
[14:58:32] <elukey>	 so from deploy1002, the deployment node, I tried
[14:58:33] <elukey>	 httpbb /srv/deployment/httpbb-tests/ores/test_ores.yaml --hosts=ores1001.eqiad.wmnet --http_port=8081
[14:58:47] <elukey>	 that works, but two requests fail, the hiwiki ones
[14:58:56] <elukey>	 that is ok since the model is not deployed yet
[14:59:13] <elukey>	 so we'll be able to check ores1001 after the canary deployment
[14:59:17] <elukey>	 and see if it is all good
[15:00:41] <elukey>	 the docs to deploy to prod are https://wikitech.wikimedia.org/wiki/ORES/Deployment#Deploy_to_production
[15:00:46] <elukey>	 aiko: --^
[15:01:08] <elukey>	 I can drive via screen sharing if you want
[15:01:18] <elukey>	 and let you do the checks
[15:01:26] <elukey>	 what do you think?
[15:01:29] <aiko>	 yess please!
[15:01:31] <aiko>	 :D
[15:03:45] <klausman>	 +1 for enthusiasm :)
[15:04:24] <elukey>	 we'll see later :D
[15:05:01] <klausman>	 You mean a sort of "you'll like it until you see it"?
[15:09:12] <elukey>	 usually an ORES deployment is an experience
[15:09:26] <elukey>	 enthusiasm tend to decrease after it :D
[15:09:33] <wikibugs>	 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Re-evaluate ip pools for ml-serve-{eqiad,codfw} - https://phabricator.wikimedia.org/T302701 (10akosiaris) >>! In T302701#7740625, @elukey wrote: > On ml-serve-eqiad (half way through loading ORES pods): >  > ` > root@deploy1002:~# kubectl get svc -...
[15:13:42] <klausman>	 elukey: I have a review for the etcd host setup. Should I let you review it or seek someone else while you're elbows-deep in ORES guts
[15:13:53] <klausman>	 Wow, that sounds way grosser than it did in my head.
[15:21:20] <elukey>	 klausman: I can check it in ~30 mins (hopefully :D)
[15:25:10] <wikibugs>	 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) last commit on ores1001:  ` elukey@o...
[15:35:49] <elukey>	 ORES deployment to the canary started
[15:40:43] <klausman>	 TIA
[15:58:47] <elukey>	 so we are having an issue with the new model for hiwiki damaging
[15:58:52] <elukey>	 the rest looks working
[16:02:59] <elukey>	 nope all working, scratch that
[16:11:39] <elukey>	 deployment completed
[16:12:37] <klausman>	 \o/
[16:13:41] <elukey>	 https://ores.wikimedia.org/v3/scores/hiwiki/777/goodfaith
[16:13:45] <elukey>	 https://ores.wikimedia.org/v3/scores/hiwiki/777/damaging
[16:13:47] <elukey>	 all good
[16:13:54] <elukey>	 nothing horrible in the logs
[16:19:11] <elukey>	 we are watching https://logstash.wikimedia.org/app/dashboards#/view/ba190230-deb8-11e8-99b8-7fba019e77c2?_g=h@6c3eb7f&_a=h@f2110be
[16:19:20] <elukey>	 and some weird errors popped up
[16:19:33] <elukey>	 from the overall metrics it seems that the cluster is adjusting after the deploy
[16:19:38] <elukey>	 so we'll git it some minutes
[16:19:47] <elukey>	 I think that we'll need to tune the scap config to be more gentle
[16:20:18] <elukey>	 for example we have
[16:20:19] <elukey>	 Feature extraction error for model 1074689111 and revision goodfaith due to: Timed out after 15 seconds.
[16:20:24] <elukey>	 that is cryptic
[16:21:31] <elukey>	 ahhhhhhh
[16:21:42] <elukey>	 aiko: I am double stupid
[16:21:57] <elukey>	 we are seeing the logs because I added them :D
[16:22:37] <elukey>	 these are all the "scores errored" that are due to serious problems
[16:23:00] <elukey>	 (yes we do have scores errored that are apparently "expected", like the ones caused by non existens revisions etc..)
[16:23:13] <elukey>	 so there is no regression
[16:23:17] <elukey>	 it is just more data to check :)
[16:23:57] <elukey>	 for example
[16:23:59] <elukey>	 Feature extraction error for model 1334902099 and revision itemquality due to: JSONDecodeError: Failed to process datasource.wikibase.revision.entity_doc: Expecting value: line 1 column 1 (char 0)
[16:24:09] <elukey>	 this is something that failed silently before, now we see it
[16:24:32] <elukey>	 (as Aiko pointed out there is a bug in logging :P)
[16:25:00] <aiko>	 elukey: that's good!
[16:25:20] <elukey>	 aiko: this was the logging patch https://github.com/wikimedia/ores/pull/355/files
[16:25:25] <elukey>	 if you want you can fix it :)
[16:25:43] <elukey>	 now I see the problem, we didn't catch in review time :(
[16:25:58] <elukey>	 so for the moment let's call the deployment a success
[16:27:33] <aiko>	 \o/
[16:30:11] <klausman>	 Well done!
[16:30:21] <accraze>	 o/
[16:30:27] <accraze>	 nice work on the ores deployment everyone!
[16:31:05] <elukey>	 \o/
[16:31:59] <elukey>	 accraze: o/
[16:32:05] <elukey>	 these faiures are interesting
[16:32:06] <elukey>	 Feature extraction error for model 1074690851 and revision articletopic due to: Timed out after 15 seconds.
[16:32:15] <elukey>	 15 seconds?
[16:32:45] <accraze>	 weird! i thought the timeout was longer?
[16:33:34] <elukey>	 but also, 15 seconds for feature extraction :O :O
[16:34:06] <accraze>	 lololol
[16:34:35] <accraze>	 yeh i mean that should be cached if it does actually does take 15 secs
[16:34:55] <accraze>	 my guess is that the extractor got stuck or something
[16:36:22] <wikibugs>	 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) Deployment completed!  The new hiwik...
[16:39:16] <klausman>	 elukey: remind me, should the installer for VMs wait for input on partition setup?
[16:39:57] <klausman>	 'cause it does :-S
[16:39:59] <elukey>	 klausman: in theory no, d-i should get through without stopping
[16:40:10] <elukey>	 did you run puppet on install servers first? :D
[16:40:12] <klausman>	 # cat /proc/cmdline 
[16:40:14] <klausman>	 BOOT_IMAGE=debian-installer/amd64/linux initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 --- console=ttyS0,115200n8 raid0.default_layout=2 BOOTIF=01-aa-00-00-75-35-eb
[16:40:21] <klausman>	 No mention of the partman recipe
[16:40:53] <elukey>	 klausman: yeah I think that the install* servers are not updated with your netboot change
[16:40:57] <klausman>	 Was I maybe too quick when booting the VM after submitting the changes?
[16:41:12] <klausman>	 But I ran `sudo cumin 'install2*'  'run-puppet-agent'`
[16:41:27] <klausman>	 Is there more needed for it to pickup changes?
[16:41:58] <elukey>	 preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg
[16:42:14] <klausman>	 Good point
[16:42:34] <elukey>	 I usually run puppet on install* and apt*
[16:42:44] <elukey>	 let's do it and retry
[16:42:44] <klausman>	 I forgot apt
[16:42:55] <elukey>	 I have it bookmarked otherwise I forget too :D
[16:43:17] <klausman>	 is apt2001 enough?
[16:43:38] <elukey>	 I'd suggest to run puppet on all install and apt node just to be sure
[16:44:54] <klausman>	 Ok, try #2
[16:51:20] <klausman>	 there we go
[16:51:32] <klausman>	 alright, now start the setup of the other two VMs as well
[16:57:23] <elukey>	 super
[17:08:42] <klausman>	 elukey: https://gerrit.wikimedia.org/r/c/operations/puppet/+/767221 when you have some time
[17:27:36] <elukey>	 folks I just got notified that tomorrow I'll not have any electricity from 17:00 to midnight my time
[17:27:49] <elukey>	 so I may need to miss the team meeting :(
[17:28:24] <klausman>	 merci
[17:28:47] <klausman>	 Do some meditation, detach from doom scrolling etc.
[17:35:06] <elukey>	 yeah :D
[17:35:14] <elukey>	 going afk now, ttl!
[17:35:22] <elukey>	 have a good rest of the day folks
[17:35:23] <elukey>	 :)
[17:35:41] <klausman>	 \o
[17:49:19] <aiko>	 bye Luca :)
[20:42:28] <wikibugs>	 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Ciell) This is great, thank you so much eluk...