[05:43:45] <wikibugs>	 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks):  Upload editquality model binaries to storage - https://phabricator.wikimedia.org/T301413 (10kevinbazira) a:03kevinbazira
[05:48:34] <wikibugs>	 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Upload editquality model binaries to storage - https://phabricator.wikimedia.org/T301413 (10kevinbazira) 76/76 editquality models were uploaded successfully to Thanos Swift.
[05:51:04] <wikibugs>	 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Add editquality isvc configurations to ml-services helmfile - https://phabricator.wikimedia.org/T301415 (10kevinbazira) Inference services were created for all the 76 editquality mo...
[05:53:14] <wikibugs>	 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Epic, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10kevinbazira)
[05:55:14] <wikibugs>	 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Epic, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10kevinbazira) a:03kevinbazira The migration of editquality models has been completed.   76/76 editquality [[ https://p...
[06:34:14] <elukey>	 good morning folks
[06:34:25] <elukey>	 I am planning to start the ores2001 reimage in a few
[06:39:25] <wikibugs>	 (03CR) 10Elukey: [V: 03+2 C: 03+2] Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey)
[07:03:35] <elukey>	 of course one puppet/erb conditional doesn't work
[07:03:42] <elukey>	 uff
[07:03:47] <elukey>	 trying to fix it before the reimage
[07:26:13] <elukey>	 all right reimage of ores2001 started
[07:26:16] <elukey>	 fingers crossed
[08:11:26] <elukey>	 the first puppet run on an ores node takes ages
[08:11:54] <elukey>	 also a ton of packages and configs on HDDs is probably not super fast
[08:27:07] <elukey>	 ores2001 up and running!
[08:27:28] <elukey>	 before repooling uwsgi, I'll leave it running as it is to check if any celery error comes up
[08:27:45] <elukey>	 (celery should connect to redis to pick up tasks independently from the "local" uwsgi instance)
[08:28:00] * elukey bbiab
[08:50:20] <elukey>	 host repooled, nothing horrible in the uwsgi logs afaics
[08:50:23] <elukey>	 celery logs are clear
[08:51:09] <elukey>	 so far it seems that it worked :)
[08:53:44] <elukey>	 (logstash error logs are clear too)
[08:58:02] <klausman>	 \o
[08:58:23] <klausman>	 https://as13030.net/status/?ticket=4266277 <- not a great way to start the week :)
[08:58:34] <klausman>	 (AS13030/init7 is my ISP)
[08:59:10] <klausman>	 Also, nice work on the first ORES host, Luca. Lmk if I can help
[09:19:30] <elukey>	 klausman: o/
[09:19:52] <elukey>	 for ORES let's keep an eye together on the logs and logstash, for atm all seems good from what I can see
[09:20:06] <elukey>	 if you could double check as well I'd be very happy and reassured :D
[09:20:56] <elukey>	 it seems too good to be true
[09:21:01] <klausman>	 :D
[09:21:09] <klausman>	 Let me dig up my logstash bookmarks
[09:21:24] <elukey>	 but yeah the only error logs on the node that I can see are
[09:21:25] <elukey>	 json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
[09:21:38] <elukey>	 and this is the wikidata itemsquality issue that Aiko is working on
[09:21:47] <elukey>	 we need to deploy a new version of revscoring to fix it
[09:21:53] <klausman>	 That is indeed suspiciously well-working
[09:22:36] <elukey>	 we tested all models carefully in deployment-prep, so in theory we should be safe
[09:25:23] <klausman>	 AFAICT the only errors are the JSON one you mentioned and a few timeouts
[09:26:03] <klausman>	 e.g. `Feature extraction error for model 1085764976 and revision damaging due to: Timed out after 15 seconds.`
[09:26:21] <klausman>	 Which is probably benign/normal
[09:26:47] <elukey>	 these are the ones that I found as well, I was trying to see if they are reproduced in other nodes
[09:27:09] <elukey>	 the model/revision logging is twisted, we need to release a patch to fix it
[09:27:21] <elukey>	 but sadly it doesn't tell much about what wiki it was
[09:27:50] <klausman>	 Yeah, that should be fixed if possible
[09:28:14] <klausman>	 So color me cautiously optimistic about this migration :)
[09:28:32] <elukey>	 IIRC there was no way to add the wiki when I added the logging, at least the info was not available
[09:29:21] <elukey>	 ok yeah I see the feature extraction errors on some past logs on ores2* nodes
[09:29:41] <elukey>	 let's keep an eye on their numbers
[09:34:24] <elukey>	 klausman: I think that we could reimage ores2002 too to have a bigger set of nodes to check
[09:34:27] <elukey>	 what do you think?
[09:34:47] <elukey>	 I also wrote the procedure that I followed in https://phabricator.wikimedia.org/T303801#7895223
[09:36:59] <elukey>	 in case, change is ready :) https://gerrit.wikimedia.org/r/c/operations/puppet/+/788293/
[09:37:52] <klausman>	 Should I do the actual deed?
[09:38:23] <elukey>	 if you want yes!
[09:38:28] <klausman>	 +1'd that change
[09:38:56] <elukey>	 the scap repo should not give any issue during the first puppet run, but in case we'll need to scap deploy --limit
[09:39:05] <elukey>	 klausman: if you have time please go ahead with the procedure
[09:39:08] <klausman>	 ack.
[09:39:43] <elukey>	 to depool/repool the node, I used the shortcut `sudo -i depool/pool` on the node itself
[09:39:48] <klausman>	 Just one question: the reimage-to-buster would happen without updating the netboot cmdline? Or are those already done for all ROES nodes?
[09:39:50] <elukey>	 but it is the same if done via conftool etc..
[09:40:19] <elukey>	 klausman: so the partman recipe is the same and we are good, the buster-specific bits are set via cookbook
[09:40:28] <klausman>	 ah, I see
[09:40:55] <klausman>	 So you have an example cmdline for that?
[09:41:12] <klausman>	 2002 is depooled now
[09:41:29] <elukey>	 ah so one thing that I noticed only now since you mentioned netboot
[09:41:42] <elukey>	 on ores2001 I see one md0 raid1 array
[09:41:49] <elukey>	 on ores2002 two, md0 and md1
[09:42:16] <elukey>	 so I believe this is due to the fact that ores now uses the "standard" partman recipes created by SRE a while ago, but surely after the last ores node reimage
[09:42:28] <elukey>	 looks good from my side but lemme know your opinion as well
[09:42:30] <elukey>	 just to be sure
[09:42:57] <elukey>	 I used `sudo cookbook sre.hosts.reimage ores2001 --os buster` on cumin1001 (forgot to add the task though)
[09:43:03] <klausman>	 I think it should be fine. WIll keep an eye on the reinstall in any case
[09:43:36] <elukey>	 +1 perfect
[09:43:48] <elukey>	 remember to puppet merge etc before the reimage
[09:44:01] <elukey>	 so the first puppet run will pick up the change
[09:44:32] <klausman>	 should I merge the change gerrit-side or will you?
[09:45:00] <elukey>	 going to run pcc quick to verify the two changes in the diff
[09:45:03] <elukey>	 then you can proceed
[09:45:34] <elukey>	 (celery's systemd unit changes, and another config bit for ores as well, string -> int)
[09:45:47] <elukey>	 (otherwise celery refused to start sigh)
[09:46:05] <elukey>	 all good! https://puppet-compiler.wmflabs.org/pcc-worker1002/35024/ores2002.codfw.wmnet/index.html
[09:46:10] <elukey>	 klausman: green light to merge!
[09:46:36] <klausman>	 ack
[09:49:39] <klausman>	 About to run: sudo cookbook sre.hosts.reimage -t 303801 --os buster ores2002
[09:49:52] <elukey>	 +1
[09:50:44] <klausman>	 Running
[09:51:20] <elukey>	 the first puppet run took ages to me, I thought it was stuck but it was simply pulling a million packages in
[09:51:36] <klausman>	 Server is POSTing...
[09:52:20] * elukey brb
[09:52:38] <klausman>	 You always run away when things get Interesting :D
[09:53:20] <klausman>	 (I kid, of course)
[09:55:49] <klausman>	 and we're in the installer
[09:57:19] <klausman>	 https://phabricator.wikimedia.org/P27336 New layout lgtm
[10:06:23] <elukey>	 yep all good
[10:06:24] <klausman>	 Man, isntalls on spinning rust sure take longer than on SSDs :D
[10:06:33] <elukey>	 yeah :D
[10:07:42] <klausman>	 apt's insistance on using O_SYNC/fsync() doesn't help
[10:07:57] <elukey>	 what could possibly go wrong on HDDs 
[10:08:29] <klausman>	 I mean, during an install, it would be easier to not sync every write and just at the end run a sync, and maybe check all file checksums
[10:08:54] <elukey>	 yes yes I was joking about how slow it is now :)
[10:09:10] <klausman>	 If an install gets interrupted, the system is likely toast anyway
[10:09:44] <klausman>	 \o/ Installing grub
[10:10:10] <elukey>	 the first puppet run will take a lot more, brace yourself :D
[10:10:53] <elukey>	 in theory we can do a couple of nodes per day if nothing comes up, a gentle pace that will allow us to have all on Buster during the next couple of weeks
[10:15:41] <klausman>	 rebooted
[10:16:58] <klausman>	 So I have to do the puppet cert dance?
[10:17:36] <klausman>	 Doesn't look like it.
[10:17:53] <elukey>	 nope all taken care by the cookbook
[10:18:01] <klausman>	 puppet running now
[10:18:02] <elukey>	 it is only for vms that we need the special handling
[10:19:37] <klausman>	 and the cookbook thinks the install filed since it thinks the puppet run failed (it didn't)
[10:19:59] <elukey>	 weird, it should retry in theory
[10:20:50] <klausman>	 https://phabricator.wikimedia.org/P27339
[10:21:40] <elukey>	 this may be a bug, the "asking operator etc.." didn't work right?
[10:22:02] <klausman>	 I told it to skip the step (I can always do a manual run), but that was apparently still a failure
[10:22:11] <elukey>	 ahhh okok
[10:22:44] <klausman>	 Question is: do I have to remove downtimes etc manually?
[10:23:05] <elukey>	 in theory yes, but we can let them expire natually
[10:23:09] <elukey>	 *naturally
[10:23:09] <klausman>	 ack.
[10:24:23] <elukey>	 have you checked the puppet error?
[10:24:42] <elukey>	 ah no it seems running
[10:25:19] <elukey>	 klausman: did you kick off another run?
[10:25:29] <klausman>	 yep
[10:25:46] <klausman>	 Maybe I did so too early and that's what "broke" the cookbook
[10:25:52] <elukey>	 okok I am following from the mgmt console
[10:35:20] <elukey>	 created https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/788308, we'll see what Riccardo has to say about it :)
[10:43:22] <klausman>	 Puppet run was done, but with non-0 exit, so rerunning it
[10:44:36] <elukey>	 ah interesting, I think that we have to do the scap deploy --limit 
[10:44:56] <elukey>	 part of what it does it to kick off some post deployment scripts, that create the venv etc..
[10:45:16] <klausman>	 I see. Is a reboot also maybe useful?
[10:45:23] <elukey>	 yep definitely
[10:45:33] <klausman>	 Ok will do that in a moment
[10:45:36] <elukey>	 we can do both, scap deploy --limit ores2002.codfw.wmnet from deploy1002
[10:45:38] <elukey>	 then a reboot
[10:45:41] <elukey>	 klausman: --^
[10:45:44] <klausman>	 Ack
[10:46:05] <klausman>	 does scap have to run in a particular directory?
[10:46:21] <klausman>	 puppet completed with 0 exit status
[10:47:06] <elukey>	 yes, /srv/deployment/ores/deploy
[10:47:19] <klausman>	 as root or normal user?
[10:49:07] <klausman>	 trying as normal user seems to have worked
[10:49:40] <klausman>	 rebooting now
[10:50:40] <elukey>	 when you have a moment, Riccardo asked a question in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/788308/ about how it was run the cookbook
[10:51:58] <klausman>	 replied
[10:52:41] <elukey>	 ahh so you were running puppet before the cookbook
[10:52:47] <klausman>	 ores2002 is back
[10:54:34] <elukey>	 still same issue with celery
[10:54:35] <elukey>	 mmmm
[10:54:59] <klausman>	 How is that manifesting?
[10:55:14] <elukey>	 the celery-ores systemd unit fails
[10:55:26] <klausman>	 Maybe we need a post-reboot deploy?
[10:55:48] <elukey>	 nono it says that /srv/deployment/ores/deploy/venv/bin/celery is missing
[10:56:07] <elukey>	 so there is some issue with scap
[10:56:15] <elukey>	 I am going to try another deployment
[10:56:19] <klausman>	 I also can't SSH into deploy1002 anymor?!
[10:56:40] <elukey>	 probably moritz is rebooting it
[10:56:46] <klausman>	 ah
[10:57:33] <klausman>	 as for celery saying that /srv/deployment/ores/deploy/venv/bin/celery isn't there: that is actually the truth
[10:57:47] <klausman>	 the dir exists, but no celery binary
[10:58:27] <elukey>	 yep I think that the post deploy scripts that scap launches are not creating the venv correctly
[10:58:36] <elukey>	 so I tried again with scap deploy --limit blabla --force
[10:58:53] <elukey>	 since I think without --force it doesn't do anything if the target node is already up-to-date with the code
[10:58:58] <elukey>	 meanwhile we want to run the scripts
[10:59:05] <elukey>	 but it failed for a zip issue on wheels
[10:59:12] <elukey>	 retrying
[10:59:26] <elukey>	 it says
[10:59:26] <elukey>	 Processing ./submodules/wheels/billiard-3.6.4.0-py3-none-any.whl
[10:59:26] <elukey>	 Exception:
[10:59:36] <elukey>	     raise BadZipFile("File is not a zip file")
[10:59:36] <elukey>	 zipfile.BadZipFile: File is not a zip file
[10:59:52] <elukey>	 but it doesn't make any sense, it worked for ores2001
[11:00:07] <elukey>	 let's use the hammer, I am going to clean up the /srv/deployment/ores dir on ores2002 and redeploy
[11:00:28] <klausman>	 ack
[11:01:20] <elukey>	 so lovely to use git-lfs
[11:05:17] <klausman>	 Was that ironic?
[11:05:24] <elukey>	 yep :)
[11:05:54] <klausman>	 I have used it personally, and it does have some rough edges, but it hasn't bitten me (yet)
[11:15:21] <elukey>	 of course I made a mistake on deploy1002 sigh
[11:19:00] <klausman>	 What happened?
[11:22:06] <klausman>	 Oh.
[11:22:10] <klausman>	 Just saw it.
[11:22:12] <elukey>	 yep
[11:22:40] <klausman>	 Well, shit happens. You're not the first and certainly not the last to do something like that
[11:23:00] <klausman>	 Plus, it's recoverable.
[11:23:10] <elukey>	 sure sure, not happy about it, very silly mistake
[11:23:11] <klausman>	 Anything I can do to help with that?
[11:25:22] <elukey>	 nah thanks
[11:25:48] <klausman>	 then I'll go find some lunch, bbiab
[14:10:14] <chrisalbon>	 Morning all!
[14:10:54] <chrisalbon>	 Any issues with the reimage?
[14:11:05] <elukey>	 hi!
[14:11:15] <elukey>	 ores2001 perfect, 2002 problems with git-lfs
[14:11:26] <chrisalbon>	 Lol fuck
[14:11:34] <elukey>	 then I caused a problem on a deployment-server by mistake and lost hours :(
[14:13:19] <chrisalbon>	 Ooophf. You all got this!
[14:14:04] <elukey>	 we got some issue while reimaging 2002 though, going to reimage it 
[14:14:08] <elukey>	 (again)
[14:14:09] <elukey>	 let's see
[15:08:00] <elukey>	 seems the same problem
[15:09:13] <elukey>	 so on 2002, the submodules/wheels repo doesn't contain all zip files (the wheels) but some git-lfs ones
[15:09:18] <elukey>	 on 2001 all zip files
[15:11:38] <elukey>	 it is probably my fault, some wheels in https://gerrit.wikimedia.org/r/plugins/gitiles/research/ores/wheels/+/refs/heads/python37/billiard-3.6.4.0-py3-none-any.whl are git-lfs entities
[15:11:42] <elukey>	 others are binaries
[15:12:39] <elukey>	 pulled from https://git-lfs.github.com/spec/v1
[15:12:52] <elukey>	 mmm I never synced the gerrit wheels in github
[15:12:58] <elukey>	 I recall that there was a procedure
[15:20:08] <elukey>	 so github was mirrored via diffusion to gerrit, but this is for model repos
[15:20:14] <elukey>	 the ores wheels one didn't have this IIRC
[15:20:25] <elukey>	 kevinbazira: do you recall?
[15:21:56] <elukey>	 ok so https://github.com/wikimedia/research-ores-wheels/commit/a5de707d2ff292c8d7d31c51ba81531ce6e9911b
[15:22:12] <elukey>	 this is an example, on  the gerrit mirror  the last time Aaron pushed the wheels
[15:42:02] <elukey>	 ok no so https://github.com/wikimedia/editquality is the principal repo
[15:42:22] <elukey>	 https://github.com/wikimedia/research-ores-wheels is a gerrit mirror
[15:52:02] <elukey>	 something is weird
[15:52:11] <elukey>	 I can't really understand the ores wheels set up
[15:52:26] <elukey>	 and why on 2001 it worked
[15:54:21] <elukey>	 for example, on deploy1002
[15:54:22] <elukey>	 elukey@deploy1002:/srv/deployment/ores/deploy/submodules/wheels$ file billiard-3.6.4.0-py3-none-any.whl 
[15:54:25] <elukey>	 billiard-3.6.4.0-py3-none-any.whl: ASCII text
[16:17:26] <klausman>	 I remember git annex (not LFS) having a separate pull command that worked in a non-obvious way. Is this maybe happening on 2002?
[16:27:49] <elukey>	 no idea, it needs more work, for the moment it seems that even on the scap repo some files are lfs some files are not
[16:28:18] <elukey>	 we have also upgraded scap to avoid a settings for git-lfs, but of course nothing like this happened in deployment-prep
[16:28:34] <elukey>	 I am curious to see if we get the same reimaging ores2001
[16:29:24] <elukey>	 when I reimaged ores2001, the scap repo on deploy1002 didn't get updated (I forgot it), so I did it afterwards and ran scap deploy to the target node
[16:29:36] <elukey>	 meanwhile 2002 had the right code since the beginning to pull from
[16:29:41] <elukey>	 maybe there is some weird race condition
[16:34:23] <elukey>	 mmm so the last commit on master, that we are running on stretch nodes, has a mixture of lfs and binary wheels
[16:34:31] <elukey>	 but it was mirrored to github
[16:35:01] <elukey>	 maybe we just need to force the mirroring on github
[16:43:06] <elukey>	 on ores2001
[16:43:07] <elukey>	 ii  git-lfs        2.7.1-1+deb10u1 amd64        Git Large File Support
[16:43:40] <elukey>	 same on 2002
[16:43:54] <elukey>	 same scap version
[16:50:41] <chrisalbon>	 Listen, I don't want to interrupt your flow, but it is okay to leave this. Go for a run. Have a glass of wine. Read a good book. And come back to this tomorrow.
[16:51:20] <elukey>	 ah yes definitely
[16:51:30] <elukey>	 I was just braindumping if others have ides :)
[16:51:34] <elukey>	 *ideas
[16:52:20] <chrisalbon>	 You can come to the Tech Dept budget meeting I'm attending in 8 minutes and let your brain rest
[16:55:17] <elukey>	 thanks for the great opportunity but I'll keep going with the SRE meeting :D
[16:55:24] <chrisalbon>	 lol
[16:56:02] <chrisalbon>	 Also I probably wont be at the DSE K8s Sync meeting because it is 2:30am my time
[16:56:20] <elukey>	 lol
[16:56:22] <chrisalbon>	 So you'll be representing us Luca
[16:58:31] <chrisalbon>	 If we do it, klausman might be best situated to do the work but I think Olja wanted a small initial meeting just to see if the idea is even viable at a basic level
[16:58:52] <elukey>	 definitely
[16:59:46] <chrisalbon>	 You got volunteered to do it because you know k8s the best on this team, unfortunately for you.
[17:00:13] <elukey>	 super fine, I have been talking about this with Ben and Joseph in the past months
[17:00:35] <elukey>	 In theory it is just a matter of creating phab tasks
[17:00:40] <elukey>	 and split the work
[17:01:23] <chrisalbon>	 Cool, the only thing, which I am sure you are aware, is that we might need to buy new hardware and the deadline is closing fast. 
[17:01:35] <elukey>	 yes yes +1
[17:02:29] <chrisalbon>	 Its okay to say "We don't know exactly what we need but our quick guess is X hardware" and we'll put in the procurement order before the deadline. Not ideal but whatever.
[17:04:16] <klausman>	 elukey: one oddity I found:
[17:04:26] <klausman>	 oot@ores2002:/srv/deployment/ores/deploy/submodules/wheels# git lfs pull
[17:04:28] <klausman>	 Skipping object checkout, Git LFS is not installed.B/s                            
[17:04:37] <klausman>	 But it happens on both 2002 and 2001
[17:05:56] <klausman>	 Plus, 2001 has one modified file (numpy)
[17:11:53] <elukey>	 klausman: yeah weird
[17:12:12] <elukey>	 I think we'll need to ask to people with some context, I have zero idea atm
[17:12:19] <klausman>	 Same. 
[17:12:21] <elukey>	 going afk for today, have a nice rest of the day folks!
[17:12:31] <chrisalbon>	 Night elukey!
[17:12:42] <klausman>	 I will ruminate on it some tonight, maybe I'll have a bright idea by tomorrow (also, fresh eyes etc)
[17:12:52] <chrisalbon>	 Thanks both of you!
[17:17:53] <quantian>	 Random question: what IRC client do you all recommend?
[17:18:32] <klausman>	 Really depends on what you want out of it, and with what kind of environment (GUI, text temrinal, web ui) you are comfortable
[17:19:57] <klausman>	 Popular choices for text/terminal clients are weechat (which I use) and irssi.
[17:20:08] <quantian>	 I have no strong preference in regards to the environment. I just want something "good". I am new to this IRC world. Right now I am using the web client of Libera chat.
[17:20:12] <quantian>	 Thaks
[17:20:18] <klausman>	 For GUI, I've heard good things about X-Chat and Quassel on Linux, but my knowledge there is limited.
[17:21:00] <klausman>	 I have no idea about GUI clients for Windows. Well, in the 1990s, mIRC was popular, but 20+ years old "knowledge" is not very useful today :)
[17:22:14] <quantian>	 I think I will go with WeeChat. Thanks.
[18:36:40] <taavi>	 hey all, do you know what 'wikilabels' is? I have a pair of stretch VMs (clouddb-wikilabels-01,02.clouddb-services, officially maintained by the Cloud VPS admins but not in practice) that seem to have a postgres database related to that
[18:44:40] <wikibugs>	 10Machine-Learning-Team, 10Data-Services, 10Cloud-VPS (Debian Stretch Deprecation), 10cloud-services-team (Kanban): Upgrade wikilabels databases to buster/bullseye - https://phabricator.wikimedia.org/T307389 (10Majavah) p:05Triage→03Medium
[19:31:10] <wikibugs>	 10Machine-Learning-Team, 10Data-Services, 10Wikilabels, 10Cloud-VPS (Debian Stretch Deprecation), 10cloud-services-team (Kanban): Upgrade wikilabels databases to buster/bullseye - https://phabricator.wikimedia.org/T307389 (10Majavah)