[04:32:54] <marostegui>	 andrewbogott: What's up!
[05:00:40] <marostegui>	 Amir1: how long was s5's downtime for?
[05:00:54] <marostegui>	 codfw might alert? it has been delayed for 1d and 18h now XD
[07:23:40] <Amir1>	 marostegui: I downtimed it again today and yesterday (for a couple of days)
[07:23:46] <marostegui>	 ok!
[07:24:41] <Amir1>	 We have to combine the alters, otherwise it'll shut down replication to the cloud for a week
[07:24:55] <Amir1>	 It should finish soon though 
[07:25:53] <marostegui>	 Amir1: In case you have some maintenance going on on s7, please stop it or make sure it finishes before tomorrow: https://phabricator.wikimedia.org/T313383
[07:26:34] <Amir1>	 Yeah. I'll be there tomorrow. I think I have on s5 thingy
[08:14:15] <jynus>	 I'm still landing, so feel free to ping me for important stuff
[08:15:13] <Emperor>	 landing seems like a bad time to be pinged :)
[08:16:26] <jynus>	 nah, you can do vfr and attend an outage at the same time, I was told :-)
[08:16:59] <marostegui>	 hahahaha
[08:17:08] <marostegui>	 welcome back jynus 
[08:17:14] <volans>	 something's wrong here... did you two steal each other's nicks?
[08:18:15] <volans>	 line content and nick don't match :D
[08:19:44] <jynus>	 good news is that I belive some backups failed during tonight, but this was detected and corrected automatically
[10:14:58] <jynus>	 s4 data size is 1.8TB now, including innodb compression
[10:25:52] <Amir1>	 marostegui: the s5 codfw replication will start by the next two hours 
[10:26:14] <Amir1>	 jynus: is it bad? it should be fixed in a couple of weeks
[10:28:16] <jynus>	 it is not bad per se, just the typical recommended shard size is ~256GB for a time to recovery under 5 minutes
[10:30:38] <Amir1>	 453G  templatelinks.ibd
[10:30:54] <Amir1>	 It'll be back to 100GB at most
[10:43:48] <jynus>	 I think I caught up on email, I may have some questions for you later, but those can wait
[10:44:00] <jynus>	 you == a few people on this channel
[10:47:17] * Emperor hides
[10:47:20] <Emperor>	 :)
[10:56:20] <jynus>	 I was going to ask you if you could migrate ms hosts to ceph + bookwork for tomorrow, but I guess it can wait :-)
[11:16:35] <Emperor>	 lol
[11:23:15] <marostegui>	 I have failed over m3-master
[11:23:29] <marostegui>	 As part of the C5 switch crash preventive actions
[11:39:33] <volans>	 that line above would have been nicer with s/over m3-master// :-P
[13:21:21] <marostegui>	 jynus: I have stopped db2078 mysql today (the old host used for misc backups in codfw), we replaced it with db2160 past week, I plan to decommission it tomorrow, all looking good from your side since the change, right?
[13:21:36] <jynus>	 yeah
[13:21:41] <marostegui>	 good!
[13:21:44] <jynus>	 will we keep it around for some time
[13:21:47] <jynus>	 just in case?
[13:21:55] <jynus>	 as in, a week or so?
[13:22:18] <marostegui>	 So db2160 has been in place for a week already, looking at dbbackups.backups I can see it is being used fine, but if you prefer me to wait to decom db2078, I can do that too
[13:22:22] <jynus>	 ah, you just said tomorrow
[13:22:43] <jynus>	 if it is still up, can I do a one time copy just in case
[13:22:45] <jynus>	 ?
[13:22:51] <marostegui>	 It is yeah
[13:23:09] <marostegui>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/811884 this is when we replaced it, 7th July
[13:23:38] <jynus>	 I know I am being super careful, just wouldn't take me much time to run a snapshot
[13:23:44] <marostegui>	 Sure, go for it
[13:23:48] <jynus>	 and you can decom tomorrow as planned
[13:23:59] <marostegui>	 Sure, if you prefer me to wait till Friday, I can also do that, no problem
[13:24:01] <jynus>	 what about eqiad
[13:24:11] <jynus>	 do you have an eta for that?
[13:24:16] <jynus>	 for decom?
[13:24:17] <marostegui>	 No, that's not being touched
[13:24:40] <jynus>	 not yet, or not ever?
[13:24:50] <marostegui>	 I don't even know when db1117 expires :)
[13:24:54] <jynus>	 ah
[13:24:55] <jynus>	 ok
[13:24:55] <marostegui>	 I guess sometime this FY maybe
[13:24:57] <marostegui>	 But no idea :)
[13:24:58] <jynus>	 so different date
[13:25:00] <jynus>	 gotcha
[13:25:16] <jynus>	 ok, so let me leave a one time snapshot running
[13:25:31] <marostegui>	 Sure, no problem
[13:25:37] <jynus>	 I left the backup snapshot running, so you will be able to see the progress
[13:25:47] <jynus>	 *backup dashboard
[13:26:48] <RhinosF1>	 marostegui: 4 years since 1117 was racked
[13:26:54] <RhinosF1>	 https://phabricator.wikimedia.org/T191792
[13:26:59] <marostegui>	 So yeah, sometime this FY I guess
[13:34:26] <Amir1>	 marostegui: the s5 replication is flowing but 18h behind, can I run it on dbstore now?
[13:34:39] <marostegui>	 sure!
[13:34:47] <Amir1>	 awesome
[13:34:55] <marostegui>	 I am just waiting for the new wiki to replicate to sanitarium codfw hosts
[13:35:00] <marostegui>	 So I can sanitize it there
[13:35:09] <marostegui>	 ERROR 1049 (42000): Unknown database 'blwiki'
[13:35:12] <marostegui>	 Still not there heeh
[13:35:32] <Amir1>	 are you checking the sanitarium master?
[13:35:44] <Amir1>	 that'll take a couple of days though 
[13:35:51] <marostegui>	 Riiiiight
[13:35:54] <marostegui>	 No rush anyways
[13:36:15] <marostegui>	 Worst case scenario we will have an email on monday saying that there's private data on codfw sanitarium hosts
[13:36:20] <marostegui>	 Which is "fine" as they are not in use
[13:36:24] <marostegui>	 I will check on FRiday though
[13:36:28] <marostegui>	 Before leaving
[13:36:43] <Amir1>	 I can run it on Saturday/Sunday
[13:36:52] <marostegui>	 Nah, no problem
[13:36:55] <Amir1>	 wait, this weekend is Berlin Pride, nvm
[13:36:56] <marostegui>	 It is not a big deal at all
[13:37:01] <marostegui>	 Haha
[13:37:10] <Amir1>	 I will be drunk most of the weekend I assume
[13:42:04] <question_mark>	 hey all, i added the objectives we had discussed in the OKR meetings into betterworks, so you can now add your KRs aligning to them as discussed
[13:42:07] <question_mark>	 if in doubt, please talk to me :)
[13:43:33] <jynus>	 I have one question for marostegui regarding OKRs- how are you seeing *that* objective you thought I could help with- do you think it could become a bigger thing with the data you have so far?
[13:43:51] <marostegui>	 jynus: Yeah, I don't know yet :(
[13:44:02] <jynus>	 basically I want to organize myself and think of a worse case situation
[13:44:05] <marostegui>	 Yeah
[13:44:19] <marostegui>	 I think worst case, we can try to see how it works in different versions (10.7 or 10.8 even) 
[13:44:32] <marostegui>	 But so far I don't think we need you for now other than brainstorms 
[13:44:40] <marostegui>	 But hard to tell
[13:44:41] <jynus>	 or... 8.X :-D ?
[13:45:10] <marostegui>	 haha
[13:45:14] <marostegui>	 I have actually thought about it
[13:47:39] <Amir1>	 question_mark: I created the mw objective earlier today so I could create the KRs but now we have two "Improve stability and scalability of MediaWiki database usage" and I can't move the KR to under yours so I could delete it
[13:47:50] <Amir1>	 I can delete yours and make modifications 
[13:47:59] <question_mark>	 that works too, if you can assign it to me
[13:48:16] <marostegui>	 Uff I forgot about betterworks!
[13:48:18] <marostegui>	 So many stuff going on
[13:48:32] <question_mark>	 no worries, i asked to do it by end of week ;)
[13:48:38] <question_mark>	 and i only just put in my objectives
[13:49:43] <RhinosF1>	 marostegui: it's blkwiki not blwiki
[13:49:50] <Amir1>	 question_mark: yup done now but I can't delete your empty objective now
[13:50:00] <marostegui>	 RhinosF1: yeah thanks, I noticed, it is not there anyways :)
[13:50:16] <RhinosF1>	 np
[13:52:56] <question_mark>	 Amir1: done
[13:53:13] <Amir1>	 thanks
[14:04:50] <andrewbogott>	 marostegui: sorry I missed your ping. I'm adding new grants to m5 for some new servers, the new grants are in this diff:  https://gerrit.wikimedia.org/r/c/operations/puppet/+/815378/1/modules/profile/templates/mariadb/grants/production-m5.sql.erb  I'm happy to just run those commands myself but I don't know if the process has changed... it's still all done by hand right?
[14:05:02] <andrewbogott>	 (also if you're rushing about handling the bad router issue then this can definitely wait!)
[14:05:21] <marostegui>	 andrewbogott: Hello! I can do it no, worries
[14:05:22] <jynus>	 ongoing transference: https://grafana.wikimedia.org/goto/2ZBPP0RVk?orgId=1 https://grafana.wikimedia.org/goto/NhqmsAgVk?orgId=1
[14:05:28] <marostegui>	 Let me get to it now andrewbogott 
[14:05:29] <andrewbogott>	 thanks!
[14:09:43] <marostegui>	 andrewbogott: Grants added, I have not merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/815378 as there's more stuff there apart from the sql file, but it can be merged anytime
[14:11:06] <andrewbogott>	 sounds good, thanks again
[14:38:39] <Amir1>	 marostegui: is it okay if I turn off innodb_flush_neighbors on a couple of pc dbs (and a couple of replicas in s6?) I'm sure all of pc is ssd and I double check before enabling anything on s6
[14:38:52] <marostegui>	 Amir1: yep, that's fine!
[14:39:02] <Amir1>	 does it need a restart?
[14:39:06] <marostegui>	 Amir1: All s6 are SSDs
[14:39:12] <marostegui>	 Amir1: no, just set global innodb.... = 0;
[14:39:18] <Amir1>	 awesome
[14:39:53] <Amir1>	 from my queries I think all of es is on hdd, a lot are not accessible 
[14:40:13] <Amir1>	 stuff like WARNING:root:Host db1123.eqiad.wmnet has unknown media type for drive ATA....
[14:40:33] <marostegui>	 es on hdd?
[14:40:37] <marostegui>	 I think we replaced all of them already
[14:41:03] <Amir1>	 let me send you the report 
[14:41:31] <volans>	 for all the disks or just 2? if it's 2 per hosts might be just the OS and/or some additional embedded one (not sure if the SDcards are reported by those commands I gave you)
[14:42:22] <Amir1>	 volans: I see it for all of raid I think https://phabricator.wikimedia.org/P31525
[14:42:36] <marostegui>	 Amir1: you are right, they are spinning disks, just check their purchase https://phabricator.wikimedia.org/T235820
[14:42:46] <marostegui>	 I think we did that to save money to have more space
[14:43:17] <marostegui>	 yep, confirmed, SATA by looking up their serial number from megacli
[14:43:33] <Amir1>	 4tb is not much space :D
[14:43:48] <volans>	 I don't see db1123 in that paste :D
[14:44:11] <Amir1>	 volans: that's only for es
[14:44:17] <Amir1>	 all of it is long
[14:44:23] <marostegui>	 Amir1: it is 12x2TB disks :)
[14:44:58] <volans>	 ah sorry
[14:45:01] <Amir1>	 volans: https://phabricator.wikimedia.org/P31526
[14:45:09] <Amir1>	 all are protected by nda
[14:45:55] <volans>	 thx
[14:45:56] <Amir1>	 marostegui: hmm, should I divide it by three (RAID?)
[14:46:19] <Amir1>	 just curios 
[14:46:50] <marostegui>	 Amir1: No, it is RAID10, so by 2
[14:46:52] <marostegui>	 11TB usable
[14:46:56] <marostegui>	 (more or less)
[14:47:01] <Amir1>	 cool
[15:12:49] <Amir1>	 I don't know how much it's related to our work or how much it's related to active/active work but codfw requests are erroring because heartbeat can't find an s8 db on s1: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-2022.07.20?id=1pMgHIIBWJDkYQXofH04
[15:14:57] <marostegui>	 That's weird
[15:15:00] <marostegui>	 that's a new host though
[15:16:00] <marostegui>	 But it is replicating in s8 and s1 (multi-instance)
[15:16:06] <marostegui>	 and it all seems fine there
[15:16:16] <marostegui>	 https://orchestrator.wikimedia.org/web/search?s=db2167
[15:16:18] <marostegui>	 No lag or anything
[15:16:22] <marostegui>	 So heartbeat is working as expected
[15:16:31] <marostegui>	 unless i pooled it on the wrong places
[15:16:32] <marostegui>	 let me check
[15:16:43] <Amir1>	 marostegui: it's pooled https://noc.wikimedia.org/dbconfig/codfw.json
[15:16:52] <marostegui>	 yeah, i know it is pooled
[15:16:56] <marostegui>	 I found the issue
[15:16:58] <Amir1>	 I mean both instances 
[15:17:00] <Amir1>	 :D
[15:17:18] <Amir1>	 sigh 
[15:17:18] <marostegui>	 Fixed!
[15:17:23] <Amir1>	 Danke
[15:17:25] <marostegui>	 https://phabricator.wikimedia.org/P31531
[15:17:28] <marostegui>	 Thanks for the heads up
[15:17:33] <marostegui>	 So many things at the same time :(
[15:17:41] <Amir1>	 thank you for fixing it
[15:18:04] <Amir1>	 we should have something to avoid pooling stuff like this
[15:18:46] <Amir1>	 marostegui: errors are normal now \o/ https://logstash.wikimedia.org/goto/ede19894b5537e826318ec2c90646145
[15:23:52] <marostegui>	 <3