[05:08:16] Amir1: s3 is all yours [05:49:00] So all those s6 errors for labswiki from cloudweb1004 should be ignored? [05:50:23] andrewbogott: can we do something to get rid of those errors? there are a lot, and I am not sure if they are valid or they should just be ignored [07:47:31] marostegui: good morning, thanks [07:54:07] marostegui: the read only ones are valid. I see a few times stashbot has failed for moritzm this morning. [08:53:05] marostegui: you know we have these long-running templatelink alter tables? [08:53:24] I have good news for you, we will have to do another round for dropping the columns as well :P [08:54:24] (jokes aside: I think it'll be much faster because what determines the time to run the schema change is the size of the table after schema change, not before, and those will get 10-20 times smaller) [08:54:59] cloudweb1003/1004 should not be receiving traffic, so I'm not exactly sure why we're serving ro errors [08:55:16] unless mediawiki caches that information somehow and the old servers end up using the cached information [13:08:04] jbond: I have ammended your wikitech edit from yesterday with some more complete and up-to-date info [13:09:50] marostegui: awesome thanks <3 [13:12:26] Finally blkwiki arrived to db2094! [13:12:32] Going to sanitize it [13:27:19] marostegui: I'm not sure what the deal is with those errors but we can move forward with the new hosts on the chance that that's related... for those I have another round of grants: https://gerrit.wikimedia.org/r/c/operations/puppet/+/816026/2/modules/profile/templates/mariadb/grants/production-core.sql.erb [13:27:51] I'm unclear on if i should just be setting up those grants myself... seems like you probably don't want 40+ random SREs modifying your db servers :) [13:28:59] Amir1: ^ do you want me to run that or you would? (as I saw you were doing stuff yesterday) [13:30:10] marostegui: I can take a look soon [13:30:20] sure, no worries [13:30:21] I think the patch is to reflect the reality [13:30:25] but double check soon [13:30:49] as far as I know those grants aren't in place yet, that's for the new servers (which couldn't talk to the db last I chcked) [13:31:47] then I suggest adding the old ones too [13:34:24] really? I'm planning to decom the old servers as soon as the new ones are working. [13:34:36] But I can make two patches, one to add and one to remove :) [13:34:44] ah, I forgot [13:35:03] okay, let's get this deployed but remind me to delete the old ones as well [13:35:15] so puppet and reality match [13:35:25] ok :) [13:41:58] once I'm done with my patch (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/816169) I'll get to the grants [15:15:53] taavi: I think @section doesn't work, I don't see it in s6 files [15:16:08] let me make a patch and check PCC [15:17:17] Amir1: hmm, that file (before I touched it) uses both @section and @shard [15:17:49] yeah I know :D [15:19:01] I think I added those [15:19:14] we never checked because these are mostly for decoration [15:30:37] taavi: yup https://puppet-compiler.wmflabs.org/pcc-worker1003/36358/db1173.eqiad.wmnet/index.html [15:34:47] taavi: andrewbogott the grants are deployed now [15:34:57] let me know if anything is not working, etc. [15:43:59] so it looks like errors have stopped [15:44:03] or reduced at least [15:57:44] I think I'm calling it a day (and you should too marostegui), see you on Monday!!! [16:02:41] yeah, I will do it too [16:03:30] I want to check a couple of things before going away [16:22:16] Amir1: thanks! the grants seem to be working [16:24:21] Errors are gone [16:37:22] Amir1: thanks! [16:38:18] taavi: I'm going to wait until Monday to actually pool those two new hosts (cloudweb100[34]) , to avoid weekend comedy.