[09:11:37] Amir1: is it possible to tell autoschema not to depool for a schema change? i have one to run that doesn't require depooling. [09:13:36] kormat: I'm not following. For any schema change it's better to depool to avoid collisions and error floods ("the schema of this table has changed while being queried") [09:14:08] Amir1: there are some schema changes that we run against a primary, live [09:14:36] e.g. T303603 [09:14:37] T303603: Add actor and comment columns to cu_changes - https://phabricator.wikimedia.org/T303603 [09:15:45] eqiad primary? it never depools eqiad primary [09:15:57] (eqiad = active dc) [09:16:04] my point is, for some schema changes we do not depool an instance. [09:16:42] you just run it on master with replication? [09:16:50] on master, without replication. [09:17:14] it never depools the master [09:17:20] * kormat facepalms [09:17:24] that's not the question :P [09:17:37] i'm saying that: for some schema changes, we don't need to depool. can auto_schema support that? [09:18:20] or, to put it another way: if it's safe to run a given schema change against the primary without depooling it, then it's also safe to run it against replicas [09:18:24] and that would save a ton of time [09:18:27] some is vague, I need to know why don't depool and which hosts [09:19:23] "if it's safe to run a given schema change against the primary without depooling it, then it's also safe to run it against replicas" I'm not sure this is correct [09:20:21] a schema change that can be done live in master doesn't necessarily mean it can be done live in replicas [09:20:46] even schema change that is safe to run on master at late evening might be not safe to run in noon [09:21:24] (you have to add lock_wait_timeout and stuff like that, otherwise writes pile up) [09:22:04] that is precisely what we have done in the past [09:23:41] as for what needs depooling and what doesn't, that's determined by https://mariadb.com/kb/en/innodb-online-ddl-overview/ [09:24:37] than I need to double check and see what I'm missing [09:25:23] i'm guessing from the lack of response that the answer is "no, auto_schema does not support this" [09:26:40] yup the answer is no it doesn't support it [09:27:09] damn, ok. [10:05:47] if there are 30 possible things that could go wrong on a reimage, I think I hit 29 [10:06:18] hw issues, bios firmwares, automation issues, partman issues, partitioning issues, os version issues [10:06:21] jynus: achievement-almost-unlocked? [10:06:42] I feel like a videogame character grinding levels of XP [10:08:41] and I have the same general problem as Emper*r- you cannot just start from a clean slate because I have 80 TB of that on this host I cannot lose [10:09:13] s/that/data/ [10:22:21] nope, spoken too soon, the host still doesn't book [10:22:23] *boot [15:50:05] I belive both Manuel and Lukasz are off today- should we add something to the SRE meeting doc? [15:58:59] I am "making up" some stuff hopefully you can review it