[08:46:56] I am not in a hurry to get it done, but defragmentation of commons was still pending on codfw, right? [08:46:56] I am not in a hurry to get it done, but defragmentation of commons was still pending on codfw, right? [08:47:22] as there is 300GB of size difference between s4 backups [08:47:22] as there is 300GB of size difference between s4 backups [08:48:36] yep [08:48:36] yep [08:48:46] I also see a significant difference between s2-eqiad and s2-codfw: 894 GB vs 1061 GB [08:48:47] I also see a significant difference between s2-eqiad and s2-codfw: 894 GB vs 1061 GB [08:48:50] I was planning to tackle it next week, as this with clinic duty... [08:48:51] I was planning to tackle it next week, as this with clinic duty... [08:49:20] There are lots of optimizations pending in general [08:49:20] There are lots of optimizations pending in general [08:49:25] As part of the pending work from the switch back [08:49:25] As part of the pending work from the switch back [08:49:38] ok, then I will wait for those to proceed and only compare then [08:49:38] ok, then I will wait for those to proceed and only compare then [08:49:42] yeah [08:49:42] yeah [08:49:49] it won't be trustable until they are all done [08:49:49] it won't be trustable until they are all done [08:50:03] I check the size not because I am short for it [08:50:03] I check the size not because I am short for it [08:50:17] but because it could be an indication of me missing things from the backup [08:50:17] but because it could be an indication of me missing things from the backup [08:50:40] but right now it is just too complicated to compare until optimizations [08:50:40] but right now it is just too complicated to compare until optimizations [08:51:50] yeah, let's wait [08:51:50] yeah, let's wait [10:54:17] for s2, I think they are flaggedtemplates clean ups IIRC [10:54:17] for s2, I think they are flaggedtemplates clean ups IIRC [10:57:15] and it'll be more, flaggedimages is being dropped [10:57:15] and it'll be more, flaggedimages is being dropped [10:59:33] Re: "P_S and tendril comments"- how sure are you that adding extra comments would make them useless? right now, they sometimes add the IP and other stuff and they keep working! [10:59:33] Re: "P_S and tendril comments"- how sure are you that adding extra comments would make them useless? right now, they sometimes add the IP and other stuff and they keep working! [10:59:58] I don't know for sure [10:59:58] I don't know for sure [11:00:08] I think both remove comments for normalization, although I am unsure about the details [11:00:08] I think both remove comments for normalization, although I am unsure about the details [11:00:19] mentioning it not to push for a particular direction [11:00:19] mentioning it not to push for a particular direction [11:00:33] jynus: the req_id can differ in each edit [11:00:33] jynus: the req_id can differ in each edit [11:00:37] but because maybe there is one less problem we have to solve :-) [11:00:38] but because maybe there is one less problem we have to solve :-) [11:00:44] if you are sure it removes the comment, then it's fine for me [11:00:44] if you are sure it removes the comment, then it's fine for me [11:01:15] I am not 100% sure, I think both tools do some kind of normalization- a test could be done to check it [11:01:15] I am not 100% sure, I think both tools do some kind of normalization- a test could be done to check it [11:01:36] if we can push and then revert if it doesn't work, I am fine with that [11:01:36] if we can push and then revert if it doesn't work, I am fine with that [11:02:00] or even easier- running a long running query with the same format, and see what happens :-) [11:02:00] or even easier- running a long running query with the same format, and see what happens :-) [11:02:05] maybe 2 [11:02:05] maybe 2 [11:02:32] I don't really have the time for that until I am back from holiadys [11:02:32] I don't really have the time for that until I am back from holiadys [11:02:35] I know you are busy this week, marostegui, mostly commenting for the other people that coudl research them on their own [11:02:36] I know you are busy this week, marostegui, mostly commenting for the other people that coudl research them on their own [11:02:36] I have enough on my plate already [11:02:36] I have enough on my plate already [11:02:43] see my comment :-) [11:02:44] see my comment :-) [11:03:30] jynus: I can try that when I have a bit of time [11:03:30] jynus: I can try that when I have a bit of time [11:03:52] marostegui: let me know if I can do something to take off stuff from your plate! [11:03:52] marostegui: let me know if I can do something to take off stuff from your plate! [11:03:56] I think the suggester was Timo, maybe he can try on his own :-D [11:03:56] I think the suggester was Timo, maybe he can try on his own :-D [11:07:17] Amir1, I think most of the issues on mw layer came from the lost features in HHVM or "long running queries" logging :-( [11:07:17] Amir1, I think most of the issues on mw layer came from the lost features in HHVM or "long running queries" logging :-( [11:07:23] *of [11:07:23] *of [11:07:58] yeah, one thing: Someone said query killer has been disabled. Is that true? [11:07:58] yeah, one thing: Someone said query killer has been disabled. Is that true? [11:08:04] no [11:08:04] no [11:08:12] Do you have access to the IR doc? [11:08:12] Do you have access to the IR doc? [11:09:44] yup [11:09:44] yup [11:09:51] I answer the question there [11:09:51] I answer the question there [11:10:00] No idea why someone mentioned it is not installed [11:10:00] No idea why someone mentioned it is not installed [15:15:22] when I reached 99.9% backup completion I thought the most difficult part was done [15:15:22] when I reached 99.9% backup completion I thought the most difficult part was done [15:15:57] now I realized I just did the "easy" part, and the 0.1% pending was the one that will give me more problems [15:15:57] now I realized I just did the "easy" part, and the 0.1% pending was the one that will give me more problems [15:27:23] hi all could someone give me an ideia of what orchestrator, i notice it is only avlible to the ops group so suspect it has some sensetive information or possibly enables some poer commands. related to the following change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/722884 [15:27:24] hi all could someone give me an ideia of what orchestrator, i notice it is only avlible to the ops group so suspect it has some sensetive information or possibly enables some poer commands. related to the following change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/722884 [15:29:21] jbond: it will eventually be opened up, but not quite yet. so for now it should stay as it is [15:29:21] jbond: it will eventually be opened up, but not quite yet. so for now it should stay as it is [15:30:53] marostegui: thanks is it ok to add sre-admins (group for sre's without root) access or should i just drop that for now [15:30:53] marostegui: thanks is it ok to add sre-admins (group for sre's without root) access or should i just drop that for now [15:31:31] which sres without root? [15:31:31] which sres without root? [15:31:37] I mean what's that group? [15:31:37] I mean what's that group? [15:33:19] junior SREs without yet full global root [15:33:19] junior SREs without yet full global root [15:33:23] its a new groups for sre's that are still being onboarded. it allows them access to all serveres but no sudo rights. basicly we tust this group with sensetive information but add restrictions e.g. no sudo so they can't shoot them self in the foot [15:33:23] its a new groups for sre's that are still being onboarded. it allows them access to all serveres but no sudo rights. basicly we tust this group with sensetive information but add restrictions e.g. no sudo so they can't shoot them self in the foot [15:33:47] so I'd say it should be ok if they have RO access to orchestrator [15:33:47] so I'd say it should be ok if they have RO access to orchestrator [15:34:05] yes or junior sre's, its a fairly new group with only one member but i the intention it to use it more and more for onboiarding and junieor sre's [15:34:05] yes or junior sre's, its a fairly new group with only one member but i the intention it to use it more and more for onboiarding and junieor sre's [15:35:19] it is fine yeah [15:35:19] it is fine yeah [15:35:25] no problem with that [15:35:25] no problem with that [15:35:37] great thanks [15:35:37] great thanks [15:36:39] marostegui: remind me, orchestrator is still just RO right? [15:36:39] marostegui: remind me, orchestrator is still just RO right? [15:37:08] for mostly everyone yes [15:37:08] for mostly everyone yes [15:37:30] so the plan is to keep it RO for 'ops' and RW just for DBAs? [15:37:30] so the plan is to keep it RO for 'ops' and RW just for DBAs? [15:40:53] short term yes [15:40:53] short term yes [15:40:59] long term, hopefully not [15:40:59] long term, hopefully not [15:41:17] I actually want, at some point, to let orchestrator recover the topologies itself [15:41:17] I actually want, at some point, to let orchestrator recover the topologies itself [15:41:32] but we are far from that [15:41:32] but we are far from that [15:42:07] ack makes sense [15:42:07] ack makes sense