[07:33:50] Amir1: https://phabricator.wikimedia.org/P17065 [07:33:55] Probably worth a ticket! [07:35:09] hmm, I'm thinking if I should squeeze even more from it [07:35:50] we can try and see how much it decreases on those two hosts to see if it is worth the time [07:36:08] it's a bit hard to measure :( [07:36:24] I meant in terms of disk space saving [07:36:55] then I'm not following [07:37:01] * Amir1 needs coffee [07:37:40] Amir1: So from your squeeze comment I understood you can clean it up further? [07:38:18] yup, I need to make a change, get it merged and backported and then run this script again but yes :D [07:39:13] Ah I see, so it does need a bit of work. So from 30GB to 17GB is almost 50% which is really good already [07:47:01] for ruwiki, it'll be fun [07:54:10] Amir1: Maybe we should just create a ticket once it is ready to be optimized on all the wikis that have flaggedtemplates enabled, and get it done before we switch back to equad [07:54:11] eqiad [07:54:50] yeah, that's my thinking [07:55:59] Let's go for that then I would say [08:15:15] yup. I probably should get list of the biggest ones in infra. Probably from backups? [08:17:03] For flaggedtemplates table? [08:18:09] yup [08:19:11] Yeah, I can help with that [08:19:44] feel free to put it in T289249 [08:19:45] T289249: flaggedtemplates table should not keep the whole history of all revisions - https://phabricator.wikimedia.org/T289249 [08:21:10] Amir1: Will do - I am going into a meeting in a sec, so I will do it later [08:21:20] Thanks! [08:21:23] have fun mwhahaha [08:22:28] Amir1: wow, arwiki 177G... [08:22:56] then you need to get me a beer once this is all done :D [08:23:28] hahaha [08:43:00] Amir1: you've got your list on the tas k :) [08:43:34] Thanks ^^ [08:49:07] I'm going to parallelize run them, one wiki per section [08:52:39] my estimation is that this cleanup will remove basically as much as image table clean up in commons (with flaggedimages drop) [08:57:50] That's pretty good [08:58:04] The flagged* tables are reaching some dangerous sizes already [08:58:07] So this clean up is very good [08:59:59] s7 started to lag bad when I started the clean up of arwiki. I made even slower [09:00:14] it's now three pages then waiting ten seconds. It'll take forever [09:07:45] Buf [09:07:54] That's going to take a long while indeed yeah [09:15:41] * godog shakes fist at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=971530 [09:15:49] and https://phabricator.wikimedia.org/T283714 on our side [09:16:28] :( [09:24:57] indeed, I'm trying to understand what's the actual fix [09:40:02] Can I get a sanity check on puppet, please? It's quite hard to figure out what things get deployed on what hosts, but I think that the mariadb::core_multiinstance role does _not_ get the mariadb::monitor::prometheus profile and thus not the prometheus::mysqld_exporter profile (instead getting the prometheus::mysqld_exporter_instance). Which means that https://gerrit.wikimedia.org/r/c/operations/puppet/+/714358 won't i [09:40:02] multiinstance hosts [09:42:23] tip: You can use the puppet compiler to know if a host will be affected or not and how [09:43:30] Emperor: the easiest way to test is to add an affected host in footer (like https://gerrit.wikimedia.org/r/c/operations/puppet/+/713503) [09:43:43] and the comment "check experimental". It'll do a PCC for you [09:44:11] (sorry if you know it already :D) [09:47:52] This is an example output for a noop change: https://puppet-compiler.wmflabs.org/compiler1003/30776/ [10:02:25] Amir1: thanks. It's best to assume I know nothing about puppet at all for the time being :-/ [10:03:38] If you have any questions ask John Bond, he is the master of puppet [10:04:11] I'm going to make their optional nit-pit changes to my CR, and I'll try and get it to PCC on a single-instance and multi-instance host at the same time. WCPGW? [10:05:17] [also, thank you for the earworm ;p ] [10:11:29] let me introduce you to my good friend manah manah https://www.youtube.com/watch?v=8N_tupPBtWQ [10:32:56] Emperor: what's wcpgw? [10:33:34] What Could _Possibly_ Go Wrong [10:34:05] Emperor: You can try to run pcc for each type of host, ie: one master, one slave, one multi-instance slave for mw (ie: db1098) and one multi-instance slave in misc (ie: db2078) [10:34:16] if all that works we can probably deploy and be ready to revert if needed :) [10:34:37] Emperor: thanks, some acronyms are very hard to process [10:35:05] no worries :) [10:41:23] yay, now I can make puppet-lint explode :( [10:42:55] seems to be having a second heredoc that's breaking it [10:49:27] I'm sure I've done something stupid, but: https://phabricator.wikimedia.org/P17066 (has both the break and the minimal change to make puppet-lint happy again) [10:50:09] I was trying to take jbond suggestion to use heredoc, and have obviously Dong It Wrong somehow... [10:56:15] any ideas? It seems unlikely that I've actually found a bug in puppet-lint... [11:00:12] can't say for sure but I think there's a space needed between # and comment [11:00:43] oh and the # is not a heredoc, you need to add one per line [11:00:55] relevant? https://github.com/rodjek/puppet-lint/issues/859 [11:02:27] Amir1: um, the heredoc is (I was trying to do) between (@EOT) and | EOT [11:03:28] aha [11:03:32] let me check [11:03:34] so not an explanation, but in your position, I would just try with content => file(), it would be equally clean and may work? [11:03:46] Amir1: cf https://puppet.com/docs/puppet/7/lang_data_string.html#lang_data_string_heredocs [11:04:13] jynus: well, I started with simple strings (and will probably revert to them), but Jbond suggested optionally replacing with heredocs instead [11:04:39] jynus: I'm running puppet-lint 2.4.2, which that issue is allegedly fixed in... [11:04:51] hmm, can be that we don't have it in puppet5, that's puppet7 [11:04:58] I'm trying to look at docs [11:05:46] https://puppet.com/docs/puppet/7/lang_data_string.html#lang_data_string_heredocs [11:05:49] ugh [11:05:50] I have a fix for you [11:05:55] why it gave puppet7 [11:05:57] content => @("EOT") [11:06:15] puppet-lint seems to like that more [11:07:17] Right, well I can do that (which turns on interpolation in the heredoc, which I don't need here). Thanks :) [11:07:44] I should see if I can produce a more minimal test-case, if it's plausible that puppet-lint isn't handling >1 uninterpolated heredoc in a file [11:08:15] look at the bug report, thet mention the problem when using more than one | symbol [11:10:10] Wait, duh, I'm a complete idiot. @(EOT) =/= (@EOT) [11:11:01] I'm not sure why puppet-lint is happy with one (@EOT) because that's the wrong syntax. [11:12:06] but with the correct syntax @(EOT) works fine in both stanzas. [11:12:21] my brilliant typing skills strike again :( [11:22:32] if you had a better editor, this would have not happened :D [12:24:31] I am going to install 10.4.21 on clouddb1015 (s4 and s6) as that's a pretty good environment to test heavy queries [12:49:22] I think I have scheduled test runs on https://gerrit.wikimedia.org/r/c/operations/puppet/+/714358 [12:55:49] Sigh, that fails on the single-instance hosts, even though it works in kormat's cloud environment on single-instance hosts :( [13:01:11] Hm, because /etc/systemd/system/mariadb.service.d is also declared elsewhere (mariadb::service). But the DB nodes in the cloud didn't have that directory... [13:03:48] Argh, kormat's cloud env has mariadb::misc::db_inventory nodes rather than mariadb::core :( [13:04:23] * Emperor goes to reconsider their life choices [13:09:19] (also, surely puppet must allow more than one thing to put stuff in the same directory (and thus want to check the directory exists?)) [13:10:40] Emperor: That's why I mentioned to check also db1117 or db2078, as those are misc hosts that run multi-instance. We currently have core, misc and clouddb hosts running mulit-instance [13:12:25] 2078 and 1096 worked fine (multi-instance) [13:12:58] db2078 has the same profile as db1117 [13:13:16] Ah, you mentioned db_inventory [13:13:28] That's not multi instance [13:20:11] marostegui: yes, I wanted to check my changes to single-instance mariadb wouldn't break multi-instance. It turns out that while my changes work in kormat's test env (with mariadb::misc::db_inventory: ) it doesn't work on db1119 nor db1163 (mariadb::core), because mariadb::core uses mariadb::service which wants to use /etc/systemd/service/mariadb.service.d/ wheras mariadb::misc::db_inventory does not [13:20:38] (I think because it just installs the mysqld packages and doesn't do anything service-related) [13:34:09] But the variety of slightly-different mysql/mariadb roles/profiles is very confusing [13:37:58] I am trying to avoid the conclusion that the last 2 and a bit days' of work on this have been entirely wasted [14:06:52] But systemd::unit will want to declare the resource /etc/systemd/service/mariadb.service.d/ and mariadb::service also wants to, and I don't think that's fixable. I mean, one could presumably faff around with virtual resources, but I don't think that necessarily helps very much [14:08:44] The strange this is this happens even though neither of those systems is creating /etc/systemd/service/mariadb.service.d/ ; it's just that mariadb::service _might_ want to that's causing everything to fail? [14:15:24] I think you will probably want some feedback from k*rmat when back, she will know much better how to organize puppet for mariadb [14:16:03] for what you discussed here, the solution looks fine, is the implementation that causes you problems [14:16:44] there are a few solutions to your issue- 2 technically independent pieces of code trying to declare the same thing [14:17:07] e.g. you can export your needs on one point of the tree and centralize its implementation elswere [14:18:11] but there are many ways to do the same, I would wait for someone with a global view to give you more specific advice- I *don't think* you work was a waste at all [14:19:09] learning == investment :-D [14:19:36] I am sure there will be a solution :-)