[07:44:42] jynus: question: https://gerrit.wikimedia.org/r/c/operations/puppet/+/820773/5/hieradata/hosts/db1133.yaml#b3 any reason to enable notifications on the test host? (I am going thru backlog, so probably missing context) [07:44:59] they are not enabled [07:45:20] ah they are disabled on icinga directly? [07:45:54] https://gerrit.wikimedia.org/r/c/operations/puppet/+/820773/5/hieradata/role/common/mariadb/core_test.yaml [07:46:09] ah sweet! [07:46:10] thanks [07:46:28] that way you can check only hosts that are temporary enbled by host [07:46:32] cleaner [07:46:44] indeed! thank you [07:46:58] now, if that is broken, it may be [07:47:38] I may have to move it to the profile, but the general idea should be staying [08:11:38] I think this should fix the issue? https://gerrit.wikimedia.org/r/c/operations/puppet/+/824131/2/hieradata/role/common/mariadb/core_test.yaml [08:43:55] marostegui: https://twitter.com/largedatabank/status/1559651463919452161 [08:53:53] hahahaha [12:18:52] https://phabricator.wikimedia.org/T315177#8161356 [12:18:57] 😭 [12:21:22] sigh, another swift disk gone. Must be a day ending in 'y' [12:21:58] Amir1: oopsie :( [12:23:06] oh, wait, no, make that two, in the same host :( [12:27:04] Not sure why the megaraid monitoring isn't noticing :( [12:28:43] you are having fun too it seems :( [12:29:43] it feels a bit like swift hw is failing faster than I can get it replaced ATM [12:40:09] do you know if eqiad dc ops prefer to assign them hw tasks? I know papaul prefers it, but not sure what is the prefered way for eqiad [12:40:20] *for me to [12:40:29] jynus: for eqiad I normally ping willy [12:40:35] ok [12:40:38] jynus: Or assign to willy and he reassigns accordingly [12:40:52] thanks, that's exactly what I wanted [12:41:09] interesting; I've tended to assign eqiad to cmjohnson; [12:41:33] well, I can see that could be unwise, if he is on vacations [12:41:36] that's always been a bit "not sure if correct", so I'll try marostegui's approach this time :) [12:41:44] Emperor: In the past chris was alone in eqiad, but now with John it is different [12:41:54] while sadly for codfw there is not much choice [12:42:24] s/sadly/I'm afraid/ [12:42:25] Just add ops-eqiad tag and leave unassigned [12:42:33] We will take care of it [12:42:41] cmjohnson1: thank you! [12:42:49] Anytime! [12:43:04] cmjohnson1: ah, thanks, useful data :) [12:43:07] * Emperor unassigns T315437 [13:16:47] x1 shrinked again this week: Last dump for x1 at codfw (db2101) taken on 2022-08-16 01:25:11 is 40 GiB, but the previous one was 46 GiB, a change of -12.8 % [13:22:54] jynus: that's me. https://phabricator.wikimedia.org/T310428 [13:23:14] and T308084 everywhere [13:23:15] T308084: Reduce DB space used by Echo notifications - https://phabricator.wikimedia.org/T308084 [13:23:25] ok [13:25:39] last I checked it was on F in list of wikis, it should get much smaller once done [14:26:32] I've moved https://gitlab.wikimedia.org/repos/data_persistence/rebasing_demo to be under our team rather than my personal space