[08:29:05] godog: I'm going to be quite busy with clinic requests this morning, so if you have some hardware-tuits, that'd be kind (a bunch of things got approved overnight, so I need to get them done now) [08:33:17] Emperor: sure I'll take a look at that host [08:33:31] good luck with clinic duty tho [08:48:56] TY [09:11:04] we might have a bot scrapping s1 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=now-24h&to=now&refresh=1m&var-site=eqiad&var-group=core&var-shard=s1&var-role=All [09:36:18] Emperor: {{done}} https://phabricator.wikimedia.org/T297933 [09:38:11] I've also added a bit of docs here https://wikitech.wikimedia.org/wiki/MegaCli#Replace_individual_disks_in_JBOD [09:54:49] re: the replacement, I'll be out next week [11:53:26] me too (other than Monday) [15:56:58] siigh so thanos-be hosts are filling up with logs from object-server, specifically it looks like a lot of traffic from tegola [15:57:40] I don't think we can leave it for the holidays as thanos-be2004 is already full [15:58:09] I'm opening a task [16:02:49] T297959 [16:02:49] T297959: thanos-be hosts filing up root filesystem with logs - https://phabricator.wikimedia.org/T297959 [16:02:50] :( [16:03:49] ingorant question, what is tegola? [16:05:52] the map tiles server [16:06:05] I'm prepping a patch to ban tegola's access logs [16:06:17] seems like the simplest solution to me [16:07:50] https://gerrit.wikimedia.org/r/c/operations/puppet/+/748150 that is [16:08:54] Emperor: ^ what do you think ? [16:09:43] I'm missing a 'then' [16:26:42] +1 [16:26:55] I mean, deploying late on a Friday afternoon WCPGW? :) [16:29:54] haha! [16:30:45] thank you, I'll deploy it now [16:32:13] seems like to be working to me [16:32:40] thanos access logs are still there but they are tiny [16:57:12] I take that back, they are not tiny but smaller [19:50:50] afk for a bit [21:00:32] back