[08:19:51] <brouberol>	 dsaez_: your home directory on stat1008 is taking 720GB (about 10% of the available disk space on that host). Can you check whether everything you have is needed? We're at 98% disk usage on that host. Thanks!
[08:21:07] <dsaez_>	 ok. I've just backuped the stat1007 there...I'll clean up today
[08:25:59] <brouberol>	 thanks!
[08:50:05] <wikibugs>	 06Data-Engineering, 10Observability-Logging, 06Traffic: Install benthos on single esams host to check performances under higher load - https://phabricator.wikimedia.org/T365968 (10Fabfur) 03NEW
[11:04:59] <wikibugs>	 07Analytics-Data-Problem, 06Data-Platform, 06Movement-Insights: Unique devices per country spikes on wikifunctions - https://phabricator.wikimedia.org/T364872#9834495 (10TAndic) Hi @Mayakp.wiki -- the only information I have is the Central Notice calendars, I checked the admin [[ https://meta.wikimedia.org/w...
[12:53:47] <elukey>	 brouberol, stevemunene o/ - I see very little space for the root partition on dse-k8s-worker1001 on alerts.w.o
[13:48:15] <wikibugs>	 (03PS1) 10Gmodena: webrequest: add error schema. [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1036272 (https://phabricator.wikimedia.org/T314956)
[13:48:40] <wikibugs>	 (03CR) 10CI reject: [V:04-1] webrequest: add error schema. [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1036272 (https://phabricator.wikimedia.org/T314956) (owner: 10Gmodena)
[13:53:15] <wikibugs>	 (03PS2) 10Gmodena: webrequest: add error schema. [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1036272 (https://phabricator.wikimedia.org/T314956)
[13:53:42] <wikibugs>	 (03CR) 10CI reject: [V:04-1] webrequest: add error schema. [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1036272 (https://phabricator.wikimedia.org/T314956) (owner: 10Gmodena)
[13:55:05] <wikibugs>	 (03PS3) 10Gmodena: webrequest: add error schema. [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1036272 (https://phabricator.wikimedia.org/T314956)
[15:13:36] <wikibugs>	 06Data-Engineering: Add page-title to the x_analytics header - https://phabricator.wikimedia.org/T366004 (10JAllemandou) 03NEW
[15:17:31] <wikibugs>	 06Data-Engineering: Add page-title to the x_analytics header - https://phabricator.wikimedia.org/T366004#9835331 (10Krinkle) See also: * {T304362} * {T240676}
[15:26:47] <wikibugs>	 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Research, 06Structured-Data-Backlog: Make HTML Dumps available in hadoop - https://phabricator.wikimedia.org/T305688#9835371 (10leila)
[15:35:11] <wikibugs>	 06Data-Engineering: Add page-title to the x_analytics header - https://phabricator.wikimedia.org/T366004#9835448 (10Krinkle) Based on <https://wikitech.wikimedia.org/wiki/Debugging_in_production>:  `lang=shell krinkle@mwdebug1002:~$ curl -I --connect-to ::$HOSTNAME 'https://test.wikipedia.org/wiki/Main_Page' … X...
[15:55:58] <brouberol>	 elukey: checking
[15:56:18] <elukey>	 brouberol: o/
[15:56:23] <brouberol>	 it's the /var/lib/docker/overlay dir
[15:56:31] <elukey>	 I just ran apt-get clean to free the root partition
[15:56:37] <brouberol>	 thanks
[15:57:39] <brouberol>	 it's weird, the biggest offender is a diff overlay pulling llvm (+7GB) 
[15:59:09] <brouberol>	 how often do we run something akin to docker system prune ?
[16:03:22] <elukey>	 overlay is used by the docker containers when running though
[16:03:30] <elukey>	 and /var/lib/docker is on a separate dir
[16:03:35] <elukey>	 I think it was the root partition the problem
[16:03:44] <elukey>	 re: docker system prune - not sure if we run it
[17:56:02] <brouberol>	 you're right, I used `ncdu /` instead of `ncdu -x /`
[17:56:23] <brouberol>	 causing /var/lib/docker data to appear
[20:19:44] <wikibugs>	 06Data-Engineering, 10Observability-Logging, 06Traffic: Upgrade Benthos package on cp hosts - https://phabricator.wikimedia.org/T366031 (10Fabfur) 03NEW
[20:20:41] <wikibugs>	 06Data-Engineering, 10Observability-Logging, 06Traffic: Upgrade Benthos package on cp hosts - https://phabricator.wikimedia.org/T366031#9836148 (10Fabfur)