[01:07:27] sonarqube being fun. I added a few lines to conftest.py, and now sonarqube fails with a line out of range error. afaict sonarqube is running against the parent patch and thus is looking at the older, slightly shorter file [08:51:42] errand [10:05:20] lunch [10:05:27] lunch 2 [11:48:21] lunch 3 [13:31:46] dcausse: I did recreate templates, but some data would be nice - index won't be created before that... [13:36:59] huh, actually ES refused to create index when indexing, returning 404 instead... [13:39:11] ah, auto_create is off on * [14:53:26] \o [14:53:51] gehel: you might have some ideas, since you do sonar-ish things (although i suspect this is a jjb problem): https://phabricator.wikimedia.org/T283778 [14:54:45] the tl/dr is nothing can merge to mjolnir that lengthens a file because sonar applies the coverage.xml against the wrong branch, and gets line out of bounds errors. fun :) [15:44:40] dcausse: are you aware of any issues that would cause rampant inflation of the `wikidata.jnl` size? [15:45:05] `wdqs2004` inexplicably has a 2.5TB `wikidata.jnl` (thus causing a disk space critical despite it being on raid0 which has 2x the space of raid10). By comparison `wdqs1006` has a 1.6T `wikidata.jnl`, and `wdqs2004` and `wdqs2001` have a 975G `wikidata.jnl`. It's not clear why there's such a big divergence [15:45:25] ryankemper: maybe free allocators bug, look at the graph and see if the servers is losing its allocators too quickly [15:46:11] wdqs2004 seems to lose allocators quicker [16:23:04] mpham: just a future request, please no red/black as distinction colors. They are the same for me unless they are large blobs of color :) [16:23:24] i've never been able to find an enemy on in-game radars that use red on black either... [16:24:36] ryankemper: e.g. https://grafana-rw.wikimedia.org/d/000000489/wikidata-query-service?viewPanel=32&orgId=1&refresh=1m&from=now-30d&to=now [16:25:01] the sudden drops are the server likely misbehaving and having disk issues [16:25:26] "disk issues" I mean larger journal sizes [16:26:04] we don't know what causes this [16:28:35] dcausse: ack. i've got a fresh journal transferring to wqds2004 (which had the 2.5TB jnl) and wdqs1006 (which had 1.6T), so they should be back to ~975 GB after the transfer finishes up [16:28:50] thanks! [16:49:31] * ebernhardson loves finding multiple ways to do things in another teams codebase, never sure which is the one i'm supposed to follow :P [16:57:13] dinner [17:30:26] hmm, airflow webserver fell over and nothing complained [17:30:33] shouldn't systemd be yelling or something? [17:31:13] mostly amounts to FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/'] [17:31:38] yea, / is full [17:55:33] not really sure whats changed, usage has been consistent 39.5-40.2 GB over last 30 days with daily sawtooth of ~300MB. Then about 50 minutes ago it added 5GB and capped out. Sadly all the out of disk errors spammed journalctl and it only goes back 25 minutes [18:03:54] hmm, so at 16:59 we start seeing a large sqlalchemy stack trace every ~5ms, for each separate process (wait_for_data, scheduler, webserver, etc) for too many connections [18:04:46] so, sql server we use filled it's connection pool, spammed the airflow logs until the disk was full, then things finished falling over [18:21:53] well...i dunno what to do with this. file a task and wait for it to happen again. Maybe re-partition the instance to separate logs from system [19:15:51] ebernhardson: about T283778, not sure at all what's going on. I'll try to check with hashar next Tuesday [19:15:52] T283778: SonarScanner fails with IllegalStateException: line nnn is out of range - https://phabricator.wikimedia.org/T283778 [19:15:59] * gehel is unable to navigate JJB without help [19:23:39] gehel: i made a patch that in theory fixes it. But mostly i'm copying from somewhere else that calls sonar and thinking this looks plausible :) [19:38:36] ebernhardson: looks plausible to me as well :) Let's see what the expert says [20:00:46] ebernhardson: just saw your note about red/black. thanks for letting me know! I'l keep it in mind in the future