[07:55:21] inflatador_: I have fixed zk logging on Bookworm with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1031465 ; zk on flink-zk can be restarted at any time to get logging working [07:56:09] I haven't restarted anything tho, will leave that to you when comfortable [08:08:51] Won’t be at triage [13:32:49] inflatador: in case you missed it -> inflatador_: I have fixed zk logging on Bookworm with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1031465 ; zk on flink-zk can be restarted at any time to get logging working I haven't restarted anything tho, will leave that to you when comfortable [16:38:33] randomly curious, mobile search pageviews increase ~30% between may 7 to 9th. very consistent 0.6% before and 0.8% after. [16:38:58] i suppose it's a %, not sure if the top or the bottom changed :P [16:39:26] dr0ptp4kt: i think the dashboards should be ready now [16:40:44] * dr0ptp4kt runs to app feed to see what was trending on those days [16:40:54] thanks ebernhardson , will look and then do verbiage things [16:58:08] regarding the % above, what changed was the denominator. ~10M less pageviews/day compared to previous week on enwiki [17:54:04] interesting ebernhardson! hi inflatador [17:56:42] dr0ptp4kt: so curious thing, i just noticed in the superset `charts` list, that it created new charts with your adjustments insted of overwriting :S [17:58:20] ebernhardson it keeps throwing errors, so i'm having to create a new dashboard. even after i did a Save As to try to have something that would be fully under my control, after the first save, it gripes for attempts to subsequently save. did you have a workaround for that? it'll be a little messy until i'm done. i tried putting your dashboard in Draft mode to see if that would fix it, but no. [17:59:15] dr0ptp4kt: hmm, i haven't run into any errors trying to save yet. [17:59:39] i can try adding you to the datasets as well, it's the only thing left :) [18:01:36] dr0ptp4kt: ok i've added you to the datasets as well. No clue if it will help [18:02:26] thx. i'm trying to do one big change on my fork, and will have to Save As that...then after that i'll circle back and see if i can fix dashboard 520 ... here on 5/20. [18:50:28] * ebernhardson hmms...saneitizer stopping backfills from completing again. I thought we fixed that (twice now :P) [19:08:34] (needed to push the new container image into the chart, done) [20:06:26] hmm, still not shutting down appropriately :S we do have `saneitize-max-runtime: 1` in the config file, but with no units that should be 1 ms. Struggling to think of why this isn't shutting down...probably going to add some logging and try again [20:30:39] ebernhardson - it kept throwing me errors on dashboard 520. and then it was complaining at me on my own dashboard 529! so i ended up doing yet another Save As. I've set all of the charts that are employed in dashboard 530 (slug of 'search') to have you as an owner in case something needs to be done. i'd like to take https://superset.wikimedia.org/superset/dashboard/search out of draft. ok? i named 520 to have "ORIGINAL" and in draft [20:33:16] dr0ptp4kt: taking a look [20:39:40] dr0ptp4kt: looks good! thanks [20:41:12] thank you! nice work! i set to Published. [21:09:43] sigh...should have been obvious so much earlier. it looks like the problem is in prod it the saneitizer starts 2 subtasks but only has one split. The task that had a split shutdown, the other one is hanging around. [21:49:42] godog ACK, we have restarted. Also noticed that the ZK that ships w/bookworm blocks some cmds, started T365400 for this [21:49:43] T365400: Enable blocked commands in Zookeeper management interface - https://phabricator.wikimedia.org/T365400