[11:02:22] phabricator down? [11:02:46] AphrontQueryTimeoutQueryException: Query timed out after 30 second(s)! [11:02:51] marostegui: It's working for me from the UK. [11:03:44] Might be the query itself....| 1 | SIMPLE | ftngram_5 | ref | key_ngram,key_object | key_ngram | 12 | const | 1731448 | Using where; Using index; Using temporary; Using filesort | [11:08:32] marostegui: what were you doing? What page? [11:08:43] RhinosF1: A simple search [11:09:01] marostegui: search for what? [11:09:10] Certain words often fail [11:09:26] RhinosF1: Debian bullseye [11:09:27] XD [11:10:54] marostegui: https://phabricator.wikimedia.org/maniphest/query/lGy3XpVuJCC5/#R runs for me [11:11:07] Normally it's short words or really common stuff [11:11:21] Phab search is incredibly poor though [11:11:44] The all documents search was slow but it ran [11:12:00] Yeah it does work for me too now, but it failed twice [11:12:07] With the query timing out [11:12:08] https://phabricator.wikimedia.org/search/query/mlmPtnlYUrp./#R [11:12:47] marostegui: is the query borderline timing out when it runs or was it just being weird? [11:14:42] https://phabricator.wikimedia.org/T258803 is the task if you find anything interesting [11:14:52] thanks! [11:15:02] https://phabricator.wikimedia.org/T291775#7378725 is you debugging last time [11:15:20] I didn't even remember that XD [13:11:38] gehel not sure if it's known, but doesn't look great https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=maps1009&service=Disk+space [13:31:14] XioNoX: thanks for the payment Ng, but these days, maps is managed by platform engineering. Probably hnowlan [13:31:47] Cassandra running out of space is likely to cause some serious trouble :( [13:38:45] gehel: oops, old habits :) [14:42:35] gehel: content transformation, not platform [14:43:28] Looks like I haven't been following this saga closely enough [14:43:40] and who can blame you [14:43:56] it's still a bit of a mishmash, serviceops is slowly taking over [14:44:06] but indeed Hugh was supporting it [14:44:15] still is in some capacity [14:45:51] 37GBs of logs [14:48:15] imposm is doing something, and I doubt this rate of logging is sustainable [14:49:05] slack to the rescue I guess [15:01:31] ok, I hot patched rsyslog config to aleviate this [15:01:48] I 'll push changes to gerrit in 1H or so and see what I can do for imposm logging during the holidays [15:13:26] <_joe_> is this just that server or all of maps? [15:41:56] imposm runs only there I think [15:42:04] not sure though, double checking [15:42:50] yeah, just maps2009 and maps1009 [15:43:09] let me upload my quick change [15:59:02] hmm can I just use Hosts: maps* in the commit message? [15:59:14] or do I need to figure out the role/profile? [15:59:21] I think that will work akosiaris [15:59:25] easy enough to try :) [15:59:57] well jenkins bot didn't like me to begin with. Now... on to the joys of CI [16:02:45] <_joe_> akosiaris: lgtm, but are we ok with restarting the importer? [16:02:57] <_joe_> because I think that will also happen [16:03:05] well, it's either that or a full disk again [16:04:34] hmm maybe I should change to Hosts: auto [16:04:37] let's see how that will work [16:05:04] cdanis: didn't work [16:05:23] <_joe_> akosiaris: what are you trying to do? [16:05:44] utilize the new stuff I read about [16:05:49] Hosts: auto in PCC [16:05:55] and that one worked [16:05:59] kudos to j.bond [16:07:25] <_joe_> no I meant what did you expect the result to be [16:07:43] of Hosts: maps* ? [16:07:48] to select all maps hosts [16:08:04] <_joe_> that you could just launch the pcc command from the commadn line without specifying the hosts? [16:08:17] <_joe_> or that CI would run the puppet compiler automatically? [16:08:30] the former [16:08:35] the latter would be a bad idea right now [16:09:17] <_joe_> yeah, that's why we never made it automatic :) [16:10:10] in other news, it appears I am not able right now to write 10 lines of code without 3-4 typos [16:13:55] <_joe_> and I didn't see them [16:14:20] <_joe_> but tbh I looked at the general idea rather than details :) [16:21:35] <_joe_> but also you made some great typos heh. file_glib :D [16:23:51] lol [16:24:03] imposm required a SIGKILL by puppet btw to stop [16:24:09] interesting that it ignores sigterm [16:24:23] em by systemd, trigerred by puppet [16:24:31] but yeah, interesting [16:25:58] Dec 22 16:25:08 maps1009 imposm[15961]: [2022-12-22T16:25:08Z] 0:01:57 [info] Importing #89845 including changes till 2022-12-12 20:00:00 +0000 UTC (236h25m8s behind) [16:26:31] ok it is a little bit behind, but that was probably true already [16:27:42] previous logs show a variety of timeframes for stuff, including 217h12m0s so it's in the same ballpark [16:28:15] ok it works fine, I 'll let the team know [16:28:21] but otherwise, crisis averted [19:03:14] I have changed some hosts's notifications enablement, I will check in a bit that all that should be disabled are disabled and all that should have it enable are enabled, once puppet runs on client and icinga hosts