[06:52:15] Amir1: Just created this https://phabricator.wikimedia.org/T308206 not sure if we should add other tags [06:52:44] marostegui: no that's good [06:53:23] good [06:54:01] possible_keys: NULL. Sigh [06:54:44] marostegui: FYI T307902#7919405 [06:54:45] T307902: Assess database requirements for link recommendations reading entry point - https://phabricator.wikimedia.org/T307902 [06:55:07] specially T308084 [06:55:08] T308084: Reduce DB space used by Echo notifications - https://phabricator.wikimedia.org/T308084 [06:55:43] 100GB is echo notifications of wikidatawiki, how much do you want to bet 99% of is bots because bots would read their notifications :P [06:56:13] hahahaha [06:56:23] same with watchlists I guess [06:56:32] we cleaned that last year [06:56:41] yeah I know [06:56:47] not that 99% of it was just bots, 90% of it was just one bot [06:57:22] what about that cebwiki bot eh ha [06:57:36] imagine its watchlist [06:57:54] I strangled that watchlist with my bare hands too [06:58:03] hahaha [07:01:24] marostegui: probably the table has issues, not the code https://phabricator.wikimedia.org/T308206#7923280 :( [07:02:32] I can take a look later [07:02:51] Awesome [07:02:53] Thanks [07:20:37] Amir1: https://phabricator.wikimedia.org/T308206#7923355 [07:21:27] wait why there is a echo table on each wiki [07:21:35] you tell me! [07:21:39] I thought they are central in x1 [07:23:28] Amir1: https://phabricator.wikimedia.org/T119154 [07:23:45] We have those wikis with their own tables [07:23:49] The task is very risky though [07:24:12] sigh [07:24:40] Amir1: the issue is that those tables do not have the index, but the x1 ones do [07:24:54] So we need to add that index to those specific wikis [07:24:59] should be easy to do [07:25:15] yeah I wonder what other drifts they have [07:25:22] shall we run it with replication? [07:25:34] probably not, but I can take care of that [07:28:10] Thanks [07:28:46] Amir1: can you double check if we need to send some patch or something for that schema change? I don't think so, but just in case [07:29:09] sure [07:30:26] it's already there https://github.com/wikimedia/mediawiki-extensions-Echo/blob/839b833033a5cc4286ca0618b12188b670b87562/echo.sql#L28 [07:30:31] excellent [07:30:32] hi, for unrelated reasons I noticed /etc/wmfbackups doesn't seem to be managed by puppet, is it created by sth else like a debian package ? [07:30:37] but double check for other drifts [07:32:47] marostegui: the weird thing is that the table already exists in x1 (and probably have the same data) but not being read :( [07:33:03] Isn't empty? [07:33:08] I would think it is empty if it is not used [07:33:51] maybe actually [07:34:18] let me double check I saw rows five at the explain so it must have something [07:35:44] nah you're right, it's empty there [07:35:56] the five is the key len 🤦 [07:38:02] hehe [07:40:50] Amir1: all fixed, closed the task [07:40:58] Awesome. Thanks! [07:41:22] ooh that would explain the 0 cardinality [07:41:36] anyway [07:50:47] recentchanges queries are terrible :( [07:53:59] in which wiki? [07:54:11] From what I am seeing now, commons [08:01:30] Improvement to the RC table would be nice. When we did our last move at Miraheze, not replicating and just regenerating RC was impossible because it took so long. [08:01:52] I know Amir was working on links which was another mess. [08:05:03] marostegui: T307328 for now but I can try taking a look at commons [08:05:04] T307328: Scalability issues of recentchanges table - https://phabricator.wikimedia.org/T307328 [08:05:49] Amir1: Yeah I am aware of that task <3 [11:07:39] marostegui: so there is a slow query caused by templatelinks migration in dewiki, I have a patch ready and hope to get it reviewed and merged soon but if you see too much pressure on s5, let me know so I revert it (T308207) [11:07:40] T308207: ApiQueryInfo::getProtectionInfo is slow on normalized templatelinks - https://phabricator.wikimedia.org/T308207 [11:08:07] revert what? XD [11:08:25] but yeah, dewiki is now the top slowest wiki [11:08:27] the read new on dewiki so it would pick the old data [11:08:36] If we can push the patch that's nice [11:08:57] let me see if James is awake :D [11:30:42] I have a few hours to compensate work- I will be off either today or tomorrow's afternoon depending on if I see upgrade issues [11:31:33] jynus: hi, before you go I have a quick question if you can, I noticed /etc/wmfbackups isn't created/managed by puppet, does a debian package or sth else create it ? [11:31:49] godog: yes, I chose to move it to the package [11:32:19] jynus: ack, thank you ! [11:32:22] that explains [11:32:31] godog: https://github.com/wikimedia/operations-software-wmfbackups/blob/master/debian/wmfbackups-check.dirs [11:32:37] that is, for example, for icinga [11:33:18] yeah exactly, I noticed it on the pontoon icinga hosts that puppet was complaining about /etc/wmfbackups [11:36:05] ok got it, wmfbackups-check 0.5 in buster doesn't look like it is shipping /etc/wmfbackups [11:36:10] if you are doing it to test a potential upgrade [11:36:38] bullseye has the new version [11:36:56] I manually backported the bullseye package to the icinga hosts [11:37:14] I can create one and upload it for buster if it helps [11:37:32] ah ok, thank you that explains why I couldn't find 0.8+deb11u1 on apt1001 [11:37:51] not a problem now but potentially breaks reimages on buster for alert hosts [11:38:14] yeah, I was betting on the next reimage being to bullseye [11:38:32] safe bet for now I'd say, that's likely what will happen [11:38:52] I can create 0.8+deb10u1 but not now, unless it is urgent [11:39:34] no not urgent at all, thanks anyways though [11:39:38] basically, in a way "I stopped maintaining buster checks" [12:17:31] just saw the merge Amir1 <3 [12:17:52] deploying now ^^ sorry for the mess [12:32:12] marostegui: fixed https://logstash.wikimedia.org/goto/6ad93f5b7900dda9e489bb6195e8fd69 [12:32:26] yaaaaaaaay! [12:32:28] thanks [13:47:07] marostegui: btw, the terrible rc queries of commons, where can I find them? slow queries dashboard? [13:47:18] yeah [13:47:42] let me see if I can send you the link [13:48:09] https://logstash.wikimedia.org/goto/7c6a8e55a447598d352f40c6036b0b99 [13:51:42] marostegui: the ones on db1141 seems to be an issue with that db only :/ [13:52:08] e.g. https://phabricator.wikimedia.org/P27810 [13:52:18] this is fast on dbstore but slow in db1141 [13:52:40] picks the wrong index [13:57:18] give it a go on the stats to see if that fixes it [13:57:46] should I depool it first? [13:57:51] * Amir1 is scared [13:57:53] yeah, better [13:57:58] ok [14:09:48] marostegui: fixed it, going to repool it back now [14:09:54] awesome! [14:09:56] thanks [14:13:39] everything in mediawiki is a rabbit hole T307328#7924716 [14:13:40] T307328: Scalability issues of recentchanges table - https://phabricator.wikimedia.org/T307328 [16:01:17] Amir1: Tim mostly answered, but the main issue to reduce it would be watchlists- when rcs had issues, most complains were about needing watchlists beyond the 7 day limit [16:02:04] jynus: yeah but a month? Maybe two weeks would be enough [16:02:17] the issue is not that people page back [16:02:35] it is that the intersection may need >7 days on the second result already [16:03:06] not sure if I am getting understood [16:03:35] you mean to fill the 50 results? It can be below that no issues [16:03:38] although to be fair- that would be mostly a watchlist issue, not an rc [16:03:55] or we can have two tables, one week for special:rc and a simpler one for watchlist [16:04:03] yes [16:04:16] or keeping rcs and moving only watchlist to elastic [16:04:51] again, not saying what I meant, transmitting the issues we detected when wikidata rc was first deployed [16:05:16] yeah, wikidata rc integration needs love [16:05:33] in other words- rcs is not that bad and would +1 to reduce it, watchlist is the main limiting factor [16:05:35] was much worse but since 2017 got much better (with killing X aspect) [16:06:15] I see, I keep that in mind. Thanks. [16:06:20] as watchlists are very sparse queries [16:06:53] yeah, it's what Tim calls pathological intersectin [16:06:59] in a way I am repeating what he said [16:07:05] "If it was just RC then it could be shorter." [16:36:36] godog: I've given it a thought and I think the best way to move forward is to remove the profile::dbbackups::check profile from alerting_host and move it to a dedicated backupmon one [16:37:02] as I am close to deploy a backup dashboard that also accesses the same db [16:37:58] that way I can keep the bullseye stack