[12:53:04] Amir1, re the spicy one (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1239156), I'll defer that to Language team to land but I can have a look and try to test that within the week. [12:53:30] yeah, I'm mostly waiting on Abijeet :) [12:53:38] Ack! [12:53:44] Thanks for looking at it <3 [14:07:08] hi, is anyone with necessary access around who could deploy some changes for me? https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260223T1400 [14:08:06] yeah [14:08:21] you can run the scripts, right? [14:09:17] MatmaRex: ^ [14:09:39] nope [14:09:44] only on beta cluster. i should do that [14:10:33] ok, let me know when to start [14:12:20] tgr_: please go ahead. beta looks good [14:13:34] hmm. there are fewer log entries on https://meta.wikimedia.beta.wmcloud.org/wiki/Special:Log/gblrights/Maintenance_script than the script says it added [14:15:02] maybe some ended up on the wrong wiki? [14:19:22] MatmaRex: do you want to debug that or do you think it is fine to go ahead? [14:19:58] not sure. i would like the log entries to be correct [14:20:06] i'm just looking around the wikis [14:20:21] from what i can see, most accounts were already added to the relevant groups before [14:20:27] so mabe it's just the counting that's wrong [14:20:31] maybe* [14:23:57] GlobalGroupAssignmentService::saveChangesToUserGroups() has an $anyChanged flag but it's not really used for stdout logging [14:25:41] wait, i don't even see the global-temporary-account-viewer group on the beta cluster [14:28:02] it does show up on Special:GlobalUserRights [14:28:34] I guess Special:GlobalGroupPermissions should have a separate section for groups defined in the software [14:30:02] i created it now [14:30:12] just so that i can use Special:GlobalUsers [14:33:41] comparing https://en.wikipedia.beta.wmcloud.org/wiki/Special:ListUsers/suppress to https://en.wikipedia.beta.wmcloud.org/wiki/Special:GlobalUsers/global-temporary-account-viewer , it looks like all users who should be in the global group are already in it [14:33:51] (but there are some messed up accounts there… eh, beta) [14:36:04] https://en.wikipedia.beta.wmcloud.org/wiki/Special:GlobalUserRights/Aft5oversighter this guy is in the global-temporary-account-viewer group, but there is no log entry for the change? [14:36:49] ah, it w3as already logged at https://en.wikipedia.beta.wmcloud.org/wiki/Special:Log/gblrights [14:37:12] Special:GlobalUsers isn't working with automatic groups? that seems like a bug [14:38:04] anyway, the logging code is really straightforward and the counting logic in the script is very convoluted, I'd trust the former in case of discrepancies [14:38:07] wait, no it wasn't logged, i looked at the wrong user [14:38:27] i think some log entries were not inserted that should be. i don't know why, i feel like i should find out first [14:39:02] sorry, i should have done this last week [14:39:08] I have a meeting coming up so let's skip for now then [14:39:25] yeah [14:39:35] we can find an empty slot on the deploy calendar later if the evening window is too late for you [17:28:30] I bet there are folks in this channel who can give some pointers on [17:30:46] i am pretty sure i answered that question somewhere once upon a time [17:31:53] ah, it was in slack. ew [17:32:00] anyway, i can reply :D [17:33:16] thanks MatmaRex. I have some vague 2006 and Yuri idea, but I've never ran down all the parts of that. [18:26:46] tgr_: re the script from earlier today. my best guess is that the log entries were not inserted, because on this line: https://gerrit.wikimedia.org/g/mediawiki/extensions/CentralAuth/+/371c54698202c270ca473165246e86a83ff2c64b/includes/GlobalGroup/GlobalGroupAssignmentService.php#184 we are not guaranteed to read from primary. does this seem plausible? [18:46:55] i asked in #-releng too, but: we don't have database backups for the beta cluster, do we? [18:52:19] (the other channel says we don't) [18:55:17] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1242458 [18:55:40] if i could get a +2 on that, i'll test it on beta again ^ [19:25:25] MatmaRex: I assume we have a binlog for beta? just re-running for a different group is probably less hassle though [19:28:58] getGlobalGroupsWithExpiration() would not reload the CAUser object from the DB after the groups have been changed, I think, so it shouldn't matter much where the object is from [19:29:02] but worth a shot [19:29:38] hmm. no idea how i would access those [19:29:51] tgr_: addToGlobalGroup() calls invalidateCache(), so i think it does re-load [19:30:25] tgr_: while you're here, does this idea seems reasonable? https://phabricator.wikimedia.org/T416541#11642023 [19:31:41] true [19:32:11] so if we are adding ten groups, the user gets reloaded ten times in that request? ugh [19:32:22] I guess it doesn't really happen in practice [19:33:20] anything is fine on beta, I'd just create a new global group though [19:34:02] that needs a deployment, sort of, but we can do it outside the normal windows [19:34:10] yeah, i don't want to have to make another config change to do it [19:34:19] and then another to undo it [19:35:03] well, most of these cases the log entry is already missing so a direct DB delete won't really make things more confusing [19:35:39] you might need to invalidate some caches though [19:36:36] I wonder if just going into shell.php, messing with the config and the running the script from there (in general or to delete the memberships) is easier [19:41:32] now that it does getPrimaryInstanceByName(), it shouldn't use the outdated cache [19:42:25] i'll try the DELETE, see if it works [19:42:30] true [19:45:27] okay, i think it worked [19:47:46] https://phabricator.wikimedia.org/T416541#11642246 [19:56:00] tgr_: would you be able to deploy that backport and the rest, or should i schedule it later? [20:04:41] (i'm away for a bit) [20:19:23] MatmaRex: can do [20:19:34] the deploy calendar is empty until the end of the hour [20:26:34] thanks [20:37:32] TTFB for logged-in users improved by 30% after fixing Module:Date on enwiki last week. [20:38:08] https://grafana.wikimedia.org/d/d8bab23f-f495-480f-b23a-9d45851d5426/navigation-timing-breakdown-logged-in?orgId=1&from=now-14d&to=now&timezone=utc&var-metric=navigationtiming_responsestart&var-quantile=75&var-skin=vector-2022&var-group=group2&var-geo_continent=$__all&var-geo_country=$__all&var-ua_family=$__all&editPanel=12 [20:38:23] https://grafana.wikimedia.org/d/d8bab23f-f495-480f-b23a-9d45851d5426/navigation-timing-breakdown-logged-in?orgId=1&from=now-14d&to=now&timezone=utc&var-metric=navigationtiming_responsestart&var-quantile=75&var-skin=vector-2022&var-group=group2&var-geo_continent=$__all&var-geo_country=$__all&var-ua_family=$__all&editPanel=12 [20:39:15] I was looking at it through the lense of T414161, but the timing didn't match up (that fix didn't hit group2 until the next day) [20:39:16] T414161: Fix missed preload query from ResourceLoader WikiModule on logged-in pageviews - https://phabricator.wikimedia.org/T414161 [20:42:43] I've put the sample rate on the same dash right above it, so take that as a disclaimer, it's based on 2-3 samples per minute. [20:43:00] but it takes 6h into account (thx to prometheus, we couldn't have done that in graphite) [20:43:09] so not too bad [20:50:47] 🎉 🥳 [20:50:53] that's a very nice result [20:51:00] congrats! [20:56:33] :o [21:07:02] you should ping the ops@ thread with that result for visibility [21:07:18] it might motivate SRE to look at these metrics more often