[08:34:02] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review, 10User-jijiki: Deploy kube-state-metrics - https://phabricator.wikimedia.org/T264625 (10fgiunchedi) >>! In T264625#9349971, @kamila wrote: >>>! In T264625#9348475, @JMeybohm wrote: >> KSM in staging-eqiad was in a half installed state (pr... [09:09:23] 10serviceops, 10Prod-Kubernetes, 10SRE, 10ops-codfw: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10Clement_Goubert) Everything looks good, back in the cluster it goes. ` 09:04 <+logmsgbot> !log cgoubert@cumin1001 START - Cookbook sre.hosts.remove-downtime for kubernete... [09:09:29] 10serviceops, 10Prod-Kubernetes, 10SRE, 10ops-codfw: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10Clement_Goubert) 05Open→03Resolved [09:21:47] hello folks [09:21:57] https://gerrit.wikimedia.org/r/c/operations/puppet/+/976660 is the next and last change to move Kafka main to puppet 7 [09:24:37] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10Gehel) p:05Triage→03High [09:30:26] elukey: excellent :-) [09:34:26] ok starting :) [09:42:28] aaand done! [09:44:52] \o/ [10:19:40] what is the preferred way to prompt ServiceOps for reviewing some pending changes on Gerrit? :) [10:23:55] <_joe_> volans: in your case, send me a good bottle of wine [10:24:06] <_joe_> volans: it's about the locking changes? [10:24:22] <_joe_> I'll take a look today, i hoped someone else would [10:25:54] What locking change? [10:29:09] claime: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/967165 and https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/967166 [10:29:19] and more that will come once I've time to do the others :D [10:39:34] Opinion given [10:40:00] I'd suggest tagging more than two persons in the team if you want the whole team's opinion though :D [10:44:17] I'd like to tag serviceops and let you freely decide who should do it based on workload/vacation/etc... [10:44:38] and at the 3rd person gerrit starts to scare me with the "do not bother too many people" banner :D [10:47:33] and thanks for the review ofc! [10:49:01] volans: yeah, the lack of team tagging in gerrit kinda sucks [10:52:09] <_joe_> gitlab will solve that [10:52:20] ™ [11:13:20] 10serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team, 10Patch-For-Review: mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690 (10DAlangi_WMF) (update): The config change got merged and deployed yesterday. So it's live now. Is there anything else needed from MediaWiki Platform Team s... [12:01:58] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Move 25% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T348122 (10Clement_Goubert) [12:32:48] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review, 10User-jijiki: Deploy kube-state-metrics - https://phabricator.wikimedia.org/T264625 (10JMeybohm) >>! In T264625#9349971, @kamila wrote: >>>! In T264625#9348475, @JMeybohm wrote: >> KSM in staging-eqiad was in a half installed state (prob... [12:36:41] there are a few ways to do pin teams though probably none of them are ideal [12:37:49] but you can add as reviewer a whole group and all members of the group end up added as reviewer [12:38:09] I once created a `ops-europe-based` group for that purpose https://gerrit.wikimedia.org/r/admin/groups/04722775e2c89043c1e53eb14892f509e0d2e9f6,members [12:38:28] so I could easily spam SRE from Europe to get a review :D [12:39:08] rotfl, and are those groups in any way updated automatically? :D [12:39:19] * claime leaves europe [12:39:30] Seems like the easiest solution [12:39:32] judging from that list is probably from ~8y ago :D [12:39:55] yeah from 2013 - 2015 [12:40:23] then I am not sure it was good idea to bother every single of those persons for a review that certainly concerned only one or two so I probably haven't used more than a handful of time [12:40:31] Gerrit also has support for LDAP group lookup [12:41:21] so tentatively members of a team could be regrouped in a LDAP group but I don't know whether there is support to add a LDAP group as a reviewer [12:41:44] and anyone, the group is expanded and every single contributor ends up added as reviewer and being spammed with attention set until they remove themselves [12:41:49] so I don't think it is practical [12:43:01] there is a plugin to support code owners (aka require review by X and Y persons for a given path/file) which prevents submitting the change until the owners approved it ( https://docs.google.com/presentation/d/1DupBnGr3apIx-jzxi9cHzSgkI-2c1ouGu1teQ4khSfc/edit#slide=id.gac190220fd_0_102 ) [12:43:21] but that is still acting on an individual level rather than flagging a team/group [12:44:20] * claime ticks hashar in the nerdsniped list [12:44:29] <3 [12:44:48] [12:44:59] but there is an alternative [12:45:27] which has at least a couple shortcomings, but is interesting none theless [12:46:02] if you head to https://gerrit.wikimedia.org/ and are logged in, the browser url bar shows `/r/dashboard/self` [12:47:18] claime: actually with ldap groups you can tag a whole team with one click, in wmde we did that [12:48:41] we had team groups, it could work as "serviceops" [12:49:13] Amir1: Yeah, that's what hashar mentioned, but the "problem" is that it adds the whole team to the attention set, which can be desirable but also really annoying. A sort of "team dashboard" where you can go check pending CRs assigned to your team would be a little less spammy, but not sure that's possible [12:49:30] that dashboard is hardcoded in Gerrit, each section is given a title and an associated search query and I don't think one can change it at all [12:49:39] :( [12:49:48] I figured [12:49:55] BUT, any project can have dashboards with hand defined sections and search queries to potentially create a team based dashboard [12:51:21] they are exposed in the Web UI although that is a bit hidden, and they can not be edited via the Web UI to my knowledge (one has to send a git commit under /refs/meta/config which is deifnitely a poor UX) [12:51:52] Timo did create a few dashboards under the `wikimedia.git` repository and they can be seen at https://gerrit.wikimedia.org/r/admin/repos/wikimedia,dashboards [12:52:10] so tentatively serviceops could use https://gerrit.wikimedia.org/r/p/wikimedia/+/dashboard/teams:sre-serviceops [12:52:17] https://gerrit.wikimedia.org/r/p/wikimedia/+/dashboard/teams:sre-serviceops < lol [12:52:47] We apparently do have a team dash [12:52:54] that is editable by cloning https://gerrit.wikimedia.org/r/wikimedia.git [12:53:08] git fetch origin refs/meta/dashboards/teams [12:53:55] which gives a list of files, each of them being a dashboard [12:54:56] here is the file content https://phabricator.wikimedia.org/P53721 [12:55:38] the queries are exactly like the one can use in the top right search [12:55:59] and can be run by clicking on the section titles of https://gerrit.wikimedia.org/r/p/wikimedia/+/dashboard/teams:sre-serviceops [12:57:58] anyway [12:59:18] one can either bookmark in their browser that dashboard URL or add it as a menu entry in their Gerrit preference at https://gerrit.wikimedia.org/r/settings/#Menu by creating an entry with: [12:59:18] New Title: Team dashboard [12:59:18] New URL: /p/wikimedia/+/dashboard/teams:sre-serviceops [12:59:44] and that will add an entryin the top menu under YOUR --> Team dashboard [13:00:05] Thanks for all the info hashar <3 [13:00:11] in conclusion, it is doable [13:01:15] but the UX is a bit crap cause: [13:01:15] 1) edits have to be done via the rather obscure /refs/meta/dashboards [13:01:15] 2) the dashboards are not easily exposed in the Web UI, one has to browse the repo settings and find the Dashboard menu [13:02:07] for reference, the doc is https://gerrit.wikimedia.org/r/Documentation/user-dashboards.html [13:03:35] and one can create dashboard "oneliner" by passing the sections/queries directly in the URL then bookmark the result (or add it to their Menu entries in the user preferences) [13:03:39] example: https://gerrit.wikimedia.org/r/dashboard/?title=User+Dashboard&Latest+10+incoming=limit:10+is:open+reviewer:self+NOT+owner:self+-label:code-review+0:self&Waiting+on+others=owner:self+is:open+label:code-review=0+NOT+is:wip&Waiting+Approval=is:open+owner:self+label:code-review=1&WIP=owner:self+is:open+is:wip [13:03:50] (which is as scary as it looks) [13:04:19] the query uses `self` rather than my name, so anyone can click that and have a representation tailored for them [13:05:13] then nowadays I entirely rely on the attention set (lines shown in Yellow in the homepage) [17:44:57] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [17:45:57] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Move 25% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T348122 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert [23:54:23] There is a lua module on mw.o that can be used to make the URL style gerrit dashboards. https://www.mediawiki.org/wiki/User:BDavis_(WMF)/Projects/Core_code_review_dashboard has an example from the long distant past.