[00:07:44] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10SRE: Check out www-portals repo in the mediawiki-webserver and in the mediawiki-multiversion images - https://phabricator.wikimedia.org/T285325 (10jeena) a:03jeena [02:13:59] 10serviceops, 10Performance-Team, 10SRE, 10MW-1.36-notes, and 3 others: Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe - https://phabricator.wikimedia.org/T264604 (10aaron) How many appserver instances are active? [06:36:27] <_joe_> nikkinikk_: doing so this morning [06:44:28] <_joe_> nikkinikk_: we should find a 30' slot to talk about this, possibly tomorrow at the cpt/serviceops syncup [06:50:01] I think that meeting is not goign to happen because we have a mandatory "Training Leadership and Team Development" conflicting [06:50:04] unless that gets moved [08:26:41] 10serviceops, 10SRE, 10conftool, 10Datacenter-Switchover: Disable maintenance scripts via conftool - https://phabricator.wikimedia.org/T266717 (10Legoktm) >>! In T266717#6723029, @RLazarus wrote: >>>! In T266717#6722705, @Joe wrote: >> I think we have a better way to avoid this. Basically we want to stop r... [09:03:47] _joe_: could you comment on https://phabricator.wikimedia.org/T285104 what you mentioned during Monday's meeting about potentially not needing a separate shellbox LVS, etc.? [09:05:23] <_joe_> yes, I'll do it during the day [09:05:44] <_joe_> legoktm: re: mw-cli-wrapper, I think we might consider rewriting it in python [09:06:03] <_joe_> or at least write a small python script that parses the yaml and gives green/red light :P [09:06:18] <_joe_> but for now the horrific chain of greps I proposed should do the job [09:07:00] My love for rewriting bash scripts into Python is only second to my love for rewriting Python scripts into Rust [09:10:24] I'll take a stab at it later today (my tomorrow) [09:16:29] <_joe_> if you try to rewrite our scripts into rust, I'll take a stab at you :P [09:16:55] <_joe_> why on earth would one want scripts to be compiled. [09:17:51] <_joe_> but yeah I know the feeling, I've almost started a rewrite of docker-pkg in golang 3 times. Then I went back to read Joel's article about rewriting from scratch [09:28:24] <_joe_> T284628 strikes again [09:29:34] <_joe_> (ノಠ益ಠ)ノ彡┻━┻ [10:03:23] I have gotten better at "write it in python first instead of bash -> python". but not perfect... [10:44:38] <_joe_> so, pulling the mediawiki image to the staging VMs takes 7 minutes on average, and most of that is spent extracting the image [10:45:17] <_joe_> jayme: I think we'll end up prewarming the image whenever we want to do a release [10:45:33] <_joe_> which means, first we pull it everywhere, then we run helmfile apply [10:49:23] _joe_: yeah. Extraction times are quite a problem. I've collected values for that during the "ever given"-tests as well if you want to weep a bit more about it [10:49:42] <_joe_> no thanks! [10:49:44] and ofc neither of d7y or k4n will fix that for us :( [10:51:43] _joe_: but there is this image-service thing from d7y (nydus) [10:51:52] not sure how that affects extraction times [10:52:49] <_joe_> n3s you mean? yeah not sure either [10:53:36] yeah, sorry :) [10:54:40] j1e and jee: better grab those nicks now if they're not already gone :-P [10:56:36] 10serviceops, 10MW-on-K8s, 10SRE: The mediawiki-webserver image should only log in json format - https://phabricator.wikimedia.org/T285384 (10Joe) [10:56:37] hrhr [10:56:46] 10serviceops, 10MW-on-K8s, 10SRE: The mediawiki-webserver image should only log in json format - https://phabricator.wikimedia.org/T285384 (10Joe) p:05Triage→03Medium [10:58:28] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Install wiki-specific php extensions in the mediawiki production image - https://phabricator.wikimedia.org/T285309 (10Joe) 05Open→03Resolved [10:58:34] 10serviceops, 10MW-on-K8s, 10SRE: Make all httpbb tests pass on the mwdebug deployment. - https://phabricator.wikimedia.org/T285298 (10Joe) [11:00:49] 10serviceops, 10MW-on-K8s, 10SRE: Make all httpbb tests pass on the mwdebug deployment. - https://phabricator.wikimedia.org/T285298 (10Joe) [11:21:22] 10serviceops, 10MW-on-K8s, 10SRE, 10User-jijiki: The mediawiki-webserver image should only log in json format - https://phabricator.wikimedia.org/T285384 (10jijiki) a:03jijiki [11:21:44] gah as many times as I read for typos. that was supposed to be j3e and so much for the joke! [11:32:24] <_joe_> it's j1e [11:32:40] <_joe_> and j3e [11:32:47] <_joe_> totally not confusing [11:32:53] <_joe_> we'll go with that [12:42:35] j.oe: do you have any pointers for https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/685462/2/images/python-build/bullseye/Dockerfile.template#1 ? I failed to find where [13:05:51] <_joe_> volans: sure, gimme 1 minute [13:05:57] <_joe_> that's something I need to do anyways [13:06:11] ack, thx [13:08:00] <_joe_> volans: turns out I already did it :P [13:08:07] lol [13:08:11] <_joe_> I76cc9cfe43ee90e7 [13:08:23] <_joe_> and known_images is the name inside docker-pkg's code [13:08:31] <_joe_> the conf variable is base_images [13:08:55] <_joe_> which makes sense as nomenclature in that context, but I was reading docker-pkg's code when I made that CR :P [13:09:34] <_joe_> so you don't need to do anything [13:09:41] ahahaha ok makes sense [13:10:04] ok so remind, me if I merge that then I need to actually build the image and upload it to the docker registry [13:10:23] or is that done in some semi-automated way from the build host? [13:11:27] <_joe_> ssh to deneb, git pull in /srv/images/production-images, run build-production-images [13:11:38] <_joe_> (both need root) [13:11:55] I am running it now btw :) [13:11:55] <_joe_> one day (TM) we'll do all this via CI [13:12:07] ack thx [13:12:13] elukey: ack, lmk when you've finished [13:12:39] volans: just completed, deneb senses when you need compute power [13:12:52] ahahah [13:13:24] <_joe_> tbh, we could do it today by making a docker builder VM and setting up a jenkins job for it [13:14:20] <_joe_> I was aiming for a more elegant solution, which is in part covered by the set of patches I created during the hackathon https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-pkg/+/699716/ and followups [13:20:36] I can check the patches in these days if you want [13:23:50] 10serviceops, 10SRE, 10Services, 10Wikibase-Quality-Constraints, and 3 others: Deploy Shellbox instance (shellbox-constraints) for Wikidata constraint regexes - https://phabricator.wikimedia.org/T285104 (10Ladsgroup) [13:25:40] interesting thing that I learned today about istioctl [13:26:12] I was confused about how to add configs for the ingress gateway, namely https + reverse proxy http to the backend [13:26:59] and after digging a bit, I discovered that istioctl takes care of the basic set up, then there are some Gateway + VirtualHost CRDs that can be used to configure the proxy [13:27:06] and in my case, those are in knative's config [13:27:37] so in the future, if istio will be used, we could in theory bootstrap it via istioctl (and also use it to inspect things on the fly, like routes/etc..) [13:27:45] and the helm to deploy specific configs [13:28:28] I'll try to make the knative-serving helm chart a little bit more configurable [13:28:35] (there is a net-istio yaml config) [13:43:55] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10wkandek) [13:45:28] 10serviceops, 10observability, 10GitLab (Initialization): Define monitoring for gitlab - https://phabricator.wikimedia.org/T275170 (10wkandek) [13:46:30] 10serviceops, 10Wikimedia-Logstash, 10observability, 10GitLab (Initialization), 10User-brennen: Logging for GitLab - https://phabricator.wikimedia.org/T274462 (10wkandek) [14:20:14] apergos: would you be able to add me to the meeting, just so I'm on it? _joe_: thanks appreciate it !! if the mtg does not happen I'm happy to setup a slot outside of the meeting to talk. [14:23:19] I am trying to figure out if I can [14:23:29] otherwise just steal if off my calendar from tomorrow [14:25:13] ah you can't do it from the pop up, you have to go into the event every time. meh [14:25:25] anyways nikkinikk_ you shouuuld be on it now, please check. [14:26:33] apergos: i dont see it 😢 [14:26:38] uh [14:26:42] hm [14:28:32] aaaand team meet in 2 minutes anyways... meh [14:40:17] <_joe_> I think urandom is the organizer btw [16:00:09] 10serviceops, 10Maps, 10SRE-swift-storage, 10User-jijiki: Swift account to store pre-rendered vector-tiles - https://phabricator.wikimedia.org/T283049 (10jijiki) 05Open→03Resolved p:05Triage→03Medium [16:00:13] 10serviceops, 10Maps, 10Product-Infrastructure-Team-Backlog, 10SRE, and 2 others: New Service Request tegola-vector-tiles - https://phabricator.wikimedia.org/T274390 (10jijiki) [16:01:01] 10serviceops, 10Maps, 10Patch-For-Review, 10User-jijiki: Deploy tegola-vector-tiles to kubernetes - https://phabricator.wikimedia.org/T283159 (10jijiki) [17:49:26] 10serviceops, 10SRE, 10conftool, 10Datacenter-Switchover, 10Patch-For-Review: Disable maintenance scripts via conftool - https://phabricator.wikimedia.org/T266717 (10RLazarus) >>! In T266717#7171743, @Legoktm wrote: > I guess the concern is that starting the scripts right away just adds extra pressure an... [17:56:10] sometimes reading the logs in here is a puzzle to decrypt all the numeronyms [17:57:15] I do think _joe_'s correct nick is actually `_3_` [18:00:01] well but he also has the more prestigious 'joe' now [18:00:12] and we tend to treat _ nicks as derived ones :-P [18:00:14] yeah, and by extension that one is just `3` [18:00:22] very tidy [18:00:23] lol [18:00:41] if he sets 3 as a stalkword, welp... [18:01:27] <_3_> just grouped! [18:01:39] lololol [18:01:54] what have I done [18:01:57] great, now I can't nick over to _rzl_ when I get grumpy :( [18:02:13] <_3_> ahahah [18:02:58] ok go ahead, get j1e, you know you want it [18:04:02] it does make you wonder if the "two week two hour" announcement is going to result in MORE registered nicks instead of fewer [18:04:19] as everybody goes "oh, right, I should be squatting! just a sec" [18:07:31] I got my standard two nicks, it's plenty [18:08:09] just because I can troll someone else into being a squatter doesn't make me responsible :-P [19:31:08] rzl: a bit of both ofc [19:31:52] I think they created that rule for cases of historic nicks being speed-squatted in the first few hours [19:32:52] yeah [19:56:59] am I doing this right? [19:58:17] or maybe I should have grabbed `B17e` for `BryanDavisIsAwesome`? [19:58:47] ooh I wouldn't, that's also `British Intelligence` [20:17:29] wow [20:18:19] I'm so sorry I started that [20:18:51] this is what I will be remembered for, long after leaving the org. not dumps, not rablle rousing, but the fact that I suggested these damn nicks [20:42:39] a5s will always be r8d [20:49:11] grrrrrr [22:45:13] apergos: <3 [22:58:06] 10serviceops, 10SRE, 10conftool, 10Datacenter-Switchover, 10Patch-For-Review: Disable maintenance scripts via conftool - https://phabricator.wikimedia.org/T266717 (10Legoktm) Ack, and we already have a separate step to re-enable them. The masking I suggested earlier doesn't seem to work exactly yet, see... [23:52:40] 10serviceops, 10SRE, 10Parsoid (Tracking): Maybe consider consolidating parsoid-* and restbase-* proxy services, respectively - https://phabricator.wikimedia.org/T285445 (10Arlolra)