[00:03:57] 06serviceops, 10Wikimedia-Apache-configuration: Change redirect target of sep11.wikipedia.org - https://phabricator.wikimedia.org/T367014#9971820 (10Dzahn) >>! In T367014#9971812, @Pppery wrote: > On second thought that's probably better than any pointer to the Wayback Machine as it explains the context. Agr... [07:40:09] 06serviceops, 10conftool: Move conftool to gitlab, turn on deb package auto-generation - https://phabricator.wikimedia.org/T369594#9972335 (10Joe) p:05Triage→03Medium [09:05:55] 06serviceops, 10conftool, 13Patch-For-Review: Move conftool to gitlab, turn on deb package auto-generation - https://phabricator.wikimedia.org/T369594#9972678 (10Joe) [09:07:42] 06serviceops, 10conftool, 13Patch-For-Review: Move conftool to gitlab, turn on deb package auto-generation - https://phabricator.wikimedia.org/T369594#9972693 (10Joe) Yesterday we merged https://gitlab.wikimedia.org/repos/sre/conftool/-/merge_requests/3 which triggered the job building the debian packages fo... [09:09:10] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 16), 13Patch-For-Review, 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9972698 (10mforns) @Scott_French thanks for all! Regarding T361835#9966404, th... [09:37:34] 06serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690#9972806 (10jijiki) [09:41:35] 06serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690#9972810 (10jijiki) **Current status: ** * all mw-* deployments use the mcrouter ds * mw-wikifunctions is using its container * mcrouter contair & ex... [12:08:17] _joe_: An imperfect search, but https://codesearch-beta.wmcloud.org/search/?q=%28api%7Cappservers%29%28%28-r%28w%7Co%29.discovery%7C.svc.%28eqiad%7Ccodfw%29%29%29.wmnet&files=&excludeFiles=&repos= suggests that at least Wikidata will need to do some work to migrate away from the old bare-metal discovery URLs. [12:09:43] <_joe_> wikidata? [12:09:51] <_joe_> or you mean the RDF code? [12:10:31] <_joe_> termbox has the correct config in production [12:10:34] Sorry, the wikibase-termbox prod config. [12:10:37] Ah, good. [12:10:43] <_joe_> it uses the service mesh [12:10:54] <_joe_> but yeah there's some stuff, the team is aware [12:10:57] The comments in the RDF code should be updated. [12:10:59] Awesome. [12:11:09] <_joe_> for instance our own mail server config :D [12:11:17] Ha. [12:11:32] I was ignoring operations/* on the basis that you're better-placed to notice that. [12:11:34] <_joe_> it's not so much stuff that it's a problem [12:12:13] * James_F nods. [12:40:31] hello, I totally forgot if in the end we're checking the etcd MW last index in k8s in the end or not. If not I'll add the required cleanup steps to the task. [12:41:32] <_joe_> no, we decided that the absence of any alerting by that for what, 7 years? was enough evidence we shouldn't spend time moving it over :) [12:45:53] ack [12:47:03] I took the liberty to edit T362323 and add it in the cleanup section [13:20:57] <_joe_> volans: thanks [13:24:46] yw :) [14:10:59] 06serviceops, 10Dumps-Generation, 10MW-on-K8s, 06Release-Engineering-Team: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#9973589 (10WDoranWMF) Thanks @Joe for the update and pushing this forward. The Dumps 2.0 work has already been picked... [14:34:25] _joe_: talking to dpe leader folks about https://phabricator.wikimedia.org/T352650 [14:34:26] qq: [14:34:35] how does the PHP upgrade bit work for your proposal [14:34:58] will the dumps mw image remain on php 7? [14:35:03] <_joe_> no [14:35:16] <_joe_> it would be on php 7 now ofc [14:35:18] <_joe_> but [14:35:31] <_joe_> mediawiki will start using code constructs that are php 8 only [14:35:43] <_joe_> and thus dumps will need to run on php 8 eventually [14:36:05] so someone would have to do work to upgrade dumps code to be php 8? [14:36:35] <_joe_> dumps code isn't mostly using mediawiki libraries? [14:36:40] i don't know, i guess! :) [14:36:42] <_joe_> and python glue as well? [14:37:03] so your proposal using mw container would help mostly to get rid of the mw puppet and scap stuff, ya? [14:37:08] <_joe_> yes [14:37:11] got it, thank you [14:37:29] <_joe_> and make the upgrade of the servers to bullseye simpler for you too [14:37:37] <_joe_> because those servers will need to be upgraded [14:37:44] <_joe_> possibly even to bookworm :) [14:41:08] right, nice. [15:09:07] I'd hope that the dumps code Just Works™ on PHP 8.x, as we've been testing MW in PHP 8.0+ for years, and there's some (imperfect, but some) unit testing of the dumps code. [15:09:37] I'd be more concerned about the load/persistence impact of moving from bare metal to k8s than the PHP version change. [15:09:53] But it has to happen eventually, of course. [15:34:31] <_joe_> we won't move to k8s [15:34:37] <_joe_> the plan is a temporary hack [15:34:48] <_joe_> we'll just push there the latest mw image [15:34:53] <_joe_> and use that to run the scripts [15:35:08] Ah, right. Yeah, that should be fine. [15:35:14] <_joe_> because i won't do an nfs mount in a k8s pod :) [15:35:17] (Famous last words.) [15:35:22] Fair. [15:35:44] it's either that or dumps 2.0 takes over all of dumps 1.0 or dumps 1.0 sticks with php7.4 [15:36:16] and the dumps2.0 plan doesn't appear to be ready before end of calendar year IIRC [15:36:56] And presumably switching dumps off until 2.0 is ready isn't acceptable. [15:36:57] <_joe_> I doubt we can turn off dumps 1.0 so quickly, realistically [15:37:17] <_joe_> well end of calendar year would work, if we can turn off dumps 1.0 by then [16:53:59] I'd like to roll out a config change, any objections? It enables an experimental special page on testwiki: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1053723 [17:22:25] _joe_: i really doubt we can turn off dumps 1.0 by end of calendar year. There are also wikidata dumps (i don't really know how this works) that IIUC have not yet been considered for dumps 2.0 [17:22:37] mayybe by end this new FY2025 [17:22:48] but I'm pessimistic and I wouldn't count on it [17:22:55] the reconcilliation process is gonna be hard and messy [17:23:13] I think your proposal makes sense. [17:25:13] btw, James_F _joe_ since I have you here and I'm not sure who else to ask. I'm considering a change to EventBus extension that removes the EVENT_TYPE and wgEnableEventBus support, in favor of being able to enable and disable per stream, instead of EventBus event 'type'. [17:25:13] https://phabricator.wikimedia.org/T346046#9903943 [17:25:13] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventBus/+/1050060 [17:25:14] Any thoughts/objections/folks I should ask? [17:26:32] So right now that's configured to TYPE_ALL by default but differently for private or odd wikis? [17:26:53] yes, and different depending on if JOB queue is enabled, etc. [17:26:55] I'd ask Security(?) whether these concerns are still at the wiki-level rather than stream-level. [17:27:07] Because you're inverting the control model. [17:27:18] they could still be controlled wiki level. just per stream [17:27:24] via EventStreamConfig [17:27:44] E.g. wikitech's disable links to T192361 which just says it can't happen whilst they're not in the main cluster. [17:29:17] the same streams would still be effectively disabled, its just that now there are two kind of conflicting ways to disable a stream: wgEnableEventBus, and EventStreamConfig [17:32:05] hmm. [17:32:20] It's probably fine, but a bit more complicated to reason about rather than a simple short list in prod config? [17:34:32] yeah...it might be. i'm imagining how the ESC config would look. [17:50:03] swfrench-wmf: was talking to jo.e earlier about doing a new conftool release, 3.0.2 [17:50:36] https://gitlab.wikimedia.org/repos/sre/conftool/-/jobs/308138 has it ready whenever you want to, or also happy to do it [17:50:54] it basically introduces the --reason option so not important in that sense but might be nice [18:00:09] sukhe: ah, neat! one of the benefits of moving to gitlab :) [18:00:09] so, I need to look a bit closer, as it looks like this only produced bullseye packages (we also need buster and bookworm for a full conftool release). [18:01:09] yeah, I'll follow up with _j.oe_ on what we want this to look like. naively, getting multi-distro builds requires multiple changelog updates, and I want to make sure we're on the same page as to what that process should look like. [18:01:50] oh indeed, I just assumed we are doing only bullseye [18:02:44] the build file does specify all three but not sure only why bullseye is being built [18:02:47] https://gitlab.wikimedia.org/repos/sre/conftool/-/blob/main/.gitlab-ci.yml?ref_type=heads [18:03:06] s/build file/gitlab CI file [18:09:28] sukhe: so, those `test-.*` jobs are just that: running the test suite under each of the per-distro test image variants (see .pipeline/blubber.yaml) [18:10:04] for the package builds, I believe it's really only taking the head of the debian/changelog file to determine which distro (and version ID) [18:10:04] yeah I see them in the pipeline https://gitlab.wikimedia.org/repos/sre/conftool/-/pipelines/63464 [18:10:24] the gitlab secret sauce for why only bullseye is being built is what I was trying to find :D [18:10:33] yeah, I guess [18:10:50] https://wikitech.wikimedia.org/wiki/Debian_packaging/Package_your_software_as_deb#A_typical_workflow <- there [18:11:12] > you need to perform separate push operations for each of the distributions [18:11:14] :( [18:14:48] there are some tricky bits here too, in terms of ensure the multi-distro builds don't have divergent views of the main branch of the repository (which can confuse reprepro in surprising ways) [18:20:45] in any case, I'll follow up with _j.oe_ about what we want this to look like :) [18:21:41] I think at the very least we will need three different branches here otherwise I just don't know how to do everything in one changelog and keep it clean [18:21:47] yeah, thanks! [18:22:07] (not sure if there is a way to just have separate branches and no overlapping changelog for the build?) [18:22:11] * sukhe goes back to other stuff :P [18:35:49] sukhe: will it help you if I told you that it used to be much much much worse? [18:36:17] I don't think I have my very obnoxious one-liner anymore, I think it died with an older build host [18:36:58] but it was a very complicated `gpb buildpackage` incantation I will never be able to reproduce and also it edited the version shown in the changelog [18:40:02] anyway, the strategy that _j.oe_ uses is to, on top of whatever version he wants to release, send three different merge requests to trigger CI, but then only merge one ;) [18:42:14] cdanis: curious to see the one-liner and just thankful that in Traffic we don't have to deal much with this otherwise I can't imagine doing this for some of the packages we build, gbp or gitlab. no easy answers. [18:46:39] sukhe: not a one-liner, but a now-hopefully-can-be-deleted explanation of the build process: https://wikitech.wikimedia.org/wiki/Conftool#Releasing_conftool [18:47:12] oh, wait! _j.oe_ already updated it, lol [18:47:57] hmmm ... I'm not sure those instructions are going to work, as it's not pushing to main [18:48:16] https://wikitech.wikimedia.org/w/index.php?title=Conftool&oldid=2197309#Releasing_conftool [18:49:47] also I answered my question about how this could ever work: the specific incantation of `dpkg-buildpackage` does not produce an orig.tar.gz. that avoids falling into the "incompatible per-distro-rebuilds" trap. [18:50:47] i don't know why we sometimes insist on things like orig.tar.gz for pieces of software we authored ourselves lol [18:51:12] heh, indeed :) [18:54:33] and now I see that the updated instructions are specifically framed around repeated MRs, so yeah that should work, albeit creating a noisy changelog (though that's inescapable given the way this works) [20:02:22] James_F: FYI I think it will be best to keep wgEnableEventBus. you are right, too confusing otherwise. [20:19:02] sukhe: I did something a little cheeky [20:19:13] https://i.imgur.com/vuLjZc2.png [20:19:19] and it looks like it kinda works maybe [21:50:02] cdanis: looks good though and not cheeky! and some of the stuff is similar to what we do on the buildhost with the pbuilder hooks [21:58:36] I think it's very arguably just the right thing to do for internal software [21:59:00] for now, there's a MR out to j.oe for the conftool case