[02:31:25] gehel: inflatador: ack, we’ll write up the report [10:23:50] btullis: re: https://phabricator.wikimedia.org/T300246#7809816 one sketch / pointer of a solution would be to read data from etcd and write it (on prometheus hosts) for prometheus::mini_textfile_exporter to pick up in a suitable format, happy to discuss more [10:28:10] godog: Thanks, I was going to check this with you. It looks like the prometheus (ops) hosts don't already have confd installed, but if I add a 'confd::file' resource then the confd package will be added as a requirement. [10:28:56] This is OK is it? Better than adding support for prometheus::mini_textfile_exporter to a puppetmaster that already has confd installed for the config-master templates? [10:29:35] btullis: IMHO yes that former would be better [10:31:13] Excellent, many thanks. Will proceed along these lines. This is the first time I've looked down the wrong end of a go template for confd, but it's sort of making sense. :-) Will hopefully get a patch out soon. [10:34:49] for sure ! [13:43:49] * inflatador looks at the BG outage info [14:00:50] moritzm: https://gerrit.wikimedia.org/r/c/operations/puppet/+/774481 look okay? [14:06:11] ottomata: o/ IIRC if you want to pull from upstream you also need to add the components under "Update:" [14:12:14] ottomata: in a meeting, will check when over [14:16:35] elukey: oh ty looking [14:17:14] elukey: where? [14:19:34] in distributions-wikimedia under "Update:" [14:27:59] Ah! [14:29:58] ty okay done [14:33:27] dcaro: OK to merge "openstack.cinder: Add patch to the backups chunkeddriver (ff9f8c216b)" ? [14:33:39] Emperor: yes please [14:34:08] done :) [14:34:12] thanks! [14:34:34] <_joe_> Emperor: replying to your question in #-operations earlier [14:35:02] <_joe_> changeprop is a simple piece of software that listens to the kafka topics where mediawiki signals a resource changed or needs to be purged [14:35:25] <_joe_> and makes, based on configuration, calls to various services to ensure they refresh their internal caches [14:36:04] <_joe_> now if we send out 4k concurrent requests from changeprop to the api servers for an expensive job, like rerendering of a page (even if the parsed html is already in cache) [14:36:14] <_joe_> that creates a congestion [14:36:17] ah yes [14:36:24] <_joe_> specifically, this was caused by "transcludes" [14:36:41] <_joe_> so one change to a template -> re-render of any page that uses that template [14:37:04] <_joe_> that can translate to many pages when the template is very popular [14:37:10] <_joe_> like say {{alert}} [14:37:12] Mmm [14:38:00] <_joe_> of course we batch such transclusions, we don't send them all out at the same time. For the jobqueue, which is the mechanism mediawiki uses, that's pretty well oiled [14:38:51] <_joe_> some of this is covered at high-level in the first few onboarding chats btw [14:39:00] <_joe_> if you want to go back and re-watch pieces of those [14:39:16] <_joe_> maybe my ramblings make more sense nowadays :) [14:46:30] I think it had just fallen out of my brain in the mean time :-/ [15:40:51] moritzm: oh, i just noticed i named my update 'conda', but some others are named like 'thirdparty/conda' [15:40:57] which is preferred? [15:41:07] should the name match the Component? [15:42:15] hm seeing as the wikitech reprepro example uses 'thirdparty/', i'll change mine to that now, let me know if that is not the preferred way [16:15:41] moritzm if e.g. https://repo.anaconda.com/pkgs/misc/debrepo/conda/dists/conda/InRelease 404s [16:15:55] does that mean we can't use reprepo updates for this? [16:16:06] or, maybe i have the urls wrong, but that seems correct. [16:16:12] e.g. this is the url for hte .deb [16:16:13] https://repo.anaconda.com/pkgs/misc/debrepo/conda/pool/main/c/conda/conda_4.11.0-0_amd64.deb [16:56:19] elukey: mmayyyybe any clues ^? :) [17:44:20] okay the Suite was wrong, it had to be 'stable' [17:44:24] got other errors now... [18:44:44] got it! trial and error :) [19:05:00] https://apt.wikimedia.org/wikimedia/pool/thirdparty/conda/c/conda/ [19:05:06] buuut not able to find package on host