[07:09:54] I have restarted sirenbot because it's been down since 13th [07:48:50] hmmm it looks like fabfur is hitting T338961 on https://puppet-compiler.wmflabs.org/output/929989/41824/cp5013.eqsin.wmnet/prod.cp5013.eqsin.wmnet.err [07:48:51] T338961: Puppet compiler fails due to unset fact wmflib.is_container - https://phabricator.wikimedia.org/T338961 [07:53:33] hmmm maybe related to cp5013 being a insetup::traffic node? [07:54:10] trying again with different host [07:57:02] confirm success, no problem now (my fault) [08:11:56] hey folks forgot to mention, all vk instances in eqsin migrated yesterday to pki, all good [08:12:04] codfw next [08:33:29] dhinus: Is there still stretch databases in wmcs land? [08:36:15] marostegui: no, Cloud VPS is fully Stretch-free [08:36:41] cool [08:36:43] thanks [08:36:53] I am going to remove that then from puppet [08:43:33] marostegui: I also think that all three puppetized mariadb instances in wmcs (cloudinfra-db, ToolsDB and deployment-prep) are fully on bullseye, so in theory buster could be removed too unless you still have hosts running it [08:43:52] taavi: We don't, analytics do [08:43:58] So we unfortunately cannot remove it yet [14:36:52] hnowlan: re "ThumbnailRender job in changeprop-jobqueue (so requests will be coming from k8s)", I'm sorry to be pedantic, it's really just that I want to make sure I understand and can otherwise update my mental model and docs - afaik these requests woudl be coming to Thumbor from MW appservers in the jobrunner cluster, not k8s or changeprop. The job descriptions indeed fly through Kafka and changeprop JS makes POST requests to the [14:36:52] jobrunners to execute the jobs. If changeprop in k8s is making requests to upload.wikimedia.org or Thumbor directly, my mental model is incorrect. [14:42:19] Krinkle: you're correct, of course the changeprop hosts themselves wouldn't be making the requests. my bad [14:42:58] which makes things a bit simpler/less clear - the jobrunners aren't in k8s (right?) [14:43:24] indeed, not at the moment. [14:43:38] although jobrunner cluster did grow recently to have several extra hosts [14:44:53] context: ref T329366 [14:44:53] T329366: Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 [14:45:06] parse1013-parse1016 temporarily became jobrunners [14:45:12] ahh I see [14:46:38] several different dashboards of wqiad on Grafan just went all "no data" [14:46:44] eqiad* [14:49:11] and I'm getting data again [20:10:13] fun question: I need to introduce a cookbook and that cookbook gonna run mysql command with password in it (sets password of something). How can I do that? I don't want it to be user input [20:10:49] I can just hardcode the password in the cookbook and push it to gerrit 😈 [20:40:59] ansible-vault ;P [20:42:08] make it try all possible passwords until it finds the correct one [20:42:09] seriously though, could you pass it thru puppet eyaml? [20:43:08] * inflatador has no idea if we use puppet eyaml [20:44:00] Amir1: probably you want to update profile::spicerack::cookbooks::production to add a config file for the cookbook to read [20:44:24] > make it try all possible passwords until it finds the correct one [20:44:27] I love that [20:44:45] yeah, probably need to change spicerack to read a yaml file in cumin [20:45:28] or can it be part of config, just set via a hiera value in private repo [20:45:29] If a human has to run the cookbook, maybe user input is OK? [20:45:45] probably still a PITA though [21:06:35] yeah, for now I go with user input, later change it to something else [21:16:41] FWiW, the reimage cookbook requires user input...others just read a YAML file off cumin