[05:54:35] hello, I'm back! [07:06:35] godog: congrats for getting rid of smokeping! [08:24:01] XioNoX: cheers! feels good to turn down services too! [08:51:56] welcome back! [09:23:18] godog: BTW, dashboards using metrics defined on prometheus/rules_ops.yaml should use eqiad/global as a datasource or can benefit from thanos too? [09:24:19] vgutierrez: the latter -- can/should use thanos as well [09:24:24] ack [09:24:25] thanks [09:25:13] sure np! [10:02:20] :q [10:02:26] oops.. wrong window :_) [10:02:44] gladly my IRC client isn't a vim instance [10:30:07] _joe_: BTW, do you have a task to follow up the restbase inability to revalidate a cache hit? [10:31:13] <_joe_> yes, let me find it [10:35:47] <_joe_> https://phabricator.wikimedia.org/T321101#8352889 [10:36:07] <_joe_> vgutierrez: i doubt it will be fixed in restbase at this point [12:03:28] hi, can someone please puppet-merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/853265/ for me? It's adding a new job to the maintenance server [12:08:43] godog: ^^ [12:13:28] <_joe_> (the job is about pushing metrics, probably that context is needed) [12:16:32] yeah, it runs extensions/GrowthExperiments/maintenance/updateMetrics.php (which pushes some numbers to statsd) daily [12:59:57] ok [13:00:52] we also have the puppet deployment window for such requests [13:02:19] urbanecm: {{done}} [13:02:34] thank you kindly [13:16:03] is the topic here for on call outdated (e.g. the bot is not working on this channel maybe)? [13:16:48] or maybe sirenbot doesn't have permissions [13:27:51] I would like to note that we will have a review ritual later in the day, consider attending, specially if you want to learn more about to topic [13:35:12] jbond: can you do a quick check before merging? https://gerrit.wikimedia.org/r/c/operations/puppet/+/849473 [13:35:55] * jbond looking [13:43:03] dcaro: done [13:58:17] jynus: I think it's because the automod bot isn't here either [14:02:16] _joe_: got a question for you re https://integration.wikimedia.org/ci/job/debian-glue-backports/212/console [14:02:48] _joe_: we're trying to build purged for debian, integration-ci complained about a missing go.mod, I've added it but it seems like it's attempting to download those dependencies from the internet [14:11:49] moritzm: maybe you can point me into the right direction... according to https://go-team.pages.debian.net/packaging.html debian packages should be installed on /usr/share/gocode/src [14:12:09] but go mod expects a module cache in /usr/share/gocode/pkg [14:12:31] <_joe_> vgutierrez: sooo, we usually don't follow the (IMHO harmful) debian guidelines for building go packages [14:12:42] <_joe_> are you trying to actually upload purged to debian proper? [14:12:54] nope [14:13:01] vgutierrez: it looks like that the go.mod you added requires newer versions of packages than what bullseye has [14:13:03] recent versions of golang expect a go.mod [14:13:04] <_joe_> ok then just run "go mod vendor" [14:13:24] <_joe_> and save your vendored dependencies into the repo [14:13:28] so I should upload the vendor/ directory in the repo? [14:13:30] (and remove the golang-*-dev packages from build-depends if going the vendoring route) [14:13:42] <_joe_> and that, yes [14:13:56] <_joe_> vgutierrez: that's how we usually treat go packaging yes [14:14:07] not a problem [14:14:19] taavi: I guess that you're referring to golang.org/x/net package version [14:15:33] yep, at least that and the prometheus client [14:16:58] so should I use debian versions if I follow the vendor/ approach? [14:19:15] hi, would anyone know how Puppet/Facter set `$facts['os']['distro']` and `$facts['networking']...`. I somehow don't have them populated locally :-\ [14:21:40] hashar: there is a cache, that has missled me in the past (you can delete it safely) [14:21:48] hashar: the os stuff comes from /etc/os-release and /etc/debian_version. the networking information is mostly just parsing the output of `ip` [14:22:03] Those are supposed to be core facts, they don't appear in `puppet facts` output or `facter -p os` ? [14:22:22] (core as in they should not need to be populated iirc) [14:22:29] hashar: those are core facts, https://puppet.com/docs/puppet/5.5/core_facts.html#os [14:22:54] there is no custom code involved there [14:23:02] ahhh thanks for the link [14:24:31] vgutierrez: I don't see any reason why you would need to stick to versions that were in Debian at some point if you're not relying on pulling in the libraries via the debian repos [14:24:55] my setup must be broken in some odd way cause neither of those commands report anything for `os.distro` or `networking` :-] [14:25:06] taavi: me neither.. just asking :) [14:25:14] hashar: try running facter with `--debug` [14:25:14] hashar: But it does for other facts ? [14:25:53] yes i have plenty of facts but networking is entirely empty and os does not have that distro key [14:26:11] I was looking for the source code which generate them but could not find any ruby file matching [14:26:35] could it just be a perm issue? running it with sudo doesn't yield them either? [14:26:42] it is not really important, I wanted to revive an old script from 4/5 years ago to generate a catalog of a node [14:26:54] ah maybe yeah I don't run it with sudo [14:27:28] (I'm throwing stuff at the wall, I don't think it's supposed to be needed but you never know) [14:30:04] hashar: i suspect you are using facter 3 which is written in c so you wont find any ruby filrs for the core facts [14:30:20] I use `bundle exec facter --json -p os` which is 2.5.7 [14:31:17] well it is not important anyway :-] [14:31:21] ahh ok then yes that is ruby however i wond;t rely on that, id just apt-get install facter [14:31:22] sigh... 15:29:26 gcc: error: /build/purged-0.19~bpo10+1/vendor/github.com/confluentinc/confluent-kafka-go/kafka/librdkafka_vendor/librdkafka_glibc_linux.a: No such file or directory this is getting old [14:32:26] jbond: I think `bundle exec puppet` relies on the Ruby version of facter, that might be the issue then :-] [14:33:09] hashar: yes that correct and we could probably bump that to version 4 (also ruby) [14:33:35] what does our Puppet master uses? [14:33:41] however nothing in the repo really uses facter so other then running it manually as you have i dont think its relly needed [14:33:52] all our hosts are running facter 3 [14:34:34] fyi i can recreate the networking issue, that could be related to what we do in modules/base/lib/facter/interface_primary.rb [14:34:45] however bundle exec facter --json -p os works fine for me [14:34:56] fun [14:36:02] which version of facter do you have? `bundle exec facter --version` [14:47:58] hashar: im pretty sure everyone will have the same 2.5.7 version. in the bundle environment as that comes from the Gemfile dependency resolving [14:48:24] bu as said i have the c version manualy installed [14:48:25] apt-cache policy facter [14:46:54] [14:48:28] facter: [14:48:30] Installed: 3.14.12-1+b2 [14:48:33] Candidate: 3.14.12-1+b2 [14:48:52] i can explore moving the repo to facter 4, however i think there is a hard < 4 dependency in the puppet 5.5. gemspec [14:49:27] and using facter 3 needs a native extension so would make installing bunlde more difficult [14:49:53] I have the C version installed as well, but `bundle exec` would inject the ruby installed one in the PATH (at least on my machine) [14:49:54] and as said i dont think facter in the bundle environment is that usefull so not sure its worth exploring [14:50:11] hashar: my point is dont use bundle exec [14:50:20] ;) [14:50:37] if there is some genuing use cas eim missing though let me know [14:51:33] I think that covered the issue :-] [14:55:52] I have removed the facter gem and everything works as expected now \o/ [15:39:29] so... using dh-golang you get proper symlinks on /usr/share/gocode/pkg [15:51:31] <_joe_> vgutierrez: yes, and it's the wrong way to manage go packaging imho [17:54:41] I guess that depends a bit on whether you want/would like the Debian golang people to help look after the package in future :) [18:01:41] would someone be interested on a minisession about "tricks jynus uses to confirm user impact" to help with status page? [18:05:44] claime: you got 3 commits pending to be merged [18:06:56] vgutierrez: my bad, merging [18:07:31] I forgot that labs dummy had to be merged on puppetmaster too [18:09:50] btw, I have a not-perfectly-good patch that was merged earlier and causes deploy servers to try and apply something at each run. I can't really roll it back because it's going to end up doing the same thing, but I have a patch lined up (it needs review) [18:10:21] Ok if deploy1002/deploy2002 stay in that state for now ? It doesn't block anything, it just complains it can't remove a dir. [18:10:47] but I'd rather not make it worse [18:19:10] vgutierrez: It's done, sorry again [18:19:18] no problem :) [23:11:59] brett: thx for https://phabricator.wikimedia.org/T262996, it makes the error console for frontend devs _almost_ empty from "default" noise. The one remaining and related to the same cookie is https://phabricator.wikimedia.org/T261803, in case you're interested and still in the same mindspace :) [23:13:32] Our pleasure. Can't promise I'll get to that in the near future, it's still a pretty daunting experience :/