[09:40:27] XioNoX: volans: im looking at the netbox report errors and from what i can tell the alerts for cables.Cables, coherence.Coherence and puppetdb.PhysicalHosts are all valid, can yuo confirm? [09:40:59] jbond: netbox or netbox-next? [09:41:00] in relation to the network.Network report the results look incorrect to me (incorrect on both netbox1001 and 1002) is this a known bug? [09:41:03] netbox [09:41:24] checking [09:41:35] thx [09:43:12] accounting, cables, coherence, network, physical hosts: are all failing and all seems to be legit [09:43:15] with some things to be fixed [09:43:53] usually T283483 gets invoked :D [09:44:05] jbond: ^^^ [09:44:24] volans: ack thanks so not related to upgrade work [09:44:48] if they are the same between netbox prod and the new infra I'd say so [09:44:57] cool thanks [10:56:19] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade Fastnetmon to 1.2.1 - https://phabricator.wikimedia.org/T271228 (10ayounsi) 05Open→03Resolved a:03ayounsi All done! [11:01:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Agree how to handle port-block speeds for QFX5120-48Y - https://phabricator.wikimedia.org/T303529 (10ayounsi) Indeed, that looks great! [11:48:43] volans: im having a bit of trouble getting the accounting.Accounting report to work from the new server. im pretty sure this is bacuse it makes a call to the google apis and as such needs some proxy config. [11:49:03] i have added poroxy config to netbox1002 https://gerrit.wikimedia.org/r/c/operations/puppet/+/802096 and also tried running the job directly with [11:49:07] https_proxy=http://webproxy:8080 /srv/deployment/netbox/venv/bin/python /srv/deployment/netbox/deploy/src/netbox/manage.py runreport accounting.Accounting [11:49:44] but im still getting [somewhat unusefull] errors https://phabricator.wikimedia.org/P29316 any ideas? [11:49:59] let me chec [11:51:31] fyi using `https_proxy=http://webproxy:8080 /srv/deployment/netbox/venv/bin/python /srv/deployment/netbox/deploy/src/netbox/manage.py nbshell` and manually calling the google apis works (we get a timeout without the proxy setting) [11:53:50] jbond: maybe here: https://github.com/wikimedia/puppet/blob/production/modules/netbox/manifests/init.pp#L172 [11:55:48] XioNoX: ack that should be easy enough to add ill draft the CR and see what vol.ans says, thx [11:59:49] how was it working before? [12:00:14] volans: netbox1001 has a public ip address [12:00:20] (no need for a proxy) [12:00:32] don't we block outgoing traffic anyway? [12:00:38] no [12:00:53] not AFAIK [12:01:41] right I can curl google [12:02:47] volans: i dont think we block anything outbound on the routers but there may be some very specific things like memchahd or something. XioNoX should be able to cofirm [12:03:37] (and the iptables OUTPUT chain is empty) [12:03:53] nah we don't block anything, it's a config issue to me [12:08:11] FYI netbox 3 has added support for HTTP_PROXIES [12:08:12] https://github.com/netbox-community/netbox/commit/05cb47e650617e95edcacbe2efc2bb343e14c152 [12:08:44] volans: https://gerrit.wikimedia.org/r/c/operations/puppet/+/802096/3/modules/netbox/templates/configuration.py.erb :) [12:08:44] I think we can live few days without the accounting report if the fix for the current setup is complex [12:08:45] volans: that was added in netbox 2.8 and is the patch i sent above [12:08:55] https://gerrit.wikimedia.org/r/c/operations/puppet/+/802096 [12:09:23] ack [12:09:26] +1 [12:10:21] fyi i also tried the following as per XioN.oX recomendation but also dosn;t work [12:10:24] grep HTTPS /etc/uwsgi/apps-enabled/netbox.ini [12:09:44] [12:10:27] env=HTTPS_PROXY=http://webproxy.eqiad.wmnet:8080 [12:10:55] ok [12:15:29] jbond: it's working from the UI at least [12:15:36] unless you changed something [12:15:44] https://netbox.wikimedia.org/extras/reports/results/3183050/ [12:15:47] XioNoX: oh i just fixed it manually was just putting the change into a cr :) [12:15:57] nice! [12:16:18] XioNoX: i think if you try again it will fail [12:17:54] yep, errored [12:19:33] this works but open to better suggestions if you have any https://gerrit.wikimedia.org/r/c/operations/software/netbox-extras/+/802127 (cc volans XioNoX ) [12:19:55] looking [12:20:46] I don't have a better suggestion :) [12:23:15] ack will see if vola.ns has anything better cheers [12:24:57] the previous CR doesn't work? [12:25:17] we did inject env variables from puppet in the past [12:25:59] volans: i tested it manually by adding the value to /etc/uwsgi/apps-available/netbox.ini and it didn;t work. however i think that makes senses as the report is not executed by uwsgi [12:26:32] depends how you run it [12:26:35] from the web UI it is [12:26:41] from the API too [12:26:51] the systemd::timer runs it with [12:26:51] sudo -u netbox /srv/deployment/netbox/venv/bin/python /srv/deployment/netbox/deploy/src/netbox/manage.py runreport accounting.Accounting [12:26:52] how do we run it in the timers? [12:26:57] ack [12:30:15] volans: FYI it also fails running it via the web interface with the uwsgi config. is this something todo with the redis job schedualer? [12:30:33] no idea [12:30:39] but let me recap one sec [12:30:49] is the HTTPS_PROXY value set in the netbox config not working? [12:30:56] not as an env variable [12:32:03] . /etc/netbox/configuration.py includes HTTP_PROXIES but that dosn;t work [12:32:27] adding env=HTTPS_PROXY=http://webproxy.eqiad.wmnet:8080 didn't work /etc/uwsgi/apps-available/netbox.ini [12:32:53] https_proxy=http://webproxy:8080 /srv/deployment/netbox/venv/bin/python /srv/deployment/netbox/deploy/src/netbox/manage.py runreport accounting.Accounting (didn;t work) [12:33:00] this last one for me is the most intresting [12:33:40] volans: the netbox and uwsgi config are both in place on netbox1002 if you wanted to test anything [12:34:30] /srv/deployment/netbox/deploy/src $ git grep HTTP_PROXY [12:34:32] returns empty [12:34:40] so I don't think the version we have supports them [12:34:48] you sure was added in 2.8? [12:35:12] volans: its HTTP_PROXIES [12:35:20] whoops :D [12:35:36] and yes :) https://docs.netbox.dev/en/stable/release-notes/version-2.8/ [12:35:46] #4502 - Enable configuration of proxies for outbound HTTP requests [12:37:10] >>> from django.conf import settings [12:37:13] >>> settings.HTTP_PROXIES [12:37:16] has the correct value [12:37:35] but it seems it was added only to the webhooks [12:37:39] in netbox/extras/webhooks_worker.py [12:37:52] and netbox/utilities/background_tasks.py [12:38:16] the original issue was about webhooks https://github.com/netbox-community/netbox/issues/4502 [12:38:24] in get_releases [12:38:31] but yes i assuymed it was only about outgoing connections it new of [12:38:31] so I don't think the reports are using it [12:38:54] *knew off [12:39:19] either way i think its usefull to leave that in place even though it dosn't help wit this specific aissue [12:39:31] agree [12:39:32] jbond: it doesn't look like you can use the ini file or env, the patch makes it look like it specifically need to be in the Django settings [12:39:49] the problem with your patch jbond is that it will add it to netbox globally [12:39:52] not only that report [12:40:41] if that run() is ever run in the uwsgi env [12:42:39] I'm wonering if we can pass it to the google APIs [12:42:41] instead [12:43:02] slyngs: whic patch are you refering too? we have a number of things https://gerrit.wikimedia.org/r/c/operations/puppet/+/802105/1/modules/netbox/manifests/init.pp adds the config to the uwsgi app lication python environment. https://gerrit.wikimedia.org/r/c/operations/puppet/+/802096 adds support to the netbox django app [12:43:20] Sorry, I was looking at the Netbox patch on Github [12:43:39] volans: i looked at passing it to the google apis when doing the check_user script and wasn't able to [12:43:47] slyngs: no problem :) [12:43:52] ack got it [12:44:00] volans: let me see if i left any usefull comments anywhere [12:44:54] apparently there is from google.auth.transport.requests import Request [12:45:16] but not sure if would work [12:45:23] reading contradicting things [12:46:50] i have a comment which says [12:46:51] # the google api libraries use httplib2 which by default [12:46:56] (in cehck_user) [12:48:22] something like this maybe? https://github.com/googleapis/google-api-python-client/issues/1078 [12:50:18] volans: yes it was something like that. [from memory] i tried creating the http object but was unable to pass it correctly [12:50:42] i can test that in the report code though [12:51:44] ahh it was possibly the authorized_http = google_auth_httplib2.AuthorizedHttp(credentials, http=http) but i was missing i think passing ..( http=http, credentials=credentials) was not compatable [12:52:21] ack [12:58:55] volans: that works will update thanks [12:59:12] ack no prob [12:59:27] jbond: and to avoid hardcoding, you could read teh config from django [12:59:51] volans: ack good idea [13:01:38] or netbox, I don't recall the more netbox-way of reading it