[12:12:21] someone knows what basic metrics are worth graphing for a django app? https://netbox.wikimedia.org/metrics [12:34:12] I've only used NewRelic to monitor Django apps, but the 500 status code and db_query_durations would be good [12:35:46] XioNoX: I would pick a few to represent availability and latency, number of requests and number of errors, unless you have specific needs? [12:36:25] anything that can help identify slowdowns [12:37:48] volans:: hi! I'm trying to implement a solution for T310740 but I'm not knowledgeable enough about Puppet [12:37:48] T310740: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 [12:37:58] I'm trying to make scap targets bootstrap themselves so they don't depend on an async process on the deployment server. They will rsync to the deploy server and pull scap themselves [12:38:10] in order to do that, the relevant classes (such as "scap::target") need to be able to reference the current deployment server. That is the hieradata value "scap::deployment_server" if I'm not mistaken [12:38:22] but according to our docs, we should only inject hiera values into profiles: https://wikitech.wikimedia.org/wiki/Puppet_coding#Hiera [12:38:31] if I do that, I'm going to have to modify dozens of calls sites to pass along the deployment server value so it eventually reaches the relevant classes [12:38:37] I really don't like that approach [12:38:40] do you know if there is a better way to do this? [12:39:25] volans: oops, that stuff above :) [12:42:00] XioNoX: Then I think the database latency and the django_http_requests_latency_seconds_by_view_method. In my experience it's helpful to be able to identify the views that are slow. I'm not really sure how to be display that though [12:42:27] thanks, yeah that's what I'm trying to figure out right now :) [12:43:00] Maybe also latency before and after middleware, to pinpoint it it's in a view or in the middleware [12:43:55] Hmm, maybe you only get "including middleware" [12:53:27] slyngs: https://grafana.wikimedia.org/d/DvXT6LCnk/arzhel-netbox?orgId=1&from=now-3h&to=now&showCategory=Panel%20options&viewPanel=15 [12:53:32] getting there [12:54:51] That's pretty good, I wonder if you can get the method to display when hovering [12:56:45] that would need something multi-dimentional, dunno how we could display that [12:57:06] jnuche: vol.ans: is off on fridays [12:58:32] jnuche: however it sounds like you could just create a global hiera value for scap_deployment_server but probably best to talk on task [13:00:18] jbond: thanks! I'll continue the conversation on the task [13:05:14] hi all! can someone please confirm that varnish will compress RESTbase responses on the fly, and in the process mess with the etags? [13:05:45] I'm trying to hunt down the source of "weak" etags in parsoid responses. They are breaking If-Match semantics... [13:38:51] just a little fun satire, https://flak.tedunangst.com/post/from-apples-to-pears [13:55:08] duesen: AFAIK varnihs wont compress the headers but your best asking in traffic for an authorative answer [13:56:14] ...more channels... [13:56:51] i think they are all also in here but its less liky to get lost in the noise if you ask there [13:59:07] jbond: thanks, I asked there. [13:59:15] What did you mean by "compress the headers"? [14:00:21] duesen: first i would air on the side of waiting for an authorative answer then trying to interprit mine :). however i ment that afaik varnish may compress the http body but i dont think it will compress the headeres like If-Match [14:01:03] no, i don't think compressed headers are a thing in http... [14:01:40] but when returnign a compressed payload, varnish *modifies* certain header fields. in aprticular, content-encodiung and, unfortunately but unavoidably, the etag [14:01:50] Per https://varnish-cache.org/docs/6.0/users-guide/compression.html [14:02:17] such a "weakened" etag is unusable with If-match, per the http spec [14:03:55] why does compression force it to weaken the etag? [14:04:11] TIL, ahh ok thanks, https://github.com/wikimedia/puppet/blob/production/modules/varnish/templates/wikimedia-frontend.vcl.erb#L913-L945 seems to be the relevent part of varnish [14:06:34] duesen: is it possible to have restbase due the compression instead, or isthat undesireable? [14:07:35] jhathaway: we don't want RESTbase to do anything anymore, ideally :) [14:07:43] fair :) [14:08:15] jhathaway: HTTP says: different content-encodings MUST have a different etag. Ideally, we'd use transfer-encoding, then we could just pass the etag along. But apprently, clients don't support that well. [14:08:25] We live in the stone age of the internet... [14:09:16] thanks for the clarification, I can at least understand the rationale [17:18:26] Regarding the upcoming Debian Buster VM terminations, I see that bastion-restricted-eqiad1-02 is a remaining cloud bastion instance still on that version. Is there a plan for upgrading that? [17:48:43] brett: not yet, we still haven't even got rid of stretch on cloud vps [17:48:56] but bastion instances are easy compared to some other parts of the wmcs infra :P [17:49:05] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Infrastructure_projects [17:49:50] ugh, I got buster and stretch mixed up. Sorry ._. [19:08:24] arnoldokoth: go ahead and merge my patch too when you get to yours. 'increase rpc_response_timeout' [19:10:11] or... ok I'm just merging yours :) [19:28:28] andrewbogott: Sorry, got hungry before running puppet-merge. Thanks though. :) [19:28:45] sure thing! [19:32:29] Anyone had success combining pontoon & terraform on openstack? https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/terraform [19:32:49] I was considering trying to spin up a testing environment with such a combination [19:41:22] jhathaway: that's something I've been interested for a while, but haven't had time to look properly yet [19:43:16] taavi: I am trying to figure out a way to test some of the changes to our email setup, so I need something that allows me to coordinate a set of machines interacting, mail servers, dns servers, etc [19:43:32] and which hopefully won't bitrot immediately after I finish the project [19:43:33] authenticating to the openstack apis is currently the biggest problem for widespread use, I hope to look at that once T267194 is done [19:43:33] T267194: CloudVPS: enable TLS in openstack API endpoints - https://phabricator.wikimedia.org/T267194 [19:43:48] yeah that would be really nice [19:44:40] that one has a bunch of open patches pending review