[08:00:26] <_joe_> oncallers: I'm deploying hiddenparma. It's a low-risk release with a couple bugfixes and a new feature that still isn't wired in varnish. there will be a couple minutes of downtime of the web ui. [08:01:57] thanks _joe_ [11:33:26] We've published a docker image with a typo in the name. Is it possible to remove it from the registry? cf https://docker-registry.wikimedia.org/repos/data-engineering/growthbook/repos/data-engineering/growthbook-next/tags/ [11:36:01] well, not exactly a typo, but somehow kokkuri inferred a crazy-looking image path/name [11:40:51] brouberol: I'm no expert, but there is https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images (can't guarantee is up to date though) [11:42:12] thanks, looking [15:39:53] brouberol: yes that works, caveat being that it will only stop being listed. The data will still be around in the registry, just not addressable. [15:57:39] gotcha, thanks. That's better that naught [18:51:46] oncallers: I'm going to be switching codfw PyBals over to watching a different etcd node in a few minutes, ahead of some etcd-related maintenance tomorrow (T352245). [18:51:46] there will be some transient icinga check noise (e.g., `Check if Pybal has been restarted after pybal.conf was changed`), but otherwise nothing notable expected. [18:51:46] T352245: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245 [19:08:10] thoughts on T410202, which rightfully brings up the question of where should we add a link to? [19:08:10] T410202: Improve bug reporting instructions - Where to report not to just who should receive it - https://phabricator.wikimedia.org/T410202 [19:08:14] > i had T410201, but the error page just says "If you report this error to the Wikimedia System Administrators, please include the details below." without any links to any report page, or any instructions on where to report. [19:08:15] T410201: Error: 503, Backend fetch failed - https://phabricator.wikimedia.org/T410201 [19:08:44] I was thinking https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue but that's not really for 503s for example, in a way. that's a specific connectivity one. [19:09:19] historically I think the official answer has been "email noc@" but there are pros and cons to putting a mailto: link in the 503 page [19:09:48] in some ways though, a phab task though is probably better than a noc@ email. because then it can directed towards the right team in the Phab interface itself [19:10:29] yeah - we'd probably want a link to a private task form, since people will be dumping IP addresses etc in there, right [19:10:58] there's a lot of shared fate there, if someone is getting other kinds of errors [19:11:09] mmm [19:11:25] (much like how if we're going to use wt:Reporting_a_connectivity_issue we should use the wikitech-static link) [19:11:42] (but I agree it's still not quite right for the use case anyway) [19:14:11] now that we are generating x-req-id in haproxy, we should maybe include it on all error pages [19:15:06] yeah that's not a bad idea. the varnish XID is already there for example [19:15:41] so any thoughts here on the right form? should we make a new generic private Phab one? [19:15:44] noc@? [19:16:46] slightly related: T401489 [19:16:46] T401489: Separate error templates for 5xx (server errors) and 4xx (IP blocks) or parametrize Varnish messaging - https://phabricator.wikimedia.org/T401489 [19:16:52] I think either noc@ or a Phab form are fine interim solutions but shouldn't be long-term ones [19:17:18] I have to afk for now though, sorry [19:17:19] taavi: hmm interesting thanks, that was certainly missed [19:17:29] cdanis: no worries thanks. let's see if others have more opinions [19:19:54] there is already "generic private phab ticket". just link to https://phabricator.wikimedia.org/maniphest/task/edit/form/23/ - private here means WMF staff and others under NDA. that's not public but a level under security task [19:20:46] unless you want custom form fields beyond the standard title/body [19:20:59] re the original task, my immediate concern is that if we add, say, noc@ on the generic 5xx error page, then we're going to end up getting DoSed from various individual reports during larger outages [19:21:42] mutante: that form is restricted to people in WMF-NDA or similar groups, so it's not useful as a general user-facing tool [19:22:20] taavi: it's restricted to WMF-NDA AND the reporter of the ticket [19:22:51] there should be a rule that is like "or subscriber" [19:23:20] you're confusing the policy that form sets to newly created tasks, and the policy controlling who can use the form to create a newn task [19:29:48] I see. checked policy of the form itself. yea, ack. maybe we should change that. in that case.. the "new security issue form" does not have that restiction. [19:33:01] FYI, I'm done messing with pybal :)