[07:33:31] <wikibugs>	 10serviceops, 10DC-Ops, 10ops-eqiad: Q4: (Need By: TBD) rack/setup/install mw14[57-98] - https://phabricator.wikimedia.org/T306121 (10akosiaris)
[07:35:03] <wikibugs>	 10serviceops, 10DC-Ops, 10ops-eqiad: Decommission mw13[07-48] - https://phabricator.wikimedia.org/T306162 (10akosiaris)
[07:35:21] <wikibugs>	 10serviceops, 10DC-Ops, 10ops-eqiad: Decommission mw13[07-48] - https://phabricator.wikimedia.org/T306162 (10akosiaris)
[07:35:43] <wikibugs>	 10serviceops, 10DC-Ops, 10ops-eqiad: Decommission mw13[07-48] - https://phabricator.wikimedia.org/T306162 (10akosiaris) 05Open→03Stalled Stalling until T306121 is done.
[08:07:44] <wikibugs>	 10serviceops, 10Scap, 10Release-Engineering-Team (Radar): Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10JMeybohm) 05In progress→03Resolved
[08:12:48] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar): docker-report-releng failing on multiple image tags because of certificate validation error - https://phabricator.wikimedia.org/T304875 (10JMeybohm) 05Open→03Resolved a:03JMeybohm I've removed the images from the...
[08:29:34] <wikibugs>	 10serviceops, 10Kubernetes: Replace kubeyaml in deployment-charts CI - https://phabricator.wikimedia.org/T306165 (10JMeybohm) p:05Triage→03Low
[08:30:44] <wikibugs>	 10serviceops, 10Kubernetes: Replace kubeyaml in deployment-charts CI - https://phabricator.wikimedia.org/T306165 (10JMeybohm)
[09:55:02] <wikibugs>	 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) datahub.wikimedia.org is now up {F35051150,width=60%} Now working on getting the datahub-gms.discovery.wmnet service up and running too.
[10:49:45] <btullis>	 Hello. Sorry to trouble you. I'm seeking reviews to make sure I'm on the right lines please. https://gerrit.wikimedia.org/r/c/operations/puppet/+/780651 for the service catalog and https://gerrit.wikimedia.org/r/c/operations/dns/+/780658 for DNS.
[11:08:36] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10SRE, 10Traffic, and 2 others: service::catalog entries and dnsdisc for Kubernetes services under Ingress - https://phabricator.wikimedia.org/T305358 (10akosiaris) > * The monitoring: stanza can't be added as having that without lvs: breaks icinga. Can potentially be ignor...
[11:21:39] <wikibugs>	 10serviceops, 10Math, 10MediaWiki-Categories, 10RESTBase, 10Patch-For-Review: <math>\land</math> – Unclear why the page appears in an error-category - https://phabricator.wikimedia.org/T305613 (10akosiaris) >>! In T305613#7850547, @Physikerwelt wrote: > OK, I am sending the output of the curl -v in priva...
[11:28:29] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10SRE, 10Traffic, and 2 others: service::catalog entries and dnsdisc for Kubernetes services under Ingress - https://phabricator.wikimedia.org/T305358 (10BTullis) >> The monitoring: stanza can't be added as having that without lvs: breaks icinga. Can potentially be ignored...
[12:28:37] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10SRE, 10Traffic, and 2 others: service::catalog entries and dnsdisc for Kubernetes services under Ingress - https://phabricator.wikimedia.org/T305358 (10JMeybohm) >>! In T305358#7854870, @akosiaris wrote: >> * The monitoring: stanza can't be added as having that without lv...
[12:31:03] <jayme>	 btullis: I'd like to postpone that for when we have an agreement on https://phabricator.wikimedia.org/T305358
[12:31:25] <jayme>	 aiui this should not be blocking for you, right?
[12:46:26] <btullis>	 jayme: Which bit would you like to postpone? Just the monitoring bit? The datahub-gms service itself is a blocker, as I need to use it to start ingesting stuff. It doesn't matter if it's active/active or using a temporary URL or not monitored at the moment though. Any temporary workarounds are probably fine.
[12:50:04] <jayme>	 all of it actually. If you can't use k8s-ingress-wikikube.discovery.wmnet for your ingestion stuff (because you'd have to set SNI), I'd only merge the DNS change for now
[12:54:40] <btullis>	 OK, yeah I don't know of a way to set SNI from the ingestion client, so I think that the DNS change would be the minimum required. Are you happy for me to merge that now and give it a go?
[12:57:30] <jayme>	 yes, but without guarantee that this will not get pulled again
[12:58:27] <btullis>	 Ack. Many thanks.
[13:28:42] <btullis>	 Is this expected? `unable to get local issuer certificate`
[13:28:46] <btullis>	 https://www.irccloud.com/pastebin/Iy0NOlz7/
[13:37:55] <vgutierrez>	  btullis don't forget to add the SNI using the -servername option
[13:39:05] <btullis>	 vgutierrez: Oh,OK. Will try now. I thought that the fact it was now using the CNAME would make SNI work anyway.
[13:39:20] <vgutierrez>	 nope, SNI is disabled by default on s_client
[13:42:33] <vgutierrez>	 btullis: BTW it looks like an issue on stat1008 regarding some root CA config
[13:43:25] <btullis>	 vgutierrez:  Thanks. Yeah, this seems to validate correctly: `openssl s_client -connect datahub-gms.discovery.wmnet:30443 -CAfile /usr/local/share/ca-certificates/Wikimedia_Internal_Root_CA.crt`
[13:43:49] <btullis>	 ...but I thought that root CA certificate was part of the normal bundle.
[13:44:00] <vgutierrez>	 it is.. at least on cp hosts
[13:44:07] <vgutierrez>	 but I'm not familiar with stat1008 and family
[13:47:14] <btullis>	 OK, maybe it's something to do with conda. Anyway, thanks all for your time. I've got my ingestion sort of working now.
[13:56:25] <btullis>	 Yeah, it was definitely conda. Apologies for wasting your time with that.
[15:35:13] <wikibugs>	 10serviceops, 10Math, 10MediaWiki-Categories, 10RESTBase, 10Patch-For-Review: <math>\land</math> – Unclear why the page appears in an error-category - https://phabricator.wikimedia.org/T305613 (10Physikerwelt) @akosiaris today I was thinking about a possible source of the problem. My hypothesis is that t...
[16:05:09] <wikibugs>	 10serviceops, 10Math, 10MediaWiki-Categories, 10RESTBase, 10Patch-For-Review: <math>\land</math> – Unclear why the page appears in an error-category - https://phabricator.wikimedia.org/T305613 (10Wurgl) Strange!  This one says "Deprecation: Alias no longer supported." curl -X POST 'https://de.wikipedia.o...
[16:58:21] <wikibugs>	 10serviceops, 10Release-Engineering-Team: Pushes to docker-registry are too slow - https://phabricator.wikimedia.org/T306201 (10dancy)
[17:15:20] <wikibugs>	 10serviceops, 10Math, 10MediaWiki-Categories, 10RESTBase, 10Patch-For-Review: <math>\land</math> – Unclear why the page appears in an error-category - https://phabricator.wikimedia.org/T305613 (10Wurgl) There is definitely some cache involved!  $ curl -X POST 'https://de.wikipedia.org/api/rest_v1/media/m...
[17:52:27] <wikibugs>	 10serviceops, 10Release-Engineering-Team: Pushes to docker-registry are too slow - https://phabricator.wikimedia.org/T306201 (10dancy)
[17:58:49] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10dancy) Problems detected today: * `WARNING: Kubernetes configuration...
[23:08:06] <wikibugs>	 10serviceops, 10GitLab (CI & Job Runners): upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn) All, 10, gitlab-runner instances in the cloud VPS project are now on bullseye.  The puppetmaster isn't but not sure if that is part of it.  And then there are other runners, used...