[05:43:35] 10serviceops, 10Data-Engineering, 10MW-on-K8s: IPInfo MediaWiki extension depends on presence of maxmind db in the container/host - https://phabricator.wikimedia.org/T288375 (10odimitrijevic) [10:58:49] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [10:59:03] jayme: o/ [10:59:57] I was about to ask you if https://gerrit.wikimedia.org/r/c/operations/puppet/+/769085 was ok but I realized that it is missing something [11:00:06] anyway, I think that we could start reimaging kubernetes2005+ [11:00:08] wdyt? [13:24:30] elukey: absolutely. Keep in mind that some of them are ganeti nodes, though! [13:24:48] *ganeti VMs [13:57:00] ah this is new to me! [14:09:54] the nodes dedicated to session store are the ganeti vms [14:18:31] https://netbox.wikimedia.org/virtualization/virtual-machines/?q=kubernetes [14:19:41] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [14:19:59] ah wow 2005 is indeed a ganeti node [14:20:08] I thought they were all bare metals [14:21:08] so at this point I'll start from 2007 [14:28:33] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [14:28:47] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: High API server request latencies (LIST) for istio API groups - https://phabricator.wikimedia.org/T303184 (10JMeybohm) 05Open→03Resolved Unfortunately I don't see istio pilot metrics exposing the issue. I will resolve this, although I'm not sure what the ca... [14:37:47] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, 10Service-deployment-requests: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) The helm charts and helmfile deployment are now passing the CI `helm-lint` stage. [14:38:52] 10serviceops: Test running php7.2 and php7.4 in parallel on the beta cluster - https://phabricator.wikimedia.org/T295578 (10JMeybohm) Thanks @Majavah ! wddx is no longer bundled with PHP >=7.4 and T295725 suggests we could just stop loading it. [14:46:03] jayme: ready for 2007 https://gerrit.wikimedia.org/r/c/operations/puppet/+/769085 [14:47:13] \o/ [14:52:30] all right draining and then reimaging :) [15:01:28] reimage started [15:17:10] if all goes fine I can probably do one/two hosts daily [15:17:29] that should allow us to have all workers on bullseye by the end of march (in theory) [15:33:26] 2007 done! [15:34:33] looks ok from my point of view, uncordoning it [15:38:51] nice [15:51:49] Sorry to be a pain, but is there anything I can do to help move these along? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/764375 and https://phabricator.wikimedia.org/T303049 [16:02:15] 10serviceops, 10SRE, 10Znuny: enhance Znuny (otrs) alerting - https://phabricator.wikimedia.org/T303190 (10Arnoldokoth) [16:15:48] jayme: next round, 2008 https://gerrit.wikimedia.org/r/c/operations/puppet/+/769463 [16:43:52] 10serviceops, 10Gerrit, 10SRE, 10Release-Engineering-Team (Seen): Deploy multi-site plugin to gerrit1001 and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10hashar) 05Open→03Declined It is really unlikely we will setup the multsite plugin though if we change our mind later we can always reope... [16:44:42] reimaging 2008 [16:47:42] btullis: unfortunately that is quite a bunch of stuff and our team is a bit understaffed currently. That's why it takes a bit of time :/ [16:50:42] jayme: OK, understood. Happy to try to help with anything I can. [16:54:33] I'll try to give the charts a another look today/tomorrow though. Do you have an idea regarding traffic expectations? [16:56:53] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, 10Service-deployment-requests: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) >>! In T303049#7753274, @BTullis wrote: > How can I tell what the source IP address(es) of my services will be, as seen by the back... [16:58:56] jayme: Thanks. Very low traffic indeed. We're really just at an MVP stage. Neither the extenal endpoint (datahub.wikimedia.org) nor the internal endpoint (datahub.discovery.wmnet) are likely to see much. The public endpoint will be password protected. [17:00:55] ack. Would you mind adding that to the service request ticket? Especially it was not clear to me that the public facing endpoint is actually restricted :) [17:08:05] jayme: still around, by any chance? [17:08:17] rzl: o/ [17:08:27] check my instinct on https://integration.wikimedia.org/ci/job/helm-lint/6925/console -- I think I just need to add a chmod 777 /var/log/envoy in the dockerfile, is that right? [17:08:43] that is, in https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/dockerfiles/helm-linter/Dockerfile.template where we already do so for /etc/envoy [17:09:04] (that validation error is on a no-op change, after bumping the envoy version to 1.18.3 in helm-lint) [17:10:39] hmm..looks like it, yeah. Maybe the new version writes something and the old does not [17:10:55] or the logfile is opened unonditionally now [17:11:24] kind of curious of something valuable is in there, though [17:12:34] I know they reworked some stuff about how the access logs work, in 1.16.0 and again in 1.18.0 -- this might be a side effect [17:13:19] I don't think anything in the linter setup ought to be talking to the admin port, so I can't imagine there's anything in the log?? but who knows, maybe [17:13:46] I'll add the chmod and see where that goes [17:14:41] ack. I think you should be good with 777 in helm-lint case! [17:14:59] and yes, nothing should be actually talking to envoy there ofc [17:15:17] thanks 👍 [17:16:21] rzl: a mkdir /var/log/envoy you might need as well [17:16:39] although..should be part of the debian package really [17:16:58] I poked around with docker run and it looks like it's created, empty with 755 permissions [17:17:24] owned by... envoy! so I guess our "USER nobody" is one of the moving parts here [17:18:32] Indeed. Btw. I think "HELM_HOME=/tmp/helm" is not needed anymore as well (leftover from the helm3 migration I guess). If you're going to rebuild the image again, you could remove that as well [17:19:01] kubernetes2008 uncordoned, all good and on bullseye :) [17:19:08] hm, I'm already blocking btullis so I'm going to keep this minimal and get it out as fast as I can, but noted for followup [17:19:13] "... v3 configuration no longer uses $HELM_HOME ..." ... [17:19:42] I'm blocking btullis as well I guess 😬 [17:19:57] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install parse100[01-24] - https://phabricator.wikimedia.org/T299573 (10Cmjohnson) @Dzahn I do not have enough space in row C to put more than 3 servers and all 3 of those are in one rack (C5). Can I put 3 in row C and then 3 in row... [17:22:45] mailed https://gerrit.wikimedia.org/r/c/integration/config/+/769477 -- actually do I bump the changelog in the same patch? looks like I do [17:24:26] fixec [17:24:29] *fixed [17:24:39] jayme: Yes, I'll add that traffic info to the service request. [18:13:07] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, 10Service-deployment-requests: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) > Those will be the IP ranges of the different k8s clusters (the non-ML ones). You can look those up in netbox: https://netbox.wikim... [18:15:10] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) [19:01:53] 10serviceops, 10Release-Engineering-Team: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10dancy) [19:18:29] 10serviceops, 10Internet-Archive, 10InternetArchiveBot: Determine appropriate API request limits for InternetArchiveBot - https://phabricator.wikimedia.org/T296577 (10Harej) 05Open→03Declined