[06:42:29] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Make all httpbb tests pass on the mwdebug deployment. - https://phabricator.wikimedia.org/T285298 (10Joe) [06:43:41] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, and 2 others: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) 05Open→03Resolved a:03Joe [09:09:42] hi folks [09:09:52] I'd need some help in debugging the failure for https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/765199 [09:10:00] I am trying to add 3 new helmfile configs [09:10:26] but afaics it seems that CI fails when trying to run helmfile -e build [09:10:36] (in collect_fixtures) [09:22:42] I can take a loook [09:24:51] damn, I should have looked before saying that :-) [09:26:28] <_joe_> jayme: I smell an issue with CI, I can take a look [09:26:44] <_joe_> but I might drop afk pretty fast as I'm waiting the lenovo tech [09:27:43] <_joe_> yes, it's an uncaught exception, damn [09:27:49] yeah, I guess it would be more efficient if you would go check [09:27:56] thanks :) [09:27:58] <_joe_> jayme: yes I know what the issue is [09:28:01] ack [09:28:06] <_joe_> it's a change I made the other day, damn [09:28:24] I thought you'd have said "the issue is Luca" [09:28:35] <_joe_> the issue is always luca [09:34:43] <_joe_> elukey: I think I have a fix, testing [09:34:50] <_joe_> then I'll rebase your change on top of it [09:36:10] thanks a lot [09:36:36] <_joe_> {{done}} [09:36:49] <_joe_> https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/765226 if you're curious [09:37:31] <_joe_> sadly I had tested adding a chart, not a deployment [09:41:59] seems to work fine :) [09:42:07] going to merge both [09:43:10] <_joe_> thanks, and sorry for the inconvenience :/ [09:43:26] <3 [12:30:31] \o/ ingress + miscweb deployed to all wikikube clusters [12:41:42] wow!! [12:43:05] jayme: with https://gerrit.wikimedia.org/r/c/operations/dns/+/764738/ we'll loose the ability to (de)pool a single service in a single DC, is that intentional? [12:44:40] taavi: what do you mean? [12:53:33] elukey: currently we have a separate dnsdisc records which lets us pool/move servicse between dcs individually, but if there's a single dnsdisc etcd record (that's what that patch looks to be setting up to me, sorry if I'm assuming that wrong) for everything behind the ingress gateway that's not possible [13:02:04] taavi: we aim to still have dnsdisc records for every service but no seperate LVS for those using the ingress [16:34:04] 10serviceops, 10Gerrit, 10SRE: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10hashar) `gerrit2001.wikimedia.org` is a replica and can also be used as a spare to switch the primary service. It also serves repos over `gerrit-replica.wikimedia.org` which is used by various scripts an... [16:35:21] <_joe_> jayme: not in staging though, there we just want one dnsdisc entry right [16:49:38] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10Jdforrester-WMF) [16:51:42] what's preventing us from just skipping LVS for the k8s stuff at this point? figuring out BGP integration with the routers? [16:52:06] (sorry, random tangential drive-by thoughts) [16:53:49] <_joe_> bblack: there are some impediments to doing that, yes [17:30:25] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10JoKalliauer) [18:19:24] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10RobH) [18:19:39] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10RobH) [18:25:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Dzahn) @RobH (cc: @Jelto ) gitlab1002 has existed as a VM in the past, when contractors used it but the... [18:26:38] we should probably not re-use the name gitlab1002 [18:36:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10RobH) a:05Jclark-ctr→03LSobanski @lsobanski: Is it ok to shift these hostnames from gitlab100[23] to gi... [18:45:10] _joe_: yes, right. For staging there is just one entry/service ofc. But that's no difference to the current state either [18:46:50] bblack: there are some implications on k8s side (skipping LVS) that we've not completely figured out yet. That's why we go with LVS in first step [19:30:29] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) I created some docs how to git push over https (instead of ssh) to Phabricator. The idea is that we can give this to people who are sti... [19:40:30] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Done by Feb 23🔥): Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) @joe I tried running this today but it failed: ` dancy@deploy1002$ sudo -u mwbuilder /usr/bin/ma... [21:14:21] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) @Osnard Ok, thank you! I have today tried to test import your repo and in principal it worked though under a namespace of my own user na... [21:28:03] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [21:28:11] 10serviceops, 10Phabricator, 10Release-Engineering-Team: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10Dzahn) [21:31:06] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [21:31:25] 10serviceops, 10Phabricator, 10Release-Engineering-Team: move "releng-secrets" git repo away from Phabricator (stop pushing over ssh to phab) - https://phabricator.wikimedia.org/T301170 (10Dzahn) 05Open→03Resolved a:03Dzahn So.. we have now tried the "push over https" to this repo and written docs for... [21:31:33] 10serviceops, 10Phabricator, 10Release-Engineering-Team: move "releng-secrets" git repo away from Phabricator (stop pushing over ssh to phab) - https://phabricator.wikimedia.org/T301170 (10Dzahn) [22:05:45] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Done by Feb 23🔥): Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) 05Open→03In progress p:05Medium→03High [22:41:19] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) This is a saved query to show all repos that are: - hosted on phabricator - active (however that is defined) - git (1 - 4 svn special c... [22:42:27] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Sustainability (Incident Followup): Raw "upstream connect error or disconnect/reset before headers. reset reason: overflow" error message shown to users during outage - https://phabricator.wikimedia.org/T287983 (10RLazarus) This came up again in T301507. [23:47:01] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.0 - https://phabricator.wikimedia.org/T302464 (10dancy) [23:47:34] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Done by Feb 23🔥): Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [23:47:40] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.0 - https://phabricator.wikimedia.org/T302464 (10dancy) [23:59:42] 10serviceops, 10Performance-Team: Deprecate /static/current at WMF in favour of similar long-cache unversioned /w/ URLs - https://phabricator.wikimedia.org/T302465 (10Krinkle)