[08:02:13] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Import coredns 1.8.x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T321159 (10elukey) [08:09:44] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Import coredns 1.8.x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T321159 (10elukey) We are currently running 1.5.2 on our k8s clusters. List of changes (https://coredns.io/tags/release/): https://coredns.io/2019/07/28/coredns-1.6.0-release/ https:... [08:26:22] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Import coredns 1.8.x (k8s 1.23 dependency) - https://phabricator.wikimedia.org/T321159 (10elukey) [08:30:14] 10serviceops, 10MW-on-K8s, 10SRE: Re-think how we separate traffic to mediawiki in clusters. - https://phabricator.wikimedia.org/T291918 (10Joe) >>! In T291918#7387775, @Joe wrote: >>>! In T291918#7387656, @jijiki wrote: >> Naming things is hard though, I do not agree with the `kube` prefix, in the future af... [08:43:10] 10serviceops, 10MW-on-K8s, 10SRE: Re-think how we separate traffic to mediawiki in clusters. - https://phabricator.wikimedia.org/T291918 (10Clement_Goubert) >>! In T291918#7387656, @jijiki wrote: > Naming things is hard though, I do not agree with the `kube` prefix, in the future after baremetal mediawiki se... [09:45:43] 10serviceops, 10MW-on-K8s, 10SRE: Re-think how we separate traffic to mediawiki in clusters. - https://phabricator.wikimedia.org/T291918 (10Joe) `mw-main` is probably the least misleading one, yes. I would like `mw-web` more, but it's going to mislead a lot of people into thinking it's just requests to wiki... [09:47:16] 10serviceops, 10MW-on-K8s, 10SRE: Re-think how we separate traffic to mediawiki in clusters. - https://phabricator.wikimedia.org/T291918 (10Clement_Goubert) This may be a stupid question but why would the api requests coming from browsers not go to the endpoint mapped to `mw-api-ext`? [09:48:03] 10serviceops, 10MW-on-K8s, 10SRE: Re-think how we separate traffic to mediawiki in clusters. - https://phabricator.wikimedia.org/T291918 (10jijiki) >>! In T291918#8328103, @Clement_Goubert wrote: >>>! In T291918#7387656, @jijiki wrote: >> Naming things is hard though, I do not agree with the `kube` prefix, i... [09:53:45] 10serviceops, 10MW-on-K8s, 10SRE: Re-think how we separate traffic to mediawiki in clusters. - https://phabricator.wikimedia.org/T291918 (10Joe) >>! In T291918#8328319, @Clement_Goubert wrote: > This may be a stupid question but why would the api requests coming from browsers not go to the endpoint mapped to... [10:01:31] 10serviceops, 10MW-on-K8s, 10SRE: Re-think how we separate traffic to mediawiki in clusters. - https://phabricator.wikimedia.org/T291918 (10Clement_Goubert) Given both of your answers, I think `mw-web` is actually the better choice, barring calling it `mw-real-users` which is kind of weird. The API calls fro... [11:42:09] 10serviceops, 10MW-on-K8s, 10SRE: Re-think how we separate traffic to mediawiki in clusters. - https://phabricator.wikimedia.org/T291918 (10Clement_Goubert) Since there seems to be consensus on everything but `mw-{app,main,web}`, I'll consider these other service names as valid going forward unless told othe... [12:50:17] https://github.com/moby/buildkit/pull/3093 [12:50:48] That's gonna be interesting to add to the pipeline at some point [12:51:30] Makes it a bit easier to track down what's being copied in the image and thus 1 more layer against some supply chain attacks [12:56:18] 10serviceops, 10Prod-Kubernetes, 10SRE, 10Kubernetes, 10User-fsero: Set up a local redis proxy since docker-registry can only connect to one redis instance for caching - https://phabricator.wikimedia.org/T215809 (10JMeybohm) p:05Medium→03Low [13:02:45] 10serviceops, 10MediaWiki-Authentication-and-authorization, 10Platform Engineering, 10SRE: Increased session loss since 20221001 - https://phabricator.wikimedia.org/T319279 (10LSobanski) [13:08:51] 10serviceops, 10SRE, 10Patch-For-Review: Deploy MediaWiki config change to use OpenSSL for PBKDF2 password hashing - https://phabricator.wikimedia.org/T320929 (10LSobanski) [13:11:42] 10serviceops, 10API Platform, 10SRE: Block non-browser requests that use generic agents - https://phabricator.wikimedia.org/T319423 (10LSobanski) [14:28:59] 10serviceops, 10MW-on-K8s: Deploy new mw-debug service - https://phabricator.wikimedia.org/T321201 (10Clement_Goubert) [15:13:53] 10serviceops, 10Observability-Tracing: OpenTelemetry Collector running as a DaemonSet on Wikikube - https://phabricator.wikimedia.org/T320564 (10Clement_Goubert) We will stay on a DaemonSet deployment for v0, with the possibility to adapt in the future depending on the service needing telemetry. [15:37:27] <_joe_> ohhh https://kubernetes.io/blog/2022/09/19/local-storage-capacity-isolation-ga/ [15:38:18] <_joe_> but it's just capacity, not iops [15:38:21] <_joe_> meh. [15:38:34] doing it for iops is quite hard [15:38:47] "Typically, both /var/lib/kubelet and /var/log are on the system's root filesystem. If users configure the local storage in different ways, kubelet might not be able to correctly measure disk usage and use this feature." wut. [15:39:39] It takes some confidence to call "/var/log on a separate partition" a atypical usage pattern [15:45:42] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Deploy new mw-debug service - https://phabricator.wikimedia.org/T321201 (10Clement_Goubert) 05Open→03In progress [15:46:20] 10serviceops, 10MediaWiki-Authentication-and-authorization, 10Platform Engineering, 10SRE: Increased session loss since 20221001 - https://phabricator.wikimedia.org/T319279 (10hnowlan) [15:50:57] 10serviceops: Ensure role::mediawiki::appserver bootstraps cleanly - https://phabricator.wikimedia.org/T319168 (10Clement_Goubert) p:05Medium→03Low [15:51:51] 10serviceops: Ensure configcluster bootstraps cleanly - https://phabricator.wikimedia.org/T318699 (10Clement_Goubert) p:05Medium→03Low [15:52:10] 10serviceops: Ensure that all appserver-related roles can be cleanly applied on bootstrap - https://phabricator.wikimedia.org/T318671 (10Clement_Goubert) p:05Medium→03Low [19:59:23] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) [20:10:40] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) @thcipriani @LSobanski status update: - /srv/repos has been synced from phab1001 to phab1004 and phab... [20:45:40] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10thcipriani) >>! In T313360#8330520, @Dzahn wrote: > @thcipriani Now all that's left is the "Ensure there's a... [20:50:35] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) @thcipriani Yes, next thing that is needed here is a deployment of phabricator to host `phab1004`. It... [20:58:43] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) migration steps from last time in 2019:..found: https://etherpad.wikimedia.org/p/Phabricator-migratio... [21:06:18] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) [21:06:40] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [21:07:03] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) 05In progress→03Resolved new etherpad started, rsync commands added: https://etherpad.wikimedia.o... [21:07:16] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) https://etherpad.wikimedia.org/p/Phabricator-migration-2022 [21:08:54] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹️): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) service has actually been removed from LVS/pybal/conftool now. (thanks bblack for...