[07:10:41] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.1 - https://phabricator.wikimedia.org/T302464 (10Joe) 05In progress→03Resolved [07:10:45] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Done by Feb 23🔥): Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10Joe) [07:13:16] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, and 2 others: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10Joe) 05Open→03Stalled a:05Joe→03None De-assigning from myself as I can't do anything more for this task in its current status. Also reflecting it... [07:14:25] 10serviceops: Test running php7.2 and php7.4 in parallel on the beta cluster - https://phabricator.wikimedia.org/T295578 (10Joe) a:05Joe→03JMeybohm re-assigning to Janis as he's been doing work in this direction in the last week. [08:10:24] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) [08:20:08] 10serviceops, 10SRE, 10observability, 10Patch-For-Review: aggregate mismatched wikiversions alert - https://phabricator.wikimedia.org/T302832 (10fgiunchedi) I think a short term easy fix would be to make the check warning (i.e. icinga/alerts.w.o only) instead of critical so it doesn't spam irc, what do you... [08:53:58] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) [09:20:26] jayme: o/ ready to add kubernetes2018 to wikikube (sent two code reviews) [09:22:00] (first one with bullseye and overlayfs) [09:22:43] also TIL profile::kubernetes::node::kubelet_node_labels [09:23:07] better: TIL region/row [09:23:12] I don't think that ML has then [09:24:30] (now that we have to reinit the cluster..) [09:32:47] elukey: do you want to add the nodes one-by-one? [09:33:34] jayme: nono but I wanted to start with one just to be sure [09:33:42] ack [09:33:42] it is the first with the new settings on wikikube [09:33:59] yeah, makes sense! [09:34:14] once we are ok we can add the others in one go (or two batches) [09:38:14] sounds good. +1ed [09:39:48] thanks! [09:40:12] all right adding the node [09:48:46] ok so puppet completed (I took 2 times due to kubelet/kubeproxy not coming up at first try, but probably some race condition) [09:49:57] at this point I am inclined to merge the bgp change too [09:50:56] I think that's fine [09:51:03] perfect [09:51:14] did you figure what kubelet/proxy was missing on first run [09:51:19] *? [09:53:18] nope I am checking logs [10:02:47] bgp sessions established, all good on the routers afaics [10:02:57] the calico pod is up and running [10:14:32] all good! [10:14:50] I think that we have the first wikikube bullseye+overlay node :) [10:15:12] \o/ [10:15:23] * jayme deploying mathoid onto it [10:17:04] $ kubectl -n mathoid get po -o wide | grep kubernetes2018 -c [10:17:05] 13 [10:19:23] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) With the help of @Majavah the correct configuration of private and public/floating IP was found. https and cloning over SSH works now. Thanks again! The t... [10:22:10] nice! [10:22:22] what did you do to create pods on it? A roll restart? [10:25:22] nah, I actually had to deploy an update right now :D [10:25:40] ah okok :D [10:25:51] JIT deployment ;) [10:26:58] but feel free to roll some other deployments to have some more work sheduled there ofc if you like [10:31:17] nah I think it is good [10:31:27] we can better check how mathoid runs on the new node, more limited [10:43:00] 10serviceops, 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, and 2 others: Upgrade Kafka Risk Evaluation - https://phabricator.wikimedia.org/T302610 (10JMeybohm) p:05Triage→03Medium [10:51:23] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: setup/install kubernetes20[1(89)|2(012)] - https://phabricator.wikimedia.org/T302208 (10elukey) Status: * We have added kubernetes2018 to the wikikube codfw cluster, it is currently serving traffic. * The remaining four nodes will be added after testing... [10:51:45] filed the remaining code reviews, we can try to add them tomorrow or later on [11:26:15] elukey: nice! I'll review them after lunch [13:13:38] 10serviceops, 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, and 2 others: Upgrade Kafka Risk Evaluation - https://phabricator.wikimedia.org/T302610 (10elukey) @EChetty hi! Could you add some details about what you expect to see in this task? [15:41:51] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.1 - https://phabricator.wikimedia.org/T302464 (10dancy) Thanks @Joe ! [15:52:36] lenovo guy will replace my mainboard (well, the one in my laptop) tomorrow morning. So if I don't show up - he failed :p [16:19:39] 10serviceops, 10MediaWiki-extensions-PropertySuggester, 10Wikidata, 10wdwb-tech, 10Service-deployment-requests: New Service Request SchemaTree - https://phabricator.wikimedia.org/T301471 (10Addshore) >>! In T301471#7714044, @Joe wrote: > You also say the service will need to read an index file. How is th... [16:22:42] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Done by Feb 23 🧟): Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [16:25:57] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10Scap: Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10dancy) [18:46:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install parse100[01-24] - https://phabricator.wikimedia.org/T299573 (10Jclark-ctr) [18:49:30] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install parse100[01-24] - https://phabricator.wikimedia.org/T299573 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson [19:16:27] 10serviceops, 10MediaWiki-extensions-PropertySuggester, 10Wikidata, 10wdwb-tech, 10Service-deployment-requests: New Service Request SchemaTree - https://phabricator.wikimedia.org/T301471 (10QChris) >>! In T301471#7725821, @Joe wrote: > Ok so a few requirements: > 1) we need the repository to be on gerri... [20:03:26] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.2 - https://phabricator.wikimedia.org/T302919 (10dancy) [20:03:40] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.2 - https://phabricator.wikimedia.org/T302919 (10dancy) p:05Triage→03High [20:21:48] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.2 - https://phabricator.wikimedia.org/T302919 (10Dzahn) a:03Dzahn [20:22:55] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.2 - https://phabricator.wikimedia.org/T302919 (10Dzahn) built following the docs, except replaced deneb with build2001, that's the new build host replacing deneb. uploaded and imported on apt1001, copied to strech and bullseye from bu... [20:42:35] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.2 - https://phabricator.wikimedia.org/T302919 (10Dzahn) ` scap was updated: 4.4.1-1 -> 4.4.2-1 mw[2252,2271-2272,2374,2376].codfw.wmnet,mw[1311,1414-1418,1447-1450 ].eqiad.wmnet,mwdebug[2001-2002].codfw.wmnet,mwdebug[1001-1002].eqiad... [20:50:10] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.2 - https://phabricator.wikimedia.org/T302919 (10Dzahn) I did everything the docs say to do (https://wikitech.wikimedia.org/wiki/Scap/Release) including test deploy etc.. except the very last step "Later or the next day you can rollout... [21:48:56] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.2 - https://phabricator.wikimedia.org/T302919 (10Dzahn) 05Open→03Resolved ` scap was updated: 4.4.1-1 -> 4.4.2-1 an-airflow[1001-1002].eqiad.wmnet,an- coord[1001-1002].eqiad.wmnet,an-launcher1002.eqiad.wmnet,an-test- client1001.e... [21:49:58] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.4.1 - https://phabricator.wikimedia.org/T302464 (10Dzahn) scap 4.4.2 built and rolled out everywhere (T302919) [22:54:11] 10serviceops, 10GitLab (CI & Job Runners), 10Patch-For-Review: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn) - created a new instance `gitlab-runner-1001 [23:25:22] 10serviceops, 10GitLab (CI & Job Runners), 10Patch-For-Review: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn) [23:25:38] 10serviceops, 10GitLab (CI & Job Runners), 10Patch-For-Review: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn) 05Open→03In progress