[04:38:49] 10serviceops, 10SRE, 10docker-pkg, 10Patch-For-Review, 10Release Pipeline (Blubber): Container image lifecycle management - https://phabricator.wikimedia.org/T287130 (10RLazarus) The hourly `imagecatalog record` timer is working on deploy1002. It's failing on deploy2002, because something keeps overwriti... [08:01:39] 10serviceops, 10Data-Engineering, 10SRE, 10observability: Upgrade Kafka to 2.x - https://phabricator.wikimedia.org/T300102 (10elukey) [08:56:20] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10JMeybohm) a:03JMeybohm [09:32:29] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10JMeybohm) Canaries have been upgraded, did a `scap pull` from mwdebug1001 which seemed fine. Updating `A:restbase-canary` now as well, rest tomorrow. [12:22:50] 10serviceops, 10envoy, 10SRE Observability (FY2021/2022-Q3), 10User-fgiunchedi: Using port in Host header for thanos-swift / thanos-query breaks vhost selection - https://phabricator.wikimedia.org/T300119 (10Joe) p:05Triage→03Medium [14:50:22] 10serviceops, 10Data-Engineering, 10SRE, 10observability: Upgrade Kafka to 2.x - https://phabricator.wikimedia.org/T300102 (10Ottomata) > A better and more stable Kafka Mirror Maker (even if after all the work that Andrew did we have something very stable as well now) This really does look great, and has s... [15:55:26] 10serviceops, 10Discovery-Search, 10Elasticsearch, 10RESTBase, and 3 others: Determine future of bare-metal hosting for services like WDQS, ElasticSearch, RESTBase Cassandra, etc. - https://phabricator.wikimedia.org/T221315 (10Gehel) 05Open→03Resolved a:03Gehel There isn't much activity here and it s... [17:09:22] 10serviceops: move micro sites from ganeti to kubernetes - https://phabricator.wikimedia.org/T300171 (10Dzahn) a:03Dzahn [17:11:34] 10serviceops: move micro sites from ganeti to kubernetes - https://phabricator.wikimedia.org/T300171 (10Dzahn) [17:13:07] <_joe_> mutante: it would be cool if microsites were organized so that we can have one microsite: 1 pod, and use an ingress [17:13:24] <_joe_> talk to jayme when it's a better time of the day [17:15:06] _joe_: ok, i'll talk to him [17:27:19] 10serviceops, 10GitLab (Infrastructure): Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10aborrero) [17:27:27] 10serviceops, 10GitLab (CI & Job Runners): upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10aborrero) [17:44:12] mutante: maybe tomorrow 16.00Z or 17:00Z [18:29:13] * bd808 subscribes to mutante's task about moving microsites from ganeti to k8s [18:43:43] jayme: I can do the earlier one but not the later one. I sent a calendar invite [18:46:35] 10serviceops, 10GitLab (Infrastructure): Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) Thanks to @aborrero our quota increase request is resolved. We should now be able to have 2 additional temp. VMs at a time. So 2 of us could do a temp test at a time without to... [18:48:22] 10serviceops, 10GitLab (Infrastructure): Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) I clicked "allocate IP to project" and added description "gitlab-ssh" and we got assigned 185.15.56.79 [19:23:55] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10dancy) [19:25:00] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10dancy) @JMeybohm A bug has been detected in 4.2.0. Please roll back to 4.1.1 while I work on a fix. [19:41:31] 10serviceops, 10SRE, 10Patch-For-Review: Remove mediawiki::packages::fonts from non thumbor servers - https://phabricator.wikimedia.org/T294378 (10Dzahn) I checked the "status code 5xx" graph in the "RED dashboard" for appservers, looked at the last 30 days, especially if anything went up since Jan 17 when I... [19:44:14] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) [19:44:58] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) >>! In T300058#7654121, @dancy wrote: > @JMeybohm A bug has been detected in 4.2.0. Please roll back to 4.1.1 while I work on a fix. Actually, please restart the process using t... [19:45:52] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) [20:17:38] 10serviceops, 10SRE, 10Patch-For-Review: Remove mediawiki::packages::fonts from non thumbor servers - https://phabricator.wikimedia.org/T294378 (10Dzahn) purged remaining font files on mw13*