[00:09:40] 10serviceops, 10GitLab (CI & Job Runners), 10Patch-For-Review: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn) The package has been imported to our repo for bullseye per above. Running puppet on the test instance in cloud VPS installed it succesfully. ` dzahn@gitla... [00:26:09] 10serviceops, 10Generated Data Platform, 10Image-Suggestions, 10SRE, and 2 others: Blubber setup for Image Suggestions Service - https://phabricator.wikimedia.org/T305155 (10Dzahn) >>! In T305155#7823133, @Dzahn wrote: > port reserved: 4017 > > https://wikitech.wikimedia.org/wiki/Kubernetes/Service_ports... [00:54:42] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) p:05Medium→03High [08:12:51] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.0 - https://phabricator.wikimedia.org/T305250 (10JMeybohm) 05Open→03Resolved Rolled out everywhere [08:44:11] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.0 - https://phabricator.wikimedia.org/T305250 (10jnuche) @JMeybohm thank you so much for the quick response [08:55:25] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Migrate kubernetes masters to bullseye - https://phabricator.wikimedia.org/T305435 (10JMeybohm) p:05Triage→03Medium [10:12:21] jayme, _joe_: here we try again... ok to install new spicerack on cumin2002 for some testing? [10:12:33] <_joe_> volans: go on [10:12:52] you can test the new stuff with https://wikitech.wikimedia.org/wiki/Spicerack#Test_newly_released_Spicerack_features [10:13:05] <_joe_> ack thanks, will get around it today hopefully [10:13:37] I'll do som ebasic dry-run testing to make sure there are no errors that could affect the rest of spicerack/cookbooks [10:14:00] entry point is https://doc.wikimedia.org/spicerack/master/api/index.html#spicerack.Spicerack.kubernetes and related docs linked there [10:14:12] what are valid group/cluster names? [10:20:28] _joe_: something missing on the puppet side? ls: cannot access '/etc/kubernetes': No such file or directory [10:20:50] this is me directly doing 'ls', I saw from the code that it would try to load data from there though [10:20:59] <_joe_> volans: ah yes [10:21:03] <_joe_> doh [10:21:11] <_joe_> we need to install the kubeconfigs there :P [10:21:17] :) [10:24:56] anyway, nothing is broken, I'll deploy to cumin1001 too [10:34:15] <_joe_> good [10:34:21] <_joe_> I'll prepare a patch today [10:38:36] thx [13:17:14] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [13:24:02] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Migrate kubernetes masters to bullseye - https://phabricator.wikimedia.org/T305435 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=47c39d99-fe15-4681-bcbe-2e46700d49e8) set by jayme@cumin1001 for 3:00:00 on 1 host(s) and... [14:31:46] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Migrate kubernetes masters to bullseye - https://phabricator.wikimedia.org/T305435 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7323edc0-9118-4382-85e3-e1fb3b72fcaf) set by jayme@cumin1001 for 3:00:00 on 1 host(s) and... [14:53:03] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Migrate kubernetes masters to bullseye - https://phabricator.wikimedia.org/T305435 (10JMeybohm) [15:12:13] 10serviceops, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog: Mobileapps is often throttled on codfw - https://phabricator.wikimedia.org/T305482 (10akosiaris) [16:10:28] 10serviceops, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog: Mobileapps is often throttled on codfw - https://phabricator.wikimedia.org/T305482 (10akosiaris) After patch was merged and deployed we have happier graphs! Before ===== avg {F35039775} max {F35039773} After ==== avg {F3503984... [19:03:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Jclark-ctr) [19:23:50] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) >>! In T274463#7830451, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://sal.toolf... [20:35:37] 10serviceops, 10SRE, 10Patch-For-Review: Debian package for httpbb - https://phabricator.wikimedia.org/T299705 (10RLazarus) 05Open→03Resolved [20:35:43] 10serviceops, 10SRE, 10Wikimedia-Apache-configuration: Build a black-box httpd testing framework - https://phabricator.wikimedia.org/T236699 (10RLazarus) [21:27:29] so.. geoIP / Maxmind database downloads. There is GeoIP v1 and v2. and then there is "enterprise". WMF has 2 licenses. one is used for the "regular" databases that we always had on appservers. and one is use for the extra ones for the IPInfo extension. so far so confusing [21:27:59] now the "regular" ones were always a mix of both some v1 and some v2 databases and the jobs to download them were called "legacy" [21:28:37] but they downloaded both types in the same job. Now.. what happened is the v1 stuff will stop working in May. That license expired.. there will be no more v1. [21:29:30] what I changed right now is that we will stop trying to download those v1 databases on the puppetmasters..where they are ending up in the "volatile" dir [21:29:51] this is only to stop trying to pull this.. it will NOT remove any files from "volatile" or from appservers [21:30:34] we simply won't get updates anymore.. technically it still worked today but we knew it would start to cause errors again in a few weeks [21:31:11] this is https://gerrit.wikimedia.org/r/c/operations/puppet/+/773843/3/modules/puppetmaster/manifests/geoip.pp and from and for https://phabricator.wikimedia.org/T303464 [21:31:41] also we don't call it "legacy" anymore to reduce confusion. the regular job/timer is called "main [21:33:54] and the other one that is just relevant for IPInfo extension is called "ipinfo" .. and as I said.. nothing got deleted from /var/lib/puppet/volatile/GeoIP on masters and with that I will give that ticket back to analytics [21:34:07] because "Identify all users of the legacy GeoIP datasets and inform them of the need to switch to GeoIP2 dataset" I am not sure how to do [21:34:16] besides appservers [21:45:15] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download - https://phabricator.wikimedia.org/T303464 (10Dzahn) [21:55:07] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download - https://phabricator.wikimedia.org/T303464 (10Dzahn) > Modify the puppet code to no longer download the databases from MaxMind and then propagate to other servers/destinations. This is done. puppet c... [21:57:39] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464 (10Dzahn) a:05Dzahn→03None