[06:08:56] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10Release-Engineering-Team (Radar): Need a service account on deploy servers for automated train pre-sync operations - https://phabricator.wikimedia.org/T303857 (10Joe) >>! In T303857#7896843, @dancy wrote: > @Joe Pinging on this t... [10:16:23] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) Thanks @JMeybohm - Those docs are really useful. I will proceed to make the changes required. There's one part that I'm not clear on from the docs. (T... [10:28:58] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) >>! In T303049#7902555, @BTullis wrote: > I understand that it's something to do with [[https://wikitech.wikimedia.org/wiki/DNS/Discovery#Read-only_an... [11:14:40] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) >>! In T303049#7902578, @JMeybohm wrote: > I was under the impression that datahub should only run/be used in the active datacenter because it relies o... [12:45:29] 10serviceops, 10Observability-Logging, 10WMF-General-or-Unknown, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Ingest logs from scheduled maintenance scripts at WMF in Logstash - https://phabricator.wikimedia.org/T285896 (10Krinkle) [13:05:36] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10akosiaris) >>! In T303049#7902665, @BTullis wrote: >>>! In T303049#7902578, @JMeybohm wrote: >> I was under the impression that datahub should only run/be used... [13:28:59] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10Ottomata) FWIW, we hope that Datahub will one day be a service for more than just analytics data, but for now, it isn't, and can be considered part of the 'anal... [13:45:57] 10serviceops, 10Observability-Logging, 10WMF-General-or-Unknown, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Ingest logs from scheduled maintenance scripts at WMF in Logstash - https://phabricator.wikimedia.org/T285896 (10Tgr) > I believe we use rsyslog for this in other places, whi... [14:29:30] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Write a cookbook to set a k8s cluster in maintenance mode - https://phabricator.wikimedia.org/T277677 (10akosiaris) [14:29:34] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Create a cookbook for depooling one or all services from one kubernetes cluster - https://phabricator.wikimedia.org/T260663 (10akosiaris) [14:43:01] 10serviceops, 10Infrastructure-Foundations, 10SRE-tools: Add a kubernetes module to spicerack - https://phabricator.wikimedia.org/T300879 (10JMeybohm) [14:50:03] Hi, does anyone know how to apply changes to changeprop and jobqueue helm values in beta? [14:50:04] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/789187 [14:50:14] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/8149dcadf8de4aba6fbf8eed9b1e44e4862dbc74/charts/changeprop#generating-beta-configurations [14:50:33] cc _joe_ or hnowlan ? [14:50:39] not really sure who to ask now that petr is gone [14:52:05] <_joe_> ottomata: beta is via puppet/hiera [14:52:20] <_joe_> but yeah hugh is probably the path of least resistance [14:53:09] figured, but the only place I see it referenced is in the helm chart values files [14:53:10] https://codesearch.wmcloud.org/search/?q=deployment-kafka-main&i=nope&files=&excludeFiles=&repos= [14:53:19] I need to replace deployment-kafka-{1,2} [14:53:26] (ignorer the eventgate ones, those nodes no longer exist) [14:53:33] there's unfortunately a manual step but I can handle that [14:54:01] ottomata: lmk when suits to deploy [14:54:10] go go go [14:54:20] the old brokers are already deleted, so job queue apprently is broken in beta [14:54:32] aha [15:15:27] 10serviceops, 10Prod-Kubernetes, 10SRE, 10Kubernetes: Convert helm releases to the new release naming schema - https://phabricator.wikimedia.org/T277849 (10akosiaris) [15:15:32] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10akosiaris) [15:19:00] ottomata: both should be updated [16:13:27] awesome, will ping the folks that pinged me :p [18:34:44] 10serviceops, 10GitLab (Infrastructure): bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) @Jelto gitlab-runner2002, gitlab-runner2003 and gitlab-runner2004 are up and running. Regarding partioning they have / and /srv logical volumes unlike 2001, the VM,... [18:35:27] 10serviceops, 10GitLab (Infrastructure): bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) I also merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/788752 when I noticed we got the 14.9 version installed. [18:47:44] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) same for gitlab-runner1002, 1003 and 1004 now: {F35103530} [18:48:43] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn)