[10:02:50] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10JMeybohm) 05Open→03Resolved [10:02:56] 10serviceops, 10MW-on-K8s, 10Scap, 10Release-Engineering-Team (Radar): Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10JMeybohm) [11:17:43] 10serviceops, 10Maps, 10Product-Infrastructure-Team-Backlog, 10User-jijiki: Investigate cache latency on tegola codfw - https://phabricator.wikimedia.org/T298251 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos It looks like things are stable for sometime now latency-wise. Moving forward we are going... [11:17:51] 10serviceops, 10Maps, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review, 10User-jijiki: Maps 2.0 roll-out plan - https://phabricator.wikimedia.org/T280767 (10Jgiannelos) [11:20:04] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Maps (Maps-data): create postgresql user for tegola service - https://phabricator.wikimedia.org/T288616 (10Jgiannelos) [12:09:53] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Cmjohnson) [12:16:19] 10serviceops, 10Maps, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: Disable unused services on maps nodes - https://phabricator.wikimedia.org/T298246 (10Jgiannelos) [13:06:50] 10serviceops, 10SRE: Provide node14 images for running production node-based services - https://phabricator.wikimedia.org/T306996 (10Esanders) > Within days of an LTS reaching EOL major nodejs libraries will be looking to remove support for it from their releases. Indeed, and many don't even wait for the EOL... [13:16:15] 10serviceops, 10SRE: Provide node14 images for running production node-based services - https://phabricator.wikimedia.org/T306996 (10MoritzMuehlenhoff) >> We can import the nodesource packages into separate repository components, e.g. thirdparty/node14 and thirdparty/node16. This way applications have the fle... [13:56:50] 10serviceops, 10SRE, 10WMF-JobQueue, 10Sustainability (Incident Followup): Videoscalers fail health checks while CPU is maxed - https://phabricator.wikimedia.org/T306860 (10akosiaris) > As a starting point: @jhathaway noted that we're running ffmpeg at niceness -19, which is quite assertive; raising that v... [13:59:29] elukey: helLloOOoooo [13:59:51] yt? seeking minikube/k8s help [14:07:29] 10serviceops: Put parse parse100[01-24] in production - https://phabricator.wikimedia.org/T307219 (10akosiaris) [14:08:51] 10serviceops: decommission wtp10[25-48] - https://phabricator.wikimedia.org/T307220 (10akosiaris) [14:10:02] ottomata: anything I can help with? [14:11:10] akosiaris: hello! [14:11:27] probably! i'm toying with knative eventing. [14:11:33] using minikube [14:11:48] running kafka in k8s locally was burning down my laptop [14:11:57] taking too much ram in the VM and the k8s apiserver would crash [14:12:13] so i am now running kafka locally on my laptop host [14:12:20] just a disclaimer, I am a total noob in knative. [14:12:28] s'ok i think my probs are all k8s now [14:12:34] i would like to connect to kafka from within minikube pods [14:12:50] i'm a k8s networking n00b [14:12:59] ottomata: https://minikube.sigs.k8s.io/docs/handbook/host-access/#hostminikubeinternal [14:13:03] does this help ? [14:13:12] i think i want the opposite [14:13:16] i want inside k8s -> outside k8s [14:13:30] yes, this is exactly that [14:13:33] pods to your laptop [14:14:30] oh.? [14:14:50] i don't think minikube added that hostname entry [14:14:51] for mme [14:15:00] at least minikube v1.10 btw [14:15:11] minikube version: v1.25.2 [14:15:50] oh [14:15:52] that is in the VM. [14:15:55] ok yes. [14:15:55] hm [14:15:57] OH [14:16:03] so that is the IP i should use ohhhh [14:16:29] yes that works [14:16:39] at least i think it should... [14:16:49] * akosiaris would pay some good money to see the stages of realization in ottomat's face in person. [14:16:56] .haha [14:18:24] akosiaris: qq, if a configmap value changes, do I have to manually recreate the pods (delete) that use it? [14:19:59] ( i think yes?) [14:22:02] that depends on how you use it, see https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically [14:22:34] hmmmmm [14:22:52] but safe bet is to roll the deployment ofc. because "will be updated eventually" ;) [14:23:06] interesting, but even if the configmap is updated, the container probably needs a restart anyway? [14:23:52] not necessarily. If the application is able to detect and hot reload a config change, that will work [14:24:05] but as said, restarting is the safer option [14:24:43] that's where we ususally have those "checksum" annotations for in deployment specs (to detect config map changes and roll the deployment) [14:25:00] ahhh nice [14:25:03] okay [15:00:23] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v6.9-Carp-On-A-Zamboni: Rotate APNS key before deploying Push Notifications to Production - https://phabricator.wikimedia.org/T288546 (10Tsevener) @Dzahn This is working for us now! Thanks so much. Can you comment o... [15:06:14] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10dancy) Thanks for the adjustments. Everything seems to be working o... [15:13:27] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Cmjohnson) >>! In T301177#7886110, @Dzahn wrote: > confirming that the "gitlab" hosts should use a public I... [15:22:43] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, and 2 others: Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Cmjohnson) [16:02:50] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab... [16:03:42] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab... [16:12:40] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, and 2 others: Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab-runner1004.eqi... [16:15:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, and 2 others: Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab1003.wikimedia.... [16:18:42] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, and 2 others: Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host gitlab1004.wikimedia.... [16:28:43] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, and 2 others: Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab-runner1002.eqiad.w... [16:31:46] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab-run... [16:34:39] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Dzahn) >>! In T301177#7891791, @Cmjohnson wrote: >>>! In T301177#7886110, @Dzahn wrote: >> confirming that... [16:37:49] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab-run... [16:41:52] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab1003... [16:44:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host gitlab1004... [16:45:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Cmjohnson) [16:46:40] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Cmjohnson) 05Open→03Resolved @Dzahn These have all been installed and resolving the task [19:37:17] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v6.9-Carp-On-A-Zamboni: Rotate APNS key before deploying Push Notifications to Production - https://phabricator.wikimedia.org/T288546 (10Dzahn) >>! In T288546#7891718, @Tsevener wrote: > @Dzahn This is working for u... [19:38:46] 10serviceops, 10SRE: Service Ops SRE support for iOS notifications update - https://phabricator.wikimedia.org/T306397 (10Dzahn) [19:38:55] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v6.9-Carp-On-A-Zamboni: Rotate APNS key before deploying Push Notifications to Production - https://phabricator.wikimedia.org/T288546 (10Dzahn) 05Open→03Resolved setting to resolved since we agree this rotation... [19:39:15] 10serviceops, 10SRE: Service Ops SRE support for iOS notifications update - https://phabricator.wikimedia.org/T306397 (10Dzahn) [19:39:30] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v6.9-Carp-On-A-Zamboni: Rotate APNS key before deploying Push Notifications to Production - https://phabricator.wikimedia.org/T288546 (10Dzahn) 05Resolved→03Open sorry, re-opening per "wait to see if @Dmantena w... [19:39:42] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v6.9-Carp-On-A-Zamboni: Rotate APNS key before deploying Push Notifications to Production - https://phabricator.wikimedia.org/T288546 (10Dzahn) a:05Tsevener→03Dmantena [19:44:15] 10serviceops, 10GitLab: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) [19:44:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Dzahn) [19:44:43] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad, 10GitLab (Infrastructure): Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Dzahn) Thank you @Cmjohnson We continue this on T307142 [19:45:09] 10serviceops, 10GitLab: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) both subtasks are resolved. we are ready to go in both DCs [19:46:23] 10serviceops, 10GitLab (Infrastructure): bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) [19:51:36] 10serviceops, 10GitLab (Infrastructure): bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) [19:56:16] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v6.9-Carp-On-A-Zamboni: Rotate APNS key before deploying Push Notifications to Production - https://phabricator.wikimedia.org/T288546 (10Dmantena) 05Open→03Resolved > It turned out it is setup in a way that [...... [19:56:24] 10serviceops, 10SRE: Service Ops SRE support for iOS notifications update - https://phabricator.wikimedia.org/T306397 (10Dmantena) [19:56:32] 10serviceops, 10Wikipedia-iOS-App-Backlog: push-notifications: follow-up task about APNS credentials - https://phabricator.wikimedia.org/T307252 (10Dzahn) [19:58:16] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v6.9-Carp-On-A-Zamboni: Rotate APNS key before deploying Push Notifications to Production - https://phabricator.wikimedia.org/T288546 (10Dzahn) Thank you as well @Dmantena @Tsevener. I made T307252 as a placeholder... [20:31:27] 10serviceops: move mw241[2-9].codfw.wmnet into production - https://phabricator.wikimedia.org/T307255 (10Dzahn) [20:31:36] 10serviceops: move mw241[2-9].codfw.wmnet into production - https://phabricator.wikimedia.org/T307255 (10Dzahn) [20:31:39] 10serviceops, 10SRE: Q1:(Need By: TBD) rack/setup/install mw241[2-9].codfw.wmnet - https://phabricator.wikimedia.org/T290192 (10Dzahn) [20:32:42] 10serviceops: move mw241[2-9].codfw.wmnet into production - https://phabricator.wikimedia.org/T307255 (10Dzahn) [20:33:47] 10serviceops: move mw241[2-9].codfw.wmnet into production - https://phabricator.wikimedia.org/T307255 (10Dzahn) roles added: https://gerrit.wikimedia.org/r/785147 conftool-data: https://gerrit.wikimedia.org/r/785918 --- after https://gerrit.wikimedia.org/r/c/operations/puppet/+/785918 the conftool-data chan... [20:34:34] 10serviceops, 10SRE: Q1:(Need By: TBD) rack/setup/install mw241[2-9].codfw.wmnet - https://phabricator.wikimedia.org/T290192 (10Dzahn) 05Open→03Resolved >>! In T290192#7886070, @Papaul wrote: > @Dzahn i think it is best to create another task for this issue and not reopen the rack/setup task. Thanks repla... [20:34:38] 10serviceops: move mw241[2-9].codfw.wmnet into production - https://phabricator.wikimedia.org/T307255 (10Dzahn) [20:34:44] 10serviceops, 10SRE: Q1:(Need By: TBD) rack/setup/install mw241[2-9].codfw.wmnet - https://phabricator.wikimedia.org/T290192 (10Dzahn) a:05Dzahn→03Papaul [20:35:21] 10serviceops: move mw241[2-9].codfw.wmnet into production - https://phabricator.wikimedia.org/T307255 (10Dzahn) mw2419 done (as jobrunner) though: https://config-master.wikimedia.org/pybal/codfw/jobrunner set to active in netbox. the others aren't