[00:17:16] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) "log.ssh.path": "/var/log/phd/ssh.log" from: root@phab1001:/srv/phab# view phabricator/conf/local/local.json ^ this is the relevant log file... [00:22:34] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) Starting to map some users to the related tools or software: @Osnard - tool-cr-grants-team-metasync.git @Urbanecm - stewardscripts.git, tool-... [00:28:08] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) @Majavah - tool-keystone-browser.git, tool-os-deprecation.git, tool-grid-jobs.git @Rxy - stewardscripts.git @Hawkeye7 - tool-milhistbot.git... [01:52:07] 10serviceops, 10Performance-Team (Radar): Migrate WMF Production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10Legoktm) [06:58:18] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: New Kubernetes nodes may end up with no Pod IPv4 block assigned - https://phabricator.wikimedia.org/T296303 (10JMeybohm) [07:05:00] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Urbanecm) Honestly, I'd prefer having SSH access to the stewardscripts repository – it's significantly more convenient than HTTPS pull/push. I can mi... [07:07:06] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10mmodell) I believe it should be possible to host private repos on gitlab, not 100% on that. [07:15:33] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Add helmfile validation for the helmfile.d/admin part - https://phabricator.wikimedia.org/T266670 (10JMeybohm) 05Open→03Resolved [07:47:31] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: New Kubernetes nodes may end up with no Pod IPv4 block assigned - https://phabricator.wikimedia.org/T296303 (10JMeybohm) [07:48:15] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: New Kubernetes nodes may end up with no Pod IPv4 block assigned - https://phabricator.wikimedia.org/T296303 (10JMeybohm) [07:55:15] 10serviceops, 10Infrastructure-Foundations, 10netops: Kubernetes1018's eth negotiated speed is 10MB/s - https://phabricator.wikimedia.org/T296369 (10elukey) [07:55:34] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: setup/install kubernetes10[18-21] - https://phabricator.wikimedia.org/T293728 (10elukey) Opened T296369 for kubernetes1018. [08:52:39] Just a short reminder: we will start re-deploy services in codfw Kubernetes cluster soon. Feel free to ping me any time. [09:21:59] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Urbanecm) >>! In T296022#7525619, @mmodell wrote: > I believe it should be possible to host private repos on gitlab, not 100% on that. Note that it ha... [09:24:57] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10mmodell) @urbanecm: This all makes perfect sense and I don't want to make people's work more difficult. To the point of onboarding and offboarding ste... [10:11:15] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Urbanecm) >>! In T296022#7525837, @mmodell wrote: > @urbanecm: This all makes perfect sense and I don't want to make people's work more difficult. As... [10:16:11] 10serviceops, 10SRE, 10ops-eqiad: Kubernetes1018's eth negotiated speed is 10MB/s - https://phabricator.wikimedia.org/T296369 (10ayounsi) That looks like a faulty cable or interface, over to DCops for troubleshooting, let us know if you need Netops help. [11:25:20] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) [11:33:49] 10serviceops, 10MW-on-K8s, 10SRE, 10SRE Observability, 10Patch-For-Review: Make logging work for mediawiki in k8s - https://phabricator.wikimedia.org/T288851 (10Joe) After deploying the changes to php-fatal-error.php, we can now see the error messages delivered by php-wmerrors in logstash. [12:34:56] _joe_: cdanis: can one of you give the following a look and see if the comment on if the values look correct. [12:35:05] also see https://gerrit.wikimedia.org/r/c/operations/puppet/+/740828/10 which allows one to update theses value via hiera [14:08:21] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) The re-deploy of codfw was successful. Some take-aways are added here which came up in the codfw migration. The plan to migrate eqiad Kubernetes to `helm3`: * Announce maintenanc... [14:08:36] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) [15:05:15] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10mmodell) > Links to existing solutions in the Wikimedia environment would be appreciated. I'm not aware of an existing automation in particular, h... [15:55:10] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10Papaul) I received 10 out of 18 hosts. Can someone please update the racking information? Thanks [17:00:22] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Seen): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Krinkle) @Dzahn Just an idea, but if we create an alias of some... [17:03:02] 10serviceops: create a public docker-registry lvs endpoint for being used behind varnish - https://phabricator.wikimedia.org/T226642 (10JMeybohm) 05Open→03Resolved a:03JMeybohm This has been fixed [17:09:53] 10serviceops, 10Citoid, 10Observability-Logging, 10SRE, and 3 others: Citoid is logging all request / response headers as separate fields - https://phabricator.wikimedia.org/T239713 (10lmata) [17:17:19] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Seen): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) @Krinkle sure, always a good idea to replace hardcoded ho... [17:17:38] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10akosiaris) Thanks @papaul. We 'll get back to you! [18:13:15] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Seen): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Majavah) >>! In T247653#7527389, @Dzahn wrote: >> should the new... [19:33:10] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Legoktm) >>! In T296022#7526783, @mmodell wrote: > >> Links to existing solutions in the Wikimedia environment would be appreciated. > > I'm not aw... [19:38:14] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Seen): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#7527732, @Majavah wrote: > Stretch is sta... [20:50:52] 10serviceops, 10SRE, 10Patch-For-Review: Remove libvips-tools from mediawiki appservers - https://phabricator.wikimedia.org/T290802 (10Legoktm) 05Open→03Resolved a:03Legoktm [21:01:20] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Seen): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Majavah) 05Stalled→03Open I don't think this is stalled on a... [22:07:00] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Seen): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) 05Open→03Stalled It's stalled on bandwith of releng a... [22:43:50] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `gitlab-runner1001.wikimedia.or... [22:52:49] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) How to check which row has least VMs: ` [ganeti1009:~] $ for row in A B C D; do echo "row ${row}: $(sudo gnt-insta... [23:10:51] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) new MAC is: aa:00:00:99:ec:c5 new IPs are: 10.64.48.71 , 2620:0:861:107:10:64:48:71 [23:37:48] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) Hi @Jelto, I removed the VM with public IP, then re-created it as gitlab-runner1001.eqiad.wmnet with private IP in... [23:52:00] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) ` [ganeti1009:~] $ sudo gnt-instance console gitlab-runner1001.eqiad.wmnet /dev/vda1: clean, 34733/1248480 files, 38...