[00:17:14] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) "log.ssh.path": "/var/log/phd/ssh.log" from: root@phab1001:/srv/phab# view phabricator/conf/local/local.json ^ this is the relevant log file... [00:22:32] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) Starting to map some users to the related tools or software: @Osnard - tool-cr-grants-team-metasync.git @Urbanecm - stewardscripts.git, tool-... [00:28:06] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) @Majavah - tool-keystone-browser.git, tool-os-deprecation.git, tool-grid-jobs.git @Rxy - stewardscripts.git @Hawkeye7 - tool-milhistbot.git... [02:24:41] (03PS1) 10Legoktm: Configure CI for labs/tools/rust-hello-world [integration/config] - 10https://gerrit.wikimedia.org/r/740956 [04:42:57] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops-radar, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Pchelolo) >>! In T296331#7525107, @Legoktm wrote: > We will need to upload packages of php-yaml to ou... [07:04:58] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Urbanecm) Honestly, I'd prefer having SSH access to the stewardscripts repository – it's significantly more convenient than HTTPS pull/push. I can mi... [07:07:04] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10mmodell) I believe it should be possible to host private repos on gitlab, not 100% on that. [09:21:57] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Urbanecm) >>! In T296022#7525619, @mmodell wrote: > I believe it should be possible to host private repos on gitlab, not 100% on that. Note that it ha... [09:24:55] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10mmodell) @urbanecm: This all makes perfect sense and I don't want to make people's work more difficult. To the point of onboarding and offboarding ste... [09:55:31] (03CR) 10Hashar: [C: 03+2] Configure CI for labs/tools/rust-hello-world [integration/config] - 10https://gerrit.wikimedia.org/r/740956 (owner: 10Legoktm) [09:57:23] (03Merged) 10jenkins-bot: Configure CI for labs/tools/rust-hello-world [integration/config] - 10https://gerrit.wikimedia.org/r/740956 (owner: 10Legoktm) [10:11:12] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Urbanecm) >>! In T296022#7525837, @mmodell wrote: > @urbanecm: This all makes perfect sense and I don't want to make people's work more difficult. As... [12:16:30] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: Generated Data Platform - https://phabricator.wikimedia.org/T296381 (10gmodena) [13:54:41] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [14:02:52] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:04:41] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [14:07:22] ah [14:07:27] thank you jinxer-wm [14:08:42] 10Quibble, 10MediaWiki-Core-Tests, 10User-kostajh: Quibble runs core:unit tests twice! - https://phabricator.wikimedia.org/T255792 (10kostajh) Is this still a problem, or a problem worth doing something about? [14:09:30] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:24:00] (03CR) 10Kosta Harlan: dockerfiles: Use opcache optimizations with built-in PHP server (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/738281 (owner: 10Kosta Harlan) [14:24:02] (03PS2) 10Kosta Harlan: dockerfiles: Use opcache optimizations with built-in PHP server [integration/config] - 10https://gerrit.wikimedia.org/r/738281 [14:24:14] (03CR) 10Kosta Harlan: dockerfiles: Use opcache optimizations with built-in PHP server (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/738281 (owner: 10Kosta Harlan) [14:24:41] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [14:27:04] that was transient [14:27:28] (03CR) 10Kosta Harlan: dockerfiles: Use opcache optimizations with built-in PHP server (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/738281 (owner: 10Kosta Harlan) [14:27:36] (03PS2) 10Kosta Harlan: jjb: Switch integration-quibble-fullrun to 1.2.0-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/738282 [14:28:37] kostajh: I had a very quick look at those php opcache settings, maybe we can add some of the service ops people to help tune them? [14:28:45] I would expect them to be familiar with php opcache config [14:29:07] sure, that could be useful [14:30:26] does it improves performance on your local machine? [15:05:13] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10mmodell) > Links to existing solutions in the Wikimedia environment would be appreciated. I'm not aware of an existing automation in particular, h... [15:12:19] (03CR) 10Kosta Harlan: [C: 04-1] "I can't build this locally, but I'll test out in the integration/quibble repo." [integration/config] - 10https://gerrit.wikimedia.org/r/738281 (owner: 10Kosta Harlan) [15:25:51] 10Continuous-Integration-Config, 10Toolforge Build Service, 10Cloud-Services-Origin-Team, 10User-dcaro, 10cloud-services-team (Kanban): Set up CI for cloud/toolforge/buildpacks repository - https://phabricator.wikimedia.org/T265685 (10dcaro) [15:26:40] 10Continuous-Integration-Config, 10Toolforge Build Service, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Project, and 2 others: Set up CI for cloud/toolforge/buildpacks repository - https://phabricator.wikimedia.org/T265685 (10dcaro) [15:29:52] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: Generated Data Platform - https://phabricator.wikimedia.org/T296381 (10gmodena) [16:12:15] (03CR) 10Daimona Eaytoy: [C: 03+1] Zuul: [extensions/WikiEditor] Add ConfirmEdit as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/740916 (https://phabricator.wikimedia.org/T296287) (owner: 10Umherirrender) [16:59:32] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Patch-For-Review: Create a Beta Cluster version of Wikifunctions.org - https://phabricator.wikimedia.org/T284162 (10Jdforrester-WMF) https://wikifunctions.beta.wmflabs.org/ now routes (to a "no wiki yet configured" page)! [17:00:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Krinkle) @Dzahn Just an idea, but if we create an alias of some... [17:01:29] kostajh: does that docker pull 404 error affect all local builds, or is that something you see for the first time with just this one? [17:01:59] \cc hashar https://gerrit.wikimedia.org/r/c/integration/config/+/738281/comment/5d87e0de_9f7444dd/ [17:02:09] o/ [17:02:48] hmm maybe wikimedia-stretch got dropped? [17:15:28] (03CR) 10Hashar: dockerfiles: Use opcache optimizations with built-in PHP server (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/738281 (owner: 10Kosta Harlan) [17:17:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) @Krinkle sure, always a good idea to replace hardcoded ho... [17:24:32] hashar: thx, James_F: nice catch on the docker-pkg bisect! [17:24:45] hashar: btw, can you +1 https://gerrit.wikimedia.org/r/650306 if my understanding is correct? [17:24:49] i think it is because the docker package got updated [17:24:56] and something is missing on our machines [17:25:25] I'm not proposing a switch right now, we can take it easy, but this seems like a small and easy next step [17:25:25] it is abandoned? [17:25:40] yeah since lars dropped it and six months passed idle on the ticket [17:25:56] yeah we missed the time to do the doc.wm.O training :-\ [17:26:06] are those hosts even existing? [17:28:54] restored, rebsaed, +1ed [17:28:57] somehow my pcc job for https://gerrit.wikimedia.org/r/c/operations/puppet/+/741713/ is "queued" even through all the compilers are idle [17:34:20] 10Release-Engineering-Team (Doing), 10MediaWiki-Core-Tests, 10MediaWiki-ResourceLoader, 10Performance-Team, 10Patch-For-Review: Move bundlesize test from npm script to MediaWikiIntegrationTest - https://phabricator.wikimedia.org/T255149 (10Mainframe98) [17:39:01] * majavah patiently waits for the 0 jobs in queue in front of him [17:39:17] majavah: I guess there is a lock somewhere but I cant find anything obvious :( [17:40:35] now finally running [17:42:23] it kept doing things such as zuul.IndependentPipelineManager: Checking for changes needed by [17:44:09] nop unrelated [17:44:14] well Idon't know what went wrong [17:44:32] I am off for dinner. Be back later tonight [18:13:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Majavah) >>! In T247653#7527389, @Dzahn wrote: >> should the new... [18:13:50] thanks hashar [18:18:16] Krinkle: Git bisect to blame changes is such fun. [18:19:28] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: Generated Data Platform - https://phabricator.wikimedia.org/T296381 (10gmodena) [19:33:08] 10Phabricator, 10Release-Engineering-Team, 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Legoktm) >>! In T296022#7526783, @mmodell wrote: > >> Links to existing solutions in the Wikimedia environment would be appreciated. > > I'm not aw... [19:36:58] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10Legoktm) [19:38:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#7527732, @Majavah wrote: > Stretch is sta... [19:42:09] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10Krinkle) [21:01:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Majavah) 05Stalled→03Open I don't think this is stalled on a... [21:23:56] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T293952 (10Legoktm) [22:06:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) 05Open→03Stalled It's stalled on bandwith of releng a... [22:07:49] mutante: I apologize for that doc replacement :( [22:08:30] mutante: I guess we need to allocate some time to do all the hardware replacement at the same time (contint machines are old, those doc vm, we also need to rebuild all the CI agents and do their stretch > bullseye migration) [22:08:36] it is a bit slow [22:08:57] majavah: that task is indeed stalled. oout of time [22:09:36] majavah: it still has to be done, but as long as the system fulfill its current service the replacement is not really a priority [22:43:48] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `gitlab-runner1... [22:44:37] well it has been a long day. Have a good afternoon! [22:52:48] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) How to check which row has least VMs: ` [ganeti1009:~] $ for row in A B C D; do echo "row ${row}:... [23:10:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) new MAC is: aa:00:00:99:ec:c5 new IPs are: 10.64.48.71 , 2620:0:861:107:10:64:48:71 [23:13:38] (03Abandoned) 10Legoktm: Edit Repo Config [libs/ObjectFactory] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/740874 (owner: 10Ksdev) [23:37:46] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) Hi @Jelto, I removed the VM with public IP, then re-created it as gitlab-runner1001.eqiad.wmnet wit... [23:51:58] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Dzahn) ` [ganeti1009:~] $ sudo gnt-instance console gitlab-runner1001.eqiad.wmnet /dev/vda1: clean, 34733/1...