[00:03:10] (03PS1) 10Arlolra: Remove Parsoid dependency on Poem [integration/config] - 10https://gerrit.wikimedia.org/r/1006605 (https://phabricator.wikimedia.org/T354215) [00:05:00] Project beta-update-databases-eqiad build #74125: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74125/ [00:25:59] (03PS2) 10Arlolra: Remove Parsoid dependency on Poem [integration/config] - 10https://gerrit.wikimedia.org/r/1006605 (https://phabricator.wikimedia.org/T358054) [01:05:00] Project beta-update-databases-eqiad build #74126: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74126/ [02:05:00] Project beta-update-databases-eqiad build #74127: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74127/ [03:05:00] Project beta-update-databases-eqiad build #74128: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74128/ [04:05:00] Project beta-update-databases-eqiad build #74129: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74129/ [05:05:00] Project beta-update-databases-eqiad build #74130: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74130/ [06:05:00] Project beta-update-databases-eqiad build #74131: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74131/ [07:02:44] 10Continuous-Integration-Infrastructure, 10SRE, 10collaboration-services, 10vm-requests: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9579177 (10ayounsi) > cloud VPS doesn't really seem feasible to me I'm curious to know more why it doesn't ? Maybe if there are limitations t... [07:05:00] Project beta-update-databases-eqiad build #74132: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74132/ [08:49:27] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9579391 (10Ladsgroup) I had to reclone it again from db11, it's usually better to set the db to read only before doing the cloning (read only if you cloning from... [09:10:21] 10Release-Engineering-Team (Now this 🫠), 10Release, 10Train Deployments: 1.42.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T354438#9579486 (10jnuche) Noting here that https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1006601 went in `1.42.0-wmf.20`. It should fix the immediate issue: T33... [09:13:53] 10Release-Engineering-Team (Now this 🫠), 10Release, 10Train Deployments: 1.42.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T354437#9579494 (10jnuche) We have a backport available with a fix for the immediate problem: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1006313. Tim added mor... [09:16:11] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Improve deployment pipeline documentation structure - https://phabricator.wikimedia.org/T352264#9579514 (10KBach) Summary of proposed changes. Navigation (see [[ https://www.mediawiki.org/wiki/User:KBach-WMF/Sandbox/GitLab_navigation | p... [09:39:15] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9579610 (10TheresNoTime) >>! In T358329#9579391, @Ladsgroup wrote: > I had to reclone it again from db11, it's usually better to set the db to read only before d... [10:05:34] 10Release-Engineering-Team (Now this 🫠), 10Release, 10Train Deployments: 1.42.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T354438#9579677 (10jnuche) [10:05:37] 10Release-Engineering-Team (Now this 🫠), 10Release, 10Train Deployments: 1.42.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T354437#9579678 (10jnuche) [10:10:43] 10Release-Engineering-Team (Now this 🫠), 10Release, 10Train Deployments: 1.42.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T354437#9579686 (10jnuche) Backported https://gerrit.wikimedia.org/r/1006313 and tentatively removed T336504 as blocker. Myself I couldn't reproduce the issue mentio... [10:13:45] hashar: no idea why https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1006530 failed that test of yours >.< [10:42:49] (03PS1) 10Dreamy Jazz: Enable 'extension-codehealth' for GlobalBlocking [integration/config] - 10https://gerrit.wikimedia.org/r/1006886 (https://phabricator.wikimedia.org/T17294) [10:45:00] TheresNoTime: if you change the error message in the maintenance script, try also changing what error message the test expects? [10:45:45] I have not had enough coffee yet [10:45:53] but thank you, that's all it was (: [10:48:44] 10Beta-Cluster-Infrastructure, 10MediaWiki-Platform-Team, 10MediaWiki-ResourceLoader: Source map error: Error: request failed with status 404 - https://phabricator.wikimedia.org/T352167#9363154 (10AlexisJazz) >>! In T352167#9575683, @hashar wrote: >>>! In T352167#9363925, @AlexisJazz wrote: >> I use my own [... [10:49:22] * RhinosF1 sends TheresNoTime coffee [10:49:30] thank you :p [11:00:05] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262#9579839 (10KBach) [11:01:49] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262#9366470 (10KBach) I have added [[ https://wikitech.wikimedia.org/wiki/Template:Deployment_pipeline_disclaimer | this warning ]] to [[ https://wikitech... [11:03:31] 10Continuous-Integration-Config, 10Release-Engineering-Team (Now this 🫠), 10Language-Team (Language-2024-January-March), 10affects-translatewiki.net: jenkins.job=mediawiki-i18n-check-docker fails: couldnot lock config file .gitconfig: File exists - https://phabricator.wikimedia.org/T331098#9579841 (10Nikera... [11:08:24] (03CR) 10Kosta Harlan: [C: 03+1] "I've created the project at https://sonarcloud.io/project/configuration?id=mediawiki-extensions-GlobalBlocking" [integration/config] - 10https://gerrit.wikimedia.org/r/1006886 (https://phabricator.wikimedia.org/T17294) (owner: 10Dreamy Jazz) [11:14:32] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9579883 (10akosiaris) [11:16:29] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9537132 (10akosiaris) The LVS traffic approach was doomed to fail, since scap... [11:18:26] (03CR) 10Kosta Harlan: [C: 03+1] "The router was already removed, though. Or are you thinking about older release branches?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/987183 (https://phabricator.wikimedia.org/T357070) (owner: 10Umherirrender) [11:22:31] (03PS2) 10Kosta Harlan: Enable 'extension-codehealth' for GlobalBlocking [integration/config] - 10https://gerrit.wikimedia.org/r/1006886 (https://phabricator.wikimedia.org/T321837) (owner: 10Dreamy Jazz) [11:30:17] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep (beta cluster): Code stewardship request - https://phabricator.wikimedia.org/T215217#9579929 (10kostajh) [11:32:28] (03CR) 10Dreamy Jazz: "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/1006886 (https://phabricator.wikimedia.org/T321837) (owner: 10Dreamy Jazz) [11:35:22] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9579968 (10ArielGlenn) >>! In T358329#9579391, @Ladsgroup wrote: > I had to reclone it again from db11, it's usually better to set the db to read only before doi... [11:37:54] Thanks apergos: Amir1: TheresNoTime: nice job with fixing the beta cluster. Now the second part, and you will hate me for that [11:38:11] it's not quite there yet. [11:38:19] Amir is still looking at it, see the task [11:38:29] So, I’m still not done with my ticket ;) [11:39:50] The last run of `addwiki.php` still failed when tried to insert the Main_Page - so once this work is done, I’ll need to run it again. From what I see I can pass —skip-clusters=….. to skip all DB’s so I should be fine [11:40:38] But just giving you a heads up , that I’ll need to run the command again. Hopefully now it shouldn’t break things. I’ll wait till Beta Cluster is healthy [11:41:28] sounds like a plan [11:42:21] 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-brennen: Allow tools to use phabricator webhooks - https://phabricator.wikimedia.org/T321790#9579978 (10kostajh) >>! In T321790#9577659, @Dzahn wrote: >>>! In T321790#9575574, @kostajh wrote: >> Hmm. At the time that I filed this,... [11:42:42] 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-brennen: Allow tools to use phabricator webhooks - https://phabricator.wikimedia.org/T321790#9579984 (10kostajh) [11:52:17] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9580018 (10akosiaris) [11:52:37] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9537132 (10akosiaris) Migration started, we are batch 1 for the next few days. [12:09:08] let's fix the update.php first, then we can take a look at that [12:10:15] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9580035 (10Ladsgroup) Yes I agree but that is inside a public git repository (from what I'm seeing) which means it'll be confusing will be potentially removed ag... [12:35:36] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9580070 (10ArielGlenn) >>! In T358329#9580035, @Ladsgroup wrote: > Yes I agree but that is inside a public git repository (from what I'm seeing) which means it'l... [13:39:09] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9580334 (10Ladsgroup) I have a feeling I know what's going on. It's trying to run virtual domain updates in the wrong way. I need to dig deeper. [13:50:42] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/GlobalBlocking] Enable 'extension-codehealth' [integration/config] - 10https://gerrit.wikimedia.org/r/1006886 (https://phabricator.wikimedia.org/T321837) (owner: 10Dreamy Jazz) [13:51:42] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/GlobalBlocking] Enable 'extension-codehealth' [integration/config] - 10https://gerrit.wikimedia.org/r/1006886 (https://phabricator.wikimedia.org/T321837) (owner: 10Dreamy Jazz) [13:53:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/GlobalBlocking] Enable 'extension-codehealth' [integration/config] - 10https://gerrit.wikimedia.org/r/1006886 (https://phabricator.wikimedia.org/T321837) (owner: 10Dreamy Jazz) [13:54:11] !log Zuul: [mediawiki/extensions/GlobalBlocking] Enable 'extension-codehealth' for T321837 [13:54:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:54:14] T321837: Repositories integrated into Codehealth Pipeline (Production) - https://phabricator.wikimedia.org/T321837 [14:01:10] TheresNoTime: sorry I was driving this mornign [14:01:44] No worries, it was something very very simple! [14:13:13] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/debs - https://phabricator.wikimedia.org/T341991#9580431 (10herron) [14:24:15] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9580452 (10akosiaris) [14:24:30] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9580465 (10akosiaris) p:05Triage→03High [14:34:38] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9580484 (10ssastry) It doesn't show up in product... [14:57:24] !log Remove mediawiki-selenium rubygems documentation since is no more used / published and has been archived # T242293 [14:57:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:57:27] T242293: Archive mediawiki/selenium - https://phabricator.wikimedia.org/T242293 [14:59:57] 10Phabricator, 10LDAP-Access-Requests, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Migrate dev user accounts for bvibber - https://phabricator.wikimedia.org/T358044#9580692 (10ayounsi) 05Open→03Resolved Puppet and LDAP updated. > I'll just need the contents on ~brion in mwmaint2002.codfw.wmnet... [15:46:04] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, 10serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9580810 (10Lucas_Werkmeister_WMDE) [15:58:00] 10Release-Engineering-Team (Radar), 10collaboration-services: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9580854 (10Dzahn) [15:58:31] 10Continuous-Integration-Infrastructure, 10SRE, 10collaboration-services, 10vm-requests: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9580852 (10Dzahn) 05In progress→03Stalled a:05Dzahn→03None [15:59:50] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9580858 (10bd808) >>! In T358329#9580070, @ArielGlenn wrote: >>>! In T358329#9580035, @Ladsgroup wrote: >> Yes I agree but that is inside a public git repository... [16:09:27] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10serviceops-radar, 10Patch-For-Review, 10Python3-Porting: git-fat replacement/removal - https://phabricator.wikimedia.org/T279509#9580867 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/226 deploy: Fix git-l... [16:11:19] 10Release-Engineering-Team (Now this 🫠), 10Release, 10Train Deployments: 1.42.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T354437#9580869 (10thcipriani) 05Open→03Resolved >>! In T354437#9579686, @jnuche wrote: > Backported https://gerrit.wikimedia.org/r/1006313 and tentatively remov... [16:22:50] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap wikiversions-inuse and wikiversions-compile: Always use the staging directory - https://phabricator.wikimedia.org/T357581#9580910 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/43 Remove --stagi... [16:23:05] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap wikiversions-inuse and wikiversions-compile: Always use the staging directory - https://phabricator.wikimedia.org/T357581#9580916 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/43 Remove --stagi... [16:39:14] 10Continuous-Integration-Infrastructure, 10SRE, 10collaboration-services, 10vm-requests: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9580969 (10LSobanski) @ayounsi we're on a tight schedule here as we're trying to get contint off of Buster by EOL. The approach of using a Gan... [16:58:48] 10Continuous-Integration-Infrastructure, 10SRE, 10collaboration-services, 10vm-requests: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9581075 (10ayounsi) To clarify, there was no blocker in any of my comments. On the last one, I was genuinely wondering why a CloudVPS was not... [16:58:52] TheresNoTime: I am continuing tests for MediaWiki db lagging I end up needing to add a service for gethostbyaddr / gethostbyname bah :D [16:59:27] cause they can't be mocked and testing maintenance/getLagTimes.php results in call to them which takes a bit of time to resolve since the names do not exist [16:59:28] bah [16:59:58] anyway [17:00:11] 1. 5121ms to run MediaWiki\\Tests\\Maintenance\\GetLagTimesTest::testReportedOutput with data set "No lag" [17:00:13] fun [17:00:53] Tsk [17:01:07] You were meant to cure the test suite of slowness, not cause it! Etc. [17:01:18] well hmm [17:01:50] I will use `localhost` [17:05:34] solved [17:08:32] 10Continuous-Integration-Infrastructure, 10SRE, 10collaboration-services, 10vm-requests: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9581125 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1002 for hosts: `contint1003.eqiad.wmnet` - contint1003... [17:23:05] 10Continuous-Integration-Infrastructure, 10SRE, 10collaboration-services, 10vm-requests: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9581169 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1002 for hosts: `contint1004.eqiad.wmnet` - contint1004... [17:33:28] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468#9581209 (10bd808) [17:38:56] 10Continuous-Integration-Infrastructure, 10SRE, 10collaboration-services, 10vm-requests: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9581213 (10Dzahn) 05Stalled→03Open [17:38:58] 10Release-Engineering-Team (Radar), 10collaboration-services: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9581214 (10Dzahn) [17:50:59] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10collaboration-services, 10Patch-For-Review, 10User-brennen: scap install fails on new Phabricator/Phorge host due to missing user - https://phabricator.wikimedia.org/T357572#9581276 (10Dzahn) a:03Dzahn [18:14:21] * hashar old man yields at phpunit [18:22:08] Huh, I didn't know Google used Zuul as well. I thought they had a diferent CI system [18:22:12] https://gerrit-review.googlesource.com/c/gitiles/+/411137 -> https://ci.gerritcodereview.com/t/gerrit/status/change/411137,1 [18:23:20] https://ci.gerritcodereview.com/t/gerrit/stream/5163d901933e4f83b3f5c0aac47760cc?logfile=console.log [18:23:26] streaming build log with search. [18:23:32] they do [18:23:56] Maybe ut's just Chromium/Android where they use buildbot [18:25:10] they primarily use Jenkins and the Jenkins Gerrit Trigger plugin. That is maintained by Luca from Gerritforge [18:25:18] who is one of the core maintainer of Gerrit [18:25:21] https://chromium-review.googlesource.com/c/chromium/src/+/5323512?tab=checks [18:25:22] and Google relies on those build [18:25:49] Chromium seems to go directly to a differnt thing [18:25:49] but then, google developers internally use Bazel (or Blaze ?) or whatever magic they get access to [18:25:49] https://ci.chromium.org/ui/p/chromium/builders/try/android-12-x64-rel/703206/overview [18:26:13] https://ci.chromium.org/ui/ [18:26:17] Gerrit is listed there as well [18:26:38] + Google has a CI system named LUCI which is used by Chromium [18:26:52] and James E. Blair went to add Zuul to test Gerrit plugins [18:26:52] https://gerrit-review.googlesource.com/c/gerrit/+/411062/1 [18:26:54] lot of things [18:27:15] Yeah, so their default seems to be Jenkins/Zuul but for big projects with a lot of compilation targets, they use LUCI. [18:27:22] 10Continuous-Integration-Infrastructure, 10collaboration-services: allow mwmaint/cumin hosts to connect to http on contint - https://phabricator.wikimedia.org/T340788#9581362 (10Dzahn) On https://gerrit.wikimedia.org/r/c/operations/puppet/+/964881 we said that we would discuss this again in a couple weeks. Let... [18:27:29] Interesing they didn't go all the way on droppping Jenkins [18:27:34] GerritCI is Jenkins [18:27:48] and I dont think they use Zuul for gerrit itself [18:27:50] only for plugins [18:27:53] yeah [18:28:14] openstack use gerrit/zuul but they let zuul execute things without Jenkins right? They run them on a grid of containers or something. [18:28:20] more direct/basic. [18:28:23] LUCI I think that is used solely for frontend stuff, possibly cause the frontend team at Google is familiar with it [18:29:14] or maybe not. I don't know really :) [18:29:30] opendev has removed Jenkins a few years ago indeed [18:29:46] in favor of using Ansible to describe the environment and steps to run [18:30:19] and I guess that runs on one off instance / freshes container pods or whatever Nodepool supports nowadays [18:30:53] I guess it's re-assuring to see that a setup closer to ours is commonly used at Google, so that would show we don't need to make many changes at the same time, e.g. "only" upgrading Zuul could be a first step. [18:31:11] instead of being required to go to what opendev do, [18:31:20] thought it might be interesting later if it seems like a simpler thing to run [18:32:46] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Onboarding 🚀), 10Patch-For-Review: Automatically clean up unused buildkitd PersistentVolumeClaims periodically - https://phabricator.wikimedia.org/T351478#9581369 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/... [18:33:46] Krinkle: ah here some discussions about Google wiling to move from Jenkins to LUCI https://groups.google.com/g/repo-discuss/c/Js7yyRKgGF4/m/Z_QKV3HPAQAJ [18:34:32] for Zuul, I don't know. James E. Blair would know ( `corvus` on OFTC IRC network ) [18:38:50] dinner [18:39:02] then I guess I will break my mind with statsd `gauge` [18:39:12] to stop a broken replica frmo reporting its lag is 0 [18:39:17] cause that is really misleading [18:39:19] :D [18:39:26] dinnre & [18:43:18] survey: how should we deal with https://git.wikimedia.org which redirects to Phabricator and is neither gerrit NOR gitlab. [18:43:32] a) delete that, nobody uses it, just confusing [18:43:51] b) should send people to some landing page that explains what we use and link to other places [18:44:09] c) nothing, keep sending people to Diffusion and let them figure it out :P [18:47:14] also: 'gitblit.hostname' => 'git.wikimedia.org', [18:47:19] gitblit .. [18:52:14] mutante: what's gitblit [18:52:35] https://wikitech.wikimedia.org/wiki/Gitblit [18:53:10] T111465 [18:53:11] T111465: [keyresult] Deprecate gitblit in favor of Diffusion - https://phabricator.wikimedia.org/T111465 [18:53:36] 10Diffusion, 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Make https://git.wikimedia.org not redirect to Phabricator Diffusion - https://phabricator.wikimedia.org/T323073#9581419 (10Dzahn) a:03Dzahn [18:54:54] 10releng-201516-q2: [keyresult] Deprecate gitblit in favor of Diffusion - https://phabricator.wikimedia.org/T111465#9581422 (10bd808) [18:54:57] 10Diffusion, 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Make https://git.wikimedia.org not redirect to Phabricator Diffusion - https://phabricator.wikimedia.org/T323073#9581421 (10bd808) [18:55:32] RhinosF1: it's what we used before "gitiles". displays files like now at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/ http://www.gitblit.com/ [18:56:09] we still have rewrite rules from git.wikimedia.org here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1006982/1/modules/phabricator/templates/gitblit_vhost.conf.erb [18:56:56] if anyone wants to vote on this, feel free to +1/-1 here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1006982 [18:58:38] mutante: this isn't a vote for or against the change, but "repos on Diffusion have [18:58:39] been phased out." is not true. We got rid of Differential (Phab's code review) but not Diffusion (Phab's git hosting). [18:59:33] 10Diffusion, 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Make https://git.wikimedia.org not redirect to Phabricator Diffusion - https://phabricator.wikimedia.org/T323073#9581432 (10Dzahn) @hashar @Jelto Fair enough, I am not opposed to deleting it all. It would mean though tha... [19:01:50] bd808: ok, but they are all readonly and none of them are active. And Andre declared this: https://phabricator.wikimedia.org/T191182#9411317 [19:02:43] yes, that is the Differential sunset task [19:02:45] you can still pull them over https but not ssh and no pushing [19:03:30] Diffusion is used as a git mirror for things in various other code forges [19:04:25] seems a conflict with "We now do not canonically host code in Diffusion anymore " [19:04:44] "canonically" meaning as the origin repo [19:05:03] ok [19:05:09] diffusion is the closest thing we have to a mirror of everything [19:05:54] it is incomplete, but it is the only place we could work to make a complete mirror (other than an external system like GitHub) [19:08:48] *nod* [19:12:57] GitLab unfortunately does not have support for pull mirroring unless we start paying them for the closed source bits -- https://docs.gitlab.com/ee/user/project/repository/mirror/pull.html [19:13:54] And I'm confident in saying that we will never have a single code forge that hosts the origin repos of all Wikimedia movement code. [19:15:32] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Update to Phorge upstream's 2023.49 stable release - https://phabricator.wikimedia.org/T358610#9581457 (10brennen) [19:15:53] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Update wmf/stable to Phorge upstream's 2023.49 stable release - https://phabricator.wikimedia.org/T358610#9581473 (10brennen) [19:21:15] 10Phabricator, 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: Update wmf/stable to Phorge upstream's 2023.49 stable release - https://phabricator.wikimedia.org/T358610#9581488 (10CodeReviewBot) brennen opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/35 D... [19:35:12] 06Release-Engineering-Team, 06Data-Platform-SRE, 06collaboration-services, 07Security: Investigate replacing Archiva with Gitlab repositories - https://phabricator.wikimedia.org/T358612 (10Gehel) [20:03:18] 10Diffusion, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Make https://git.wikimedia.org not redirect to Phabricator Diffusion - https://phabricator.wikimedia.org/T323073#9581649 (10Jdforrester-WMF) More generally I feel we should be using descriptors not software names for sit... [20:06:36] 10Beta-Cluster-Infrastructure: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9581653 (10Ladsgroup) I ran the update with debug mode on. It required a bit of magic. I ran eval.php with `-d 3` argument And then: ` > require_once __DIR__ .... [20:09:27] 10Diffusion, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Make https://git.wikimedia.org not redirect to Phabricator Diffusion - https://phabricator.wikimedia.org/T323073#9581656 (10Dzahn) >>! In T323073#9581649, @Jdforrester-WMF wrote: > we should be using descriptors not soft... [20:26:59] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329#9581701 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup After pulling the patch and running update.php, now it runs fast: ` Purging cac... [20:32:21] !log Disabling deployment-db12 and deployment-db13 as follow-up to T358329; no longer used [20:32:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:32:25] T358329: beta-update-databases-eqiad job times out / beta databases are having issues - https://phabricator.wikimedia.org/T358329 [21:43:14] 10Diffusion, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Make https://git.wikimedia.org not redirect to Phabricator Diffusion - https://phabricator.wikimedia.org/T323073#9581854 (10Krinkle) * 1,500+ outgoing links on www.mediawiki.org: https://www.mediawiki.org/wiki/Special:Li... [22:31:31] 10Diffusion, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Make https://git.wikimedia.org not redirect to Phabricator Diffusion - https://phabricator.wikimedia.org/T323073#9581944 (10Dzahn) Fair enough, though as an example, just by substracting "User:MarkAHershberger/sandbox" a... [22:32:15] 10Continuous-Integration-Infrastructure, 06SRE, 06collaboration-services, 10vm-requests: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9581945 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host contint1003.eqiad.wmnet with OS bu... [23:34:58] (03PS1) 10Dzahn: archive wikimedia/iegreview [integration/config] - 10https://gerrit.wikimedia.org/r/1007012 (https://phabricator.wikimedia.org/T351889) [23:36:45] (03CR) 10CI reject: [V: 04-1] archive wikimedia/iegreview [integration/config] - 10https://gerrit.wikimedia.org/r/1007012 (https://phabricator.wikimedia.org/T351889) (owner: 10Dzahn) [23:43:09] (03PS1) 10Dzahn: archive ImageCompare extension [integration/config] - 10https://gerrit.wikimedia.org/r/1007015 (https://phabricator.wikimedia.org/T353968) [23:44:11] (03CR) 10CI reject: [V: 04-1] archive ImageCompare extension [integration/config] - 10https://gerrit.wikimedia.org/r/1007015 (https://phabricator.wikimedia.org/T353968) (owner: 10Dzahn) [23:45:04] (03PS1) 10Dzahn: archive MagicNumberedHeadings extension [integration/config] - 10https://gerrit.wikimedia.org/r/1007016 (https://phabricator.wikimedia.org/T353966) [23:45:55] (03PS2) 10Dzahn: archive ImageCompare extension [integration/config] - 10https://gerrit.wikimedia.org/r/1007015 (https://phabricator.wikimedia.org/T353968) [23:46:13] (03CR) 10CI reject: [V: 04-1] archive MagicNumberedHeadings extension [integration/config] - 10https://gerrit.wikimedia.org/r/1007016 (https://phabricator.wikimedia.org/T353966) (owner: 10Dzahn) [23:46:30] (03PS2) 10Dzahn: archive wikimedia/iegreview [integration/config] - 10https://gerrit.wikimedia.org/r/1007012 (https://phabricator.wikimedia.org/T351889)