[00:02:57] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:26:16] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [00:28:26] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 7 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [02:39:42] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [02:51:18] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 5 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [03:12:14] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [03:26:22] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 6 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [03:56:50] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [04:01:34] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 3 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [04:02:57] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [05:02:20] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [05:07:04] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 4 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [05:35:28] Krinkle: thanks, I should have checked that page first [06:45:43] zuul's rather stuck I think.. I think I'm going to restart it (which may lose a few jobs) unless someone has a better idea [07:02:00] taavi: did you restart it? it's still stuck and there are hundreds of jobs in gate-and-submit-l10n [07:07:12] Nikerabbit: not yet, I'm still poking on what's causing it to get stuck [07:07:45] ok, it doesn't seem to affect all queues [07:08:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Zuul test-prio queue backlogged, coverage queue wait now > 9hrs - https://phabricator.wikimedia.org/T315818 (10taavi) Seems like Zuul got in an infinite loop for some Puppet patches? ` taavi@contint2001 ~ $ grep "Merging for change 824836,8."... [07:11:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Zuul test-prio queue backlogged, coverage queue wait now > 9hrs - https://phabricator.wikimedia.org/T315818 (10taavi) ` taavi@contint2001 ~ $ sudo journalctl -u zuul-merger.service -f --lines 25 -- Logs begin at Sun 2022-08-14 14:31:05 UTC. -... [07:12:24] !log restart zuul-merger on contint2001 T315818 [07:12:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:12:27] T315818: Zuul test-prio queue backlogged, coverage queue wait now > 9hrs - https://phabricator.wikimedia.org/T315818 [07:14:50] restarting the merger didn't seem to help [07:17:27] !log trying to disconnect jenkins from gearman and then re-connect to see if it helps with T315818 [07:17:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:17:30] T315818: Zuul test-prio queue backlogged, coverage queue wait now > 9hrs - https://phabricator.wikimedia.org/T315818 [07:19:08] I think https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions would be the next thing to try, but I don't think I have access for that [08:02:57] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [09:29:37] (03CR) 10Jaime Nuche: [C: 03+2] "LGTM. Could see the new reporter output in train-dev." [tools/scap] - 10https://gerrit.wikimedia.org/r/824774 (https://phabricator.wikimedia.org/T302631) (owner: 10Ahmon Dancy) [09:30:00] taavi: it doesn't look to be backlogged anymore [09:34:44] (03Merged) 10jenkins-bot: Add progress reporting to php-fpm-restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/824774 (https://phabricator.wikimedia.org/T302631) (owner: 10Ahmon Dancy) [09:39:44] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [09:42:42] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [09:48:53] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.39.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T314187 (10hashar) [09:53:21] hashar: ori is also doing https://phabricator.wikimedia.org/T314868 this week [09:57:42] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [10:48:04] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.39.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T314187 (10Jdlrobson) [11:42:25] 10Project-Admins: Create subproject + milestone for Wikimedia España - https://phabricator.wikimedia.org/T315654 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project has been created: https://phabricator.wikimedia.org/project/profile/6107/ [Recommended practices for project and workboard m... [12:02:52] 10GitLab, 10serviceops, 10serviceops-collab: Configure a default cleanup policy for GitLab package registry - https://phabricator.wikimedia.org/T315877 (10Jelto) [12:04:55] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) While doing some research about GitLab disk space usage I found some projects which use significantly more disk space. Most of the... [12:22:14] 12:53:09 22 08 2022 11:53:09.080:WARN [Chrome Headless 90.0.4430.212 (Linux x86_64)]: Disconnected (0 times) , because no message in 60000 ms. [12:22:14] 12:53:09 Chrome Headless 90.0.4430.212 (Linux x86_64) ERROR [12:22:15] 12:53:09 Disconnected , because no message in 60000 ms. [12:22:19] This seems replicable on a patch... [12:29:44] It appears codesearch is ~5 days behind for MW core [12:29:57] https://codesearch.wmcloud.org/deployed/?q=selectNamedDB&i=nope&files=&excludeFiles=&repos= -> https://gerrit.wikimedia.org/g/mediawiki/core/+/c7099b957afae584dfdd4c2a0c814ca847bd4763/includes/api/ApiQueryWatchlist.php#100 [12:30:05] This not accounting for https://gerrit.wikimedia.org/r/c/mediawiki/core/+/823699 [12:30:51] Reedy: which patch? core is merging as recently as 40min ago [12:31:00] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TrustedXFF/+/824876 [12:31:54] Reedy: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-noselenium-docker/170248/artifact/log/mw-error.log/*view*/ [12:32:14] yay, buried errors [12:32:33] cheers [12:33:32] Though, that's made me notice a bigger issue... [12:33:57] Oh, wait, no it hasn't [12:37:01] there's a bit of a paradox with TrustedXFF both being a stateless singleton for hooks and an ip-specific handle for calling a method, could be solved either toward more static or toward more specific and split.. [12:37:59] actually not sure what the instance is for, it says "for tests" but tests don't appear to vary its behaviour? [12:38:07] anyway, have fun :) [12:39:09] I think it's leftovers from me hacking things around to simplify it all [14:00:57] 10Release-Engineering-Team (Doing), 10Scap, 10Documentation, 10Patch-For-Review: scap documentation is no more generated - https://phabricator.wikimedia.org/T315541 (10hashar) [14:09:12] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream: GitHub -> Phabricator import system - https://phabricator.wikimedia.org/T715 (10Krinkle) 05Open→03Declined Per T130094 [14:09:43] 10Gerrit, 10Developer-Advocacy: Implement a way to bring GitHub pull requests into gerrit - https://phabricator.wikimedia.org/T37497 (10Krinkle) [14:26:45] 10Gerrit, 10Release-Engineering-Team (Seen), 10Developer-Advocacy, 10Upstream: Gitiles should add username in git clone links, triggers "Permission denied (publickey)" error - https://phabricator.wikimedia.org/T189182 (10Krinkle) I believe the reason this appears to not affect everyone, is that: 1. If you... [14:30:05] 10Release-Engineering-Team (Radar), 10LDAP-Access-Requests, 10SRE: Grant Access to gerritadmin for junuche, demon, jhuneidi - https://phabricator.wikimedia.org/T315887 (10thcipriani) [14:30:45] 10Gerrit, 10Documentation: Gitiles does not display 
 blocks in repository descriptions - https://phabricator.wikimedia.org/T193468 (10Krinkle) 05Open→03Declined
[14:32:30] 	 10Gerrit, 10Release-Engineering-Team: Set a custom Gitiles header title - https://phabricator.wikimedia.org/T315888 (10Krinkle)
[14:32:38] 	 10Gerrit, 10Release-Engineering-Team: Enable a custom Gitiles header title - https://phabricator.wikimedia.org/T315888 (10Krinkle)
[14:32:50] 	 10Gerrit, 10Release-Engineering-Team: Enable a custom Gitiles header title - https://phabricator.wikimedia.org/T315888 (10Krinkle)
[15:06:32] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap: scap: add progress reporting to php-fpm-restarts - https://phabricator.wikimedia.org/T302631 (10dancy) 05Open→03Resolved This will go out in the next scap release.
[15:08:43] 	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Zuul test-prio queue backlogged, coverage queue wait now > 9hrs - https://phabricator.wikimedia.org/T315818 (10hashar)
[15:24:15] 	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10SRE-OnFire, 10Wikidata, and 3 others: Beta cluster Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T315350 (10Gehel) This does not seem to be related to Search / WDQS, so I'll untag the Search Platform team. Ping us aga...
[15:29:26] 	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Container image builds started failing on https://releases-jenkins.wikimedia.org/ - https://phabricator.wikimedia.org/T315897 (10dancy)
[15:29:54] 	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Container image builds started failing on https://releases-jenkins.wikimedia.org/ 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10dancy)
[15:35:35] 	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Container image builds started failing on https://releases-jenkins.wikimedia.org/ 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10TheresNoTime) 👀 https://ldap.toolforge.org/user/theresnotime is my "doppleganger" wikitec...
[15:39:17] 	 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Container image builds started failing on https://releases-jenkins.wikimedia.org/ 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10TheresNoTime) (adding @bd808 and @hashar — y'all might be interested in this & were invol...
[15:47:15] 	 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10hashar) a:05hashar→03None
[15:58:46] 	 10Continuous-Integration-Config, 10ci-test-error: wmf-quibble-vendor-mysql-php72-docker: This test uses TestCase::prophesize(), but phpspec/prophecy is not installed. - https://phabricator.wikimedia.org/T315900 (10TheresNoTime)
[15:59:36] 	 10Continuous-Integration-Config, 10ci-test-error: wmf-quibble-vendor-mysql-php72-docker: This test uses TestCase::prophesize(), but phpspec/prophecy is not installed. - https://phabricator.wikimedia.org/T315900 (10Reedy) I think that's related to a PHPUnit version change..
[16:01:10] 	 TheresNoTime: I had to do https://github.com/wikimedia/composer-merge-plugin/commit/fc70157e31d5699174f8a53635e6ac89e56f63bc for similar reasons...
[16:01:53] 	 aaaaaaaaaaaaaaaah
[16:03:24] 	 10Continuous-Integration-Config, 10ci-test-error: wmf-quibble-vendor-mysql-php72-docker: This test uses TestCase::prophesize(), but phpspec/prophecy is not installed. - https://phabricator.wikimedia.org/T315900 (10TheresNoTime)
[16:03:25] 	 my bad, I didn't search - already reported in T315892
[16:03:25] 	 T315892: PHPUnit\Framework\Exception: This test uses TestCase::prophesize(), but phpspec/prophecy is not installed. - https://phabricator.wikimedia.org/T315892
[16:04:02] 	 >phpspec/prophecy is gone and PHPUnit has increased its patch version. Why is that? o.O
[16:04:10] 	 heh, I didn't narrow it down, but guessed that was the case
[16:04:12] * Reedy facepalms
[16:07:27] 	 my mind has just blanked o.o what's the thing you comment in gerrit to re-run a failed CI run?
[16:08:06] 	 TheresNoTime: recheck
[16:08:25] 	 phew thank you, that was bugging me
[16:08:55] * TheresNoTime was just about to try things like "retry" / "again" etc until it worked :)
[16:09:29] 	 I don't know that the word and when/why to use it is actually documented anywhere on mw.o...
[16:09:52] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️): Ensure phab1002/phab2002 are scap3 targets - https://phabricator.wikimedia.org/T313954 (10brennen) a:03brennen
[16:10:00] 	 a search of "retry jenkins" didn't find it fwiw :D
[16:10:15] 	 it's mentioned on https://www.mediawiki.org/wiki/Continuous_integration/Allow_list
[16:12:11] 	 ah yes that makes sense o.O I've added a note to https://www.mediawiki.org/wiki/Continuous_integration/Jenkins too now :P
[16:13:32] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Release, 10Train Deployments: 1.39.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T314187 (10thcipriani)
[16:13:39] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10Patch-For-Review: Scap3-ify Phabricator - https://phabricator.wikimedia.org/T313953 (10brennen)
[16:13:48] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Release, 10Train Deployments: 1.39.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T314188 (10thcipriani)
[16:14:37] 	 10Release-Engineering-Team, 10Scap: scap backport: Reduce output when waiting for mediawiki/core (or extensions) commits - https://phabricator.wikimedia.org/T315901 (10dancy)
[16:14:44] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️): Configure keyholder on devtools deploy host for phabricator deployment - https://phabricator.wikimedia.org/T314195 (10brennen) 05In progress→03Resolved
[16:14:48] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10brennen)
[16:15:11] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap: scap backport: Reduce output when waiting for mediawiki/core (or extensions) commits - https://phabricator.wikimedia.org/T315901 (10dancy)
[16:15:40] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10thcipriani) a:03thcipriani
[16:15:50] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10hashar) This depends on whether we stick on `git-fat` (in which case we might need to do the porting, and even it is not immediately needed si...
[16:17:10] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10demon) a:03demon
[16:17:31] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap: scap backport: Reduce output when waiting for mediawiki/core (or extensions) commits - https://phabricator.wikimedia.org/T315901 (10taavi) I like having some feedback that it's still waiting and hasn't died somehow. Maybe a spinner or a "time elapsed" counte...
[16:17:39] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10MoritzMuehlenhoff) >>! In T279509#8174957, @hashar wrote: > This depends on whether we stick on `git-fat` (in which case we might need to do t...
[16:17:54] 	 10Release-Engineering-Team (Bonus Level 🕹️): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10demon) a:03demon
[16:21:42] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10hashar) > Bullseye doesn't ship Python 2.7 in a supported version, it's only included to _build_ a few packages (e.g. qtwebkit).  **Oops** my...
[16:24:01] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap, 10MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), 10Patch-For-Review, and 2 others: Localisation cache must be purged after or during train deploy, not (just) before - https://phabricator.wikimedia.org/T263872 (10dancy) a:03dancy
[16:24:16] 	 10Release-Engineering-Team (Bonus Level 🕹️): Weekly train branch cut job should wait until Jenkins has merged the mediawiki/core branch commit - https://phabricator.wikimedia.org/T315452 (10dancy) a:03dancy
[16:44:51] 	 10Scap, 10MediaWiki-ResourceLoader, 10MediaWiki-extensions-WikimediaMaintenance, 10Performance-Team, 10Technical-Debt: Remove old refreshMessageBlobs.php script from WikimediaMaintenance - https://phabricator.wikimedia.org/T314947 (10Krinkle)
[16:45:03] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Scap, 10MW-1.37-notes (1.37.0-wmf.3; 2021-04-27), 10Patch-For-Review, and 2 others: Localisation cache must be purged after or during train deploy, not (just) before - https://phabricator.wikimedia.org/T263872 (10Krinkle)
[17:00:22] 	 hashar: I am taking inspiration from you with 
[17:00:24] 	 https://github.com/miraheze/puppet/pull/2788
[17:09:03] 	 hashar: what would it take to increase job output retention a bit? Last few weeks I'm spending a signifcant portion of my days rechecking people's patches and waiting 30min for changes that only ran a few days ago.
[17:09:17] 	 Is fresnel still the bottleneck? If so, we can keep that where it is.
[17:09:30] 	 Or look at what else stands out and tweak that at the benefit of all else
[17:17:44] 	 RhinosF1: James from a javascript one (requiring Grunt) : https://github.com/wikimedia/grunt-tyops
[17:18:03] 	 RhinosF1: and I think that `grep` can surely be optimized, but it is surely good enough as is. +1!
[17:18:45] 	 Krinkle: the global default retention is 30 days, but for a bunch of high traffic / heavy payload jobs it is reduced to 15 / 7 days
[17:19:05] 	 Krinkle: maybe the mediawiki* jobs are lighter nowadays and we can keep a bit more of them
[17:19:36] 	 it is driven by the disk space available on the Jenkins primary
[17:20:33] 	 https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=12&orgId=1&var-server=contint2001&var-datasource=thanos&var-cluster=ci&from=now-180d&to=now
[17:20:54] * Krinkle was on the same dash, checking 2002
[17:20:56] 	 I guess it's 2001
[17:21:17] 	 someone had the idea of having different retention for failing vs success builds
[17:21:34] 	 which is probably not that complicated to implement but requires some java / Jenkins knowledge
[17:21:45] 	 that sounds complicated to me :)
[17:21:51] 	 so we're at 50% for / and 25% for /src
[17:21:54] 	 srv*
[17:21:58] 	 these are on srv* right?
[17:22:09] 	 yes
[17:22:10] 	 simple jobs like mediawiki*composertest currently seem to be gone from Aug 8
[17:22:14] 	 which is less than 30 days
[17:22:20] 	 I'd like to bump those to start with
[17:22:32] 	 can I just do that, or what should I ask/follow/check
[17:22:45] 	 Filesystem            Size  Used Avail Use% Mounted on
[17:22:45] 	 /dev/mapper/vg0-srv   658G  147G  478G  24% /srv
[17:23:00] 	 and 4.4M inodes used out of 42M
[17:23:01] 	 also, we should put a 2TB disk in there like on various other hosts that store lots of data
[17:23:06] 	 should cost less than my keyboard
[17:23:44] 	 it has a 1TB disk
[17:23:55] 	 which still has extra 185G free
[17:23:58] 	 so we can surely grow it
[17:24:18] 	 if you are talking about contint machines, now is the right time to request different hardware:
[17:24:21] 	 https://phabricator.wikimedia.org/T294276
[17:24:28] 	 do it quick before they are replaced 
[17:24:47] 	 otherwise "add new disk into existing machine" will be more of an issue than it may seem
[17:25:04] 	 I am off for dinner
[17:25:07] 	 mutante: what's the largest reasonable disk size that won't raise eye brows if antoine or me says we need it to avoid wasting developer time re-running the same jobs to see what happened?
[17:26:00] 	 Krinkle: eye brows will be risen just based on a request to add disks into an existing machine instead of ordering a standard machine from the dcops-selection menu
[17:26:09] 	 that's why I say it's the perfect time
[17:26:09] 	 we could probably use small SSD on those boxes as well
[17:29:24] 	 mutante: ack, I'm referring to the new req
[17:29:41] 	 I assume if I edit the task and write down 200TB, someone will ask difficult questions
[17:29:45] 	 I assume 1 or 2TB won't raise questions
[17:29:48] 	 wha's the max on that?
[17:30:14] 	 Krinkle: give me a moment, I am trying to get the list of possible options
[17:30:20] 	 in other words, what's a reasonable ask without self descriminating with 10y old numbers
[17:30:39] 	 we probably can't even get less than 1TB these days
[17:35:43] 	 mutante: according to the tickets, 2 servers were ordered, to become/replace gerrit2001 and contint2001 and it looks like they have been received already
[17:35:56] 	 https://phabricator.wikimedia.org/T299575#7715373
[17:36:54] 	 hashar: anyway, so 24% use - what can I do to increase it, or is it declined until we have more margins/larger disk?
[17:37:47] 	 I'm confused how 24% use translates to 150G of 960G free.
[17:38:42] 	 ah its 660G not 1000G, and it has 478G free, not 185G
[17:39:00] 	 I guess some of it is partioned away or something?
[17:39:55] 	 Krinkle: I just let Google send you a link to a spreadsheet that is visible to all WMF staff
[17:40:09] 	 it's called "upcoming procurement"
[17:40:32] 	 in there click the "Standard HW Configs" link and there are the options "config A" through "config J"
[17:40:44] 	 the largest is 12 x 2 TB HDD ;p
[17:41:23] 	 and yes, some percentage will be kept as "free extents" or spare to be able to resize 
[17:41:32] 	 at least with most partman recipes
[17:42:06] 	 yes, gerrit2002 is already in use. the contint is not
[17:49:24] 	 meanwhile found a better way to share it than a google doc:
[17:49:26] 	 https://netbox.wikimedia.org/dcim/device-types/?q=Config&manufacturer_id=4
[17:50:12] 	 there are only supposed to be these config options anymore
[17:58:11] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10dduvall) @brennen and I were just discussing the issue around group ownership of phab's config files and scap deployment. Seems li...
[18:06:08] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10Dzahn) > ..the finalize script that puppet manages and makes sudo-able...  > ..After successfully deploying a new rev, scap needs...
[18:06:21] 	 Krinkle: you can bump the jobs retention yes
[18:06:56] 	 if unsure send a dummy jjb change and I will amend as needed tomorrow
[18:07:52] 	 looks like since May there is less disk pressure, which might be the Junit XML files no more being collected
[18:08:05] 	 and/or docker images being purged on a schedule
[18:08:38] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10Dzahn) I think I would prefer to have a separate cleanup script that can be managed the same way the finalize script is managed ra...
[18:10:28] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️): Ensure phab1004/phab2002 are scap3 targets - https://phabricator.wikimedia.org/T313954 (10Dzahn)
[18:11:56] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️): Ensure phab1004/phab2002 are scap3 targets - https://phabricator.wikimedia.org/T313954 (10Dzahn) Since changes last Friday (https://gerrit.wikimedia.org/r/q/topic:phab2002) allowed https://gerrit.wikimedia.org/r/c/operations/puppet/+/823636   phab2002...
[18:12:54] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️): Ensure phab1004/phab2002 are scap3 targets - https://phabricator.wikimedia.org/T313954 (10Dzahn) ` [phab2002:~] $ which scap /usr/bin/scap `
[18:38:35] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10dduvall) >>! In T313950#8175409, @Dzahn wrote: >> ..the finalize script that puppet manages and makes sudo-able...  >> ..After suc...
[18:53:45] 	 10Release-Engineering-Team (Bonus Level 🕹️), 10Release, 10Train Deployments: 1.39.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T314187 (10matmarex)
[19:19:25] 	 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10Patch-For-Review: Move Phabricator configuration into deployment repo - https://phabricator.wikimedia.org/T313950 (10brennen) > We might be able to modify scap to expose the old rev paths as an environment variable, or we could implement logic in...
[22:00:59] 	 10Gerrit, 10Release-Engineering-Team: Enable a custom Gitiles header title - https://phabricator.wikimedia.org/T315888 (10hashar) That might be the config setting `customVariant` which has lead me to https://gerrit-review.googlesource.com/c/gitiles/+/76230  It also refers to > `customHeadTagPart`  -> at the en...
[22:02:46] 	 10Release-Engineering-Team (Radar), 10LDAP-Access-Requests, 10SRE: Grant Access to gerritadmin for junuche, demon, jhuneidi - https://phabricator.wikimedia.org/T315887 (10Dzahn) also see T273164
[22:07:03] 	 10Gerrit, 10Release-Engineering-Team: Enable a custom Gitiles header title - https://phabricator.wikimedia.org/T315888 (10hashar) https://gerrit-review.googlesource.com/c/gitiles/+/76230 originates from a thread on upstream mailing list https://groups.google.com/g/repo-discuss/c/4rZj21voAGM which has more deta...