[14:42:21] <nemo-yiannis>	 👋 Heads up, this is blocking our release process: https://phabricator.wikimedia.org/T431838
[14:54:32] <swfrench-wmf>	 nemo-yiannis: just to confirm, your release process requires running the RT test from your local machine, rather than testreduce, and this required x-wikimedia-debug to function?
[14:54:53] <swfrench-wmf>	 s/this required/thus requires/
[14:55:16] <nemo-yiannis>	 The testreduce is running in cloud VPS
[14:55:59] <nemo-yiannis>	 The ticket has a single command to replicate one of the ~150k tickets of the testsuite
[14:56:55] <nemo-yiannis>	 Even on testreduce we use `x-wikimedia-debug`
[14:57:31] <swfrench-wmf>	 nemo-yiannis: sorry to be unclear. what I mean is, did this always use x-wikimedia-debug, or is that somehow an artifact of the switch to using mw-parsoid?
[14:57:51] <swfrench-wmf>	 basically, back when parsoidtest1001 was the target, x-wm-d should never have been involved
[14:57:54] <nemo-yiannis>	 This is part of the new setup using mw-parsoid
[14:58:01] <nemo-yiannis>	 Before it was only internal traffic
[14:58:21] <swfrench-wmf>	 can you explain what you mean by "Before it was only internal traffic" ?
[14:58:30] <swfrench-wmf>	 did you move the location of the client?
[14:59:58] <nemo-yiannis>	 Yes, before we migrated to mw-parsoid and using x-wikimedia-debug headers to reach the rest.php endpoints,  we had a dedicated machine for mediawiki (parsoidtestXXXX) and a dedicated machine for running our testsuite (testreduceXXX)
[15:03:24] <swfrench-wmf>	 got it, and so as part of the new process, it's not just the backend that's changed (parsoidtest1001 -> mw-parsoid), you also changed the client from testreduce to a Cloud VPS that - for example - cannot possibly resolve a hostname like "mw-parsoid.eqiad.wmnet" ?
[15:03:42] <nemo-yiannis>	 yes
[15:04:40] <swfrench-wmf>	 many thanks for your patience with all of my questions - just trying to understand how this all has changed so that the traffic is now external :)
[15:09:31] <swfrench-wmf>	 unless you have some mechanism to (1) temporarily fall back to testreduce and (2) force the RT test to target mw-parsoid.eqiad.wmnet instead of parsoidtest1001, then I think the only way we can unblock you is to roll back https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306905.
[15:10:57] <swfrench-wmf>	 I don't think claime is around at the moment, but I think this should be trivially safe to revert - i.e., it just re-introduces certain classes of broken-ness for _other_ x-wikimedia-debug use cases
[15:12:45] <nemo-yiannis>	 I don't think we can fallback to the old env because the last run was in the new env (cloudvps + mw-parsoid) and every time we need prepare a new release we need to compare with the previous known run
[15:14:13] <nemo-yiannis>	 *we need to prepare
[15:14:33] <swfrench-wmf>	 got it, so the state you would compare to is only available on the VPS
[15:14:36] <nemo-yiannis>	 yes
[15:15:45] <swfrench-wmf>	 one last question: this can't wait for early (i.e., Europe morning) on Monday, correct? I just want to confirm before pursing a Friday ATS change
[15:16:01] <nemo-yiannis>	 I think it can wait until Monday (~noon)
[15:16:20] <nemo-yiannis>	 we sometimes run rt-test on Mondays (late EU friendly tz)
[15:17:35] <swfrench-wmf>	 ah, that is very good to know! the reason I ask isn't just the Friday thing - we're having trouble with one of the puppetservers at the moment, which means that a controlled rollout of this (actuated via puppet) gets kind of messy
[15:18:23] <nemo-yiannis>	 yeah, i think it can wait until early monday
[15:19:13] <swfrench-wmf>	 amazing, thank you - I'll follow up on the various tasks
[15:19:51] <nemo-yiannis>	 👍 thanks for the help