[06:13:23] <wikibugs>	 06Project-Admins, 14Release-Engineering-Team (Next), 06Trusted-Contributors: Create Trusted Contributors project? - https://phabricator.wikimedia.org/T145832#10527668 (10HormigasAIS) https://github.com/HormigasAIS-ux/https-github.com-Thrumanshow-HormigasAIS/blob/main/.gitignore#L1-L171
[06:14:29] <wikibugs>	 06Project-Admins, 14Release-Engineering-Team (Next), 06Trusted-Contributors: Create Trusted Contributors project? - https://phabricator.wikimedia.org/T145832#10527670 (10HormigasAIS) {F58366460}
[06:36:30] <wmf-insecte>	 Project beta-scap-sync-world build #192631: 04FAILURE in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192631/
[06:46:42] <wmf-insecte>	 Yippee, build fixed!
[06:46:42] <wmf-insecte>	 Project beta-scap-sync-world build #192632: 09FIXED in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192632/
[08:12:22] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 2 others: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10527813 (10Jelto)
[08:54:06] <wikibugs>	 10Phabricator, 10CAS-SSO, 06Infrastructure-Foundations: Phabricator should use IDP for developer account logins - https://phabricator.wikimedia.org/T377061#10527871 (10Aklapper) Uh, thanks a lot for finding those, that's very helpful! https://secure.phabricator.com/T524#249767 also looks interesting.
[09:01:29] <wikibugs>	 10Phabricator, 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Phabricator should use IDP for developer account logins - https://phabricator.wikimedia.org/T377061#10527892 (10SLyngshede-WMF) We haven't tried to do a "pure" OAuth2 client yet, but I'm happy to setup both an OAuth2 or OIDC, which...
[09:24:55] <wikibugs>	 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10527946 (10Ottomata)
[09:34:04] <wikibugs>	 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10527963 (10Ottomata) Related:  {T367322} {T379187}  @brennen it'd be nice if we could use the repos/wmf-packages project for npm package...
[10:16:15] <wikibugs>	 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team, 06Data-Engineering: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10528084 (10Ottomata)
[11:06:36] <wikibugs>	 (03PS1) 10Zfilipin: zuul: Update Jared G. Blumer email [integration/config] - 10https://gerrit.wikimedia.org/r/1117867 (https://phabricator.wikimedia.org/T385688)
[11:10:59] <wikibugs>	 (03CR) 10Zfilipin: [C:03+2] zuul: Update Jared G. Blumer email [integration/config] - 10https://gerrit.wikimedia.org/r/1117867 (https://phabricator.wikimedia.org/T385688) (owner: 10Zfilipin)
[11:11:21] <andrewbogott>	 hey folks, I sent a cloud-announce email earlier in the week about hard rebooting VMs; the list includes many integration agents.  It looks like those are still pending reboots but I'm sure if I reboot them with a blind script it will make a (temporary) mess of CI...
[11:11:38] <andrewbogott>	 is that your preference, that I just reboot them willy-nilly? Or did that email get missed?
[11:12:09] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: Update Jared G. Blumer email [integration/config] - 10https://gerrit.wikimedia.org/r/1117867 (https://phabricator.wikimedia.org/T385688) (owner: 10Zfilipin)
[12:14:41] <wmf-insecte>	 Project beta-code-update-eqiad build #533965: 04FAILURE in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/533965/
[12:20:12] <wmf-insecte>	 Project beta-update-databases-eqiad build #82382: 04FAILURE in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82382/
[12:26:43] <wmf-insecte>	 Yippee, build fixed!
[12:26:43] <wmf-insecte>	 Project beta-code-update-eqiad build #533966: 09FIXED in 3 min 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/533966/
[12:28:31] <wmf-insecte>	 Project beta-scap-sync-world build #192665: 04FAILURE in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192665/
[12:33:54] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 2 others: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10528533 (10WMDE-leszek) Thanks for making the move, and addressing the issue that emerged. I wonder if I shoul...
[12:36:09] <wmf-insecte>	 Project beta-scap-sync-world build #192666: 04STILL FAILING in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192666/
[12:46:24] <wmf-insecte>	 Project beta-scap-sync-world build #192667: 04STILL FAILING in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192667/
[12:56:36] <wmf-insecte>	 Project beta-scap-sync-world build #192668: 04STILL FAILING in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192668/
[13:06:17] <wmf-insecte>	 Project beta-scap-sync-world build #192669: 04STILL FAILING in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192669/
[13:16:25] <wmf-insecte>	 Project beta-scap-sync-world build #192670: 04STILL FAILING in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192670/
[13:20:02] <wmf-insecte>	 Project beta-update-databases-eqiad build #82383: 04STILL FAILING in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82383/
[13:26:29] <wmf-insecte>	 Project beta-scap-sync-world build #192671: 04STILL FAILING in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192671/
[13:39:43] <wmf-insecte>	 Project beta-scap-sync-world build #192672: 04STILL FAILING in 4 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192672/
[13:46:45] <wmf-insecte>	 Project beta-scap-sync-world build #192673: 04STILL FAILING in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192673/
[13:56:27] <wmf-insecte>	 Project beta-scap-sync-world build #192674: 04STILL FAILING in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192674/
[14:04:32] <wmf-insecte>	 Project beta-code-update-eqiad build #533976: 04FAILURE in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/533976/
[14:16:31] <wmf-insecte>	 Yippee, build fixed!
[14:16:31] <wmf-insecte>	 Project beta-code-update-eqiad build #533977: 09FIXED in 3 min 31 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/533977/
[14:18:28] <wmf-insecte>	 Project beta-scap-sync-world build #192675: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192675/
[14:20:04] <wmf-insecte>	 Project beta-update-databases-eqiad build #82384: 04STILL FAILING in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82384/
[14:21:20] <wikibugs>	 10Deployments, 10MW-on-K8s, 06serviceops, 06SRE, 13Patch-For-Review: mwscript-k8s gives Invalid value: "<script>": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric c... - https://phabricator.wikimedia.org/T380533#10528799
[14:24:17] <RhinosF1>	 DB is RO
[14:26:22] <wmf-insecte>	 Project beta-scap-sync-world build #192676: 04STILL FAILING in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192676/
[14:36:05] <wmf-insecte>	 Project beta-scap-sync-world build #192677: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192677/
[14:42:25] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803 (10RhinosF1) 03NEW
[14:42:32] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10528866 (10RhinosF1) p:05Triage→03High
[14:49:03] <wmf-insecte>	 Project beta-scap-sync-world build #192678: 04STILL FAILING in 4 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192678/
[14:50:37] <jinxer-wm>	 FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError    - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[14:51:40] <jinxer-wm>	 FIRING: [2x] DatasourceNoData: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData
[14:55:36] <jinxer-wm>	 RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError    - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[14:55:52] <jinxer-wm>	 FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError    - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[14:56:28] <wmf-insecte>	 Project beta-scap-sync-world build #192679: 04STILL FAILING in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192679/
[15:00:52] <jinxer-wm>	 RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError    - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[15:01:40] <jinxer-wm>	 RESOLVED: [2x] DatasourceNoData: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData
[15:06:09] <wmf-insecte>	 Project beta-scap-sync-world build #192680: 04STILL FAILING in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192680/
[15:10:09] <wikibugs>	 10Phabricator, 06collaboration-services, 06SRE, 06Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#10528968 (10LSobanski)
[15:15:11] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10529016 (10RhinosF1) 15:14:33 <@andrewbogott> RhinosF1: ok, I can't arm keyholder on that host with the scap password or the deploy-service  password or the mwdeploy password
[15:16:17] <wmf-insecte>	 Project beta-scap-sync-world build #192681: 04STILL FAILING in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192681/
[15:16:38] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10529017 (10RhinosF1)
[15:22:49] <wikibugs>	 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Diffusion repository URIs are a mix of observing gerrit-replica vs gerrit - https://phabricator.wikimedia.org/T385128#10529062 (10LSobanski)
[15:24:36] <jinxer-wm>	 FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError    - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[15:25:02] <wikibugs>	 (03CR) 10Hslater: "Yes, this change will affect REL1_39, but we've already released its final version. ExtJSBase was removed in REL1_43, where it’s no longer" [integration/config] - 10https://gerrit.wikimedia.org/r/1115423 (owner: 10Hslater)
[15:25:31] <jinxer-wm>	 FIRING: [2x] DatasourceNoData: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData
[15:31:34] <wmf-insecte>	 Yippee, build fixed!
[15:31:34] <wmf-insecte>	 Project beta-update-databases-eqiad build #82385: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/82385/
[15:33:53] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10529120 (10RhinosF1) > 15:31:32 <wmf-insecte> Yippee, build fixed! > 15:31:33 <wmf-insecte> Project beta-update-databases-eqiad build #82385: FIXED in 11 min: https://integratio...
[15:34:45] <wmf-insecte>	 Yippee, build fixed!
[15:34:46] <wmf-insecte>	 Project beta-scap-sync-world build #192682: 09FIXED in 9 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192682/
[15:39:36] <jinxer-wm>	 RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError    - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[15:40:31] <jinxer-wm>	 RESOLVED: [2x] DatasourceNoData: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData
[15:56:03] <dancy>	 I'm taking a look at deployment-prep (T385803)
[15:56:04] <stashbot>	 T385803: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803
[15:58:15] <wikibugs>	 10Phabricator, 06collaboration-services, 06SRE, 06Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#10529263 (10DLynch) Incidentally, Kosta and I wound up making [an API proxy](https://toolhub.wikimedia.org/tools/toolforge-phabroxy) t...
[15:58:33] <RhinosF1>	 dancy: i think sessions are broken but no idea where to start beyond that
[16:04:38] <dancy>	 RhinosF1: Hmm.. that's outside of my realm of knowledge (at the moment).  
[16:05:01] <RhinosF1>	 dancy: does beta have any logs?
[16:05:10] <dancy>	 Yes
[16:05:35] <dancy>	 Lemme find that.
[16:08:22] <dancy>	 Located.  Looking around
[16:08:54] <RhinosF1>	 dancy:  https://phabricator.wikimedia.org/T383513 might be useful
[16:09:06] <RhinosF1>	 it at least tells us the data is stored is cassandra 
[16:09:19] <RhinosF1>	 on deployment-sessionstore06
[16:11:06] <dancy>	 The most prominent error in logs is related to Kafka.
[16:11:20] <dancy>	 I'll add to the current ticket.
[16:14:21] <dancy>	 ah, sessionstore errors found.  
[16:15:19] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10529383 (10bd808) Puppet is broken in a way I haven't seen before on deployment-sessionstore06.deployment-prep.eqiad1.wikimedia.cloud: `lang=shell-session root@deployment-sessio...
[16:16:24] <RhinosF1>	 dancy: can you ping deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
[16:16:27] <RhinosF1>	 or it's ip
[16:16:54] <RhinosF1>	 172.16.1.160
[16:17:15] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10529392 (10dancy) This is what I see in `objectcache.log`.  ` deployment-mediawiki13 enwiki 1.44.0-alpha objectcache WARNING: Error fetching URL "http://sessionstore.svc.deploym...
[16:17:51] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10529396 (10taavi) >>! In T385803#10529383, @bd808 wrote: > Puppet is broken in a way I haven't seen before on deployment-sessionstore06.deployment-prep.eqiad1.wikimedia.cloud: T...
[16:17:52] <dancy>	 Oooh, looks like bd808 is poking around too.  
[16:18:26] <dancy>	 And other folks more qualified than me.  That's good news.
[16:18:28] * RhinosF1 waves at bd808 
[16:20:04] <taavi>	 i'm not at all qualified for anything beta-related these days.. (even https://www.whyisbetabroken.com/ is outdated these days!)
[16:20:44] <bd808>	 !log Rebooted deployment-sessionstore06 (T385803)
[16:20:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:20:46] <stashbot>	 T385803: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803
[16:21:21] <bd808>	 oops. hopefully I didn't step on your toes dancy 
[16:21:41] <dancy>	 Nope!  I'm glad to have your assistance
[16:21:44] <RhinosF1>	 taavi: is anyone qualified for beta?
[16:22:35] <bd808>	 deployment-sessionstore06 seems a bit happier after that reboot
[16:23:09] <bd808>	 login is still not happy though
[16:23:33] <taavi>	 it is accepting my normal key now
[16:24:03] <RhinosF1>	 i can make an edit
[16:24:06] <RhinosF1>	 bd808: fine for me
[16:24:12] <RhinosF1>	 i can login and make an edit
[16:24:45] <bd808>	 me too now. I maybe just didn't wait long enough after the reboot
[16:24:55] <dancy>	 Yay!
[16:26:14] <bd808>	 My IT Crowd approach to SRE wins again!
[16:27:17] <RhinosF1>	 given someone now supposedly owns deployment-prep, is there any reason no one did these reboots themselves rather than leave it for WMCS and let fallout happen?
[16:30:46] <bd808>	 RhinosF1: Levi is my boss and he closed my ticket about deployment-prep being un-owned, but my understanding is that he closed if because we are working to find new home for beta workflows.
[16:31:06] <bd808>	 Nobody is currently clearly given time to maintain beta
[16:32:15] <RhinosF1>	 that's poor
[16:32:19] <RhinosF1>	 imo
[16:32:48] <bd808>	 WMCS is also part of the Developer Experience team that Levi said are the stewards of beta
[16:34:06] <RhinosF1>	 is it really their role to know what needs to be done when restarting parts of beta
[16:38:08] <bd808>	 I think the last time that beta had official staffing Faidon was still a member of the "labs" team
[16:39:24] <bd808>	 I am not trying to argue that this is the way the world should ideally be organized. I'm just rying to be realistic and work in the world that I have found myself.
[16:44:04] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10529523 (10bd808) 05Open→03Resolved a:03bd808 Things appear to be working now. The networking issues on deployment-sessionstore06 were "fixed" by the additional reboot.
[16:47:13] <dancy>	 RhinosF1: I think the "supposedly" part of your question is most relevant.  That ownership clearly hasn't become real yet.  But I take your point.  This particular reboot process was not handled properly and was allowed to fall through the cracks. Should have been handled better indeed.
[16:48:21] <RhinosF1>	 dancy: yup, if the "ownership" of beta is working properly, it wouldn't fall through the cracks
[16:48:48] <RhinosF1>	 cause someone would go, my team own these VMs and we don't want an uncontrolled reboot so we'll do it now
[16:50:06] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Documentation: Document post-restart activities needed across deployment-prep services - https://phabricator.wikimedia.org/T385822 (10bd808) 03NEW
[16:51:59] <wikibugs>	 10Phabricator, 06collaboration-services, 06SRE, 06Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#10529569 (10taavi) >>! In T274228#10503971, @Aklapper wrote: > Does anyone have sufficient understanding to outline the next potential...
[16:52:04] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10529573 (10bd808)
[16:52:13] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep (beta cluster): Code stewardship request - https://phabricator.wikimedia.org/T215217#10529574 (10bd808)
[17:06:04] <greg-g>	 James_F: hah, what did "Bug: TBC" break? did wikibugs fall over?
[17:06:31] <James_F>	 greg-g: ReleaseTaggerBot is fatalling on reading its e-mails.
[17:06:36] <greg-g>	 :(
[17:06:40] <greg-g>	 sorry
[17:06:49] <James_F>	 I/someone needs to go into the mail queue and delete it.
[17:06:56] * greg-g is catching up on the "fun" from last night (2am my time) where paymentswiki was broken
[17:07:01] <greg-g>	 bummer
[17:07:45] <James_F>	 Merlijn did it because he's awesome.
[17:07:57] <greg-g>	 <3
[20:53:36] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Kubernetes: Re-consider setting up a Kubernetes cluster on the Beta cluster - https://phabricator.wikimedia.org/T276650#10530225 (10bd808)
[20:53:39] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Figure out how to provision a Kubernetes cluster using Magnum and OpenTofu - https://phabricator.wikimedia.org/T372498#10530226 (10bd808)
[20:54:38] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap (SpiderPig 🕸️): Scap SpiderPig: Daemonize services for deployment hosts - https://phabricator.wikimedia.org/T383945#10530228 (10dancy) 05Open→03In progress p:05Triage→03Medium a:03dancy
[20:55:09] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 10Scap (SpiderPig 🕸️): Scap SpiderPig: Daemonize services for deployment hosts - https://phabricator.wikimedia.org/T383945#10530234 (10dancy)
[20:58:19] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Kubernetes: Re-consider setting up a Kubernetes cluster on the Beta cluster - https://phabricator.wikimedia.org/T276650#10530239 (10bd808) {T372498} resulted in https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning which currently provisions a minimal Ku...
[21:33:17] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.44.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T382366#10530324 (10thcipriani) 05Open→03Resolved
[22:24:21] <wikibugs>	 10Deployments, 10MW-on-K8s, 06serviceops, 06SRE: mwscript-k8s gives Invalid value: "<script>": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character - https://phabricator.wikimedia.org/T380533#10530441 (10RLaza...
[23:12:01] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap (SpiderPig 🕸️), 06collaboration-services: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10530757 (10dancy) >>! In T383946#10493413, @Dzahn wrote: > I think the next step needed is some envoy config, adding a sn...
[23:40:23] <bd808>	 dancy: should there be something on deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud that makes sure spiderpig is running? I think it didn't come back after the reboot.
[23:40:50] <dancy>	 There was for a while, up until today when I removed it.  
[23:41:23] <dancy>	 I've moved on to focusing on getting it to prod.
[23:41:48] <bd808>	 ok. so should we remove the https://spiderpig.wmcloud.org/ proxy?
[23:42:07] * bd808 is auditing deployment-prep things
[23:42:11] <dancy>	 Yes please.
[23:43:29] <bd808>	 !log Deleted recently orphaned spiderpig.wmcloud.org proxy after discussion with dancy 
[23:43:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[23:50:24] <bd808>	 !log Deleted beta-prometheus.wmflabs.org; it was pointed to an IP now owned by the mdwikioffline project.
[23:50:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[23:53:43] <bd808>	 !log Updated citoid-beta.wmflabs.org to point to deployment-docker-citoid02
[23:53:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL