[09:32:31] <btullis>	 Good morning. We have just made a change to the `maintain-views.yaml` file in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1166346 so this is to be applied to clouddb* servers at your convenience.
[09:33:17] <btullis>	 It relates to T398602 - we have already deployed it to an-redacteddb1001 in order to fix a sqoop issue.
[09:33:18] <stashbot>	 T398602: Errors from refinery-sqoop-whole-mediawiki.service - July 2025 - https://phabricator.wikimedia.org/T398602
[12:26:45] <taavi>	 i might be missing something obvious here but is there no easy way to load a locally built docker image into the lima-kilo VM? and the expected workflow is to use the image built by gitlab and pushed to the toolsbeta registry?
[12:51:17] <taavi>	 T398712
[12:51:18] <stashbot>	 T398712: Cloud VPS project creation cookbook times out really often - https://phabricator.wikimedia.org/T398712
[12:54:18] <taavi>	 hmmm, the toolsbeta harbor instance has filled up its disk
[13:04:08] <taavi>	 I've silenced the alert for now, but could still use a hand if someone has an idea what to do here. I can't seem to be able to even log in to the harbor UI. should I just bump the volume size?
[13:05:52] * dhinus pagesd NFS read/writable 
[13:06:07] <dhinus>	 I'm not working today, can somebody have a look?
[13:06:31] <dhinus>	 It's already resolved 
[13:11:19] <taavi>	 dhinus: I am seeing no breakage but can keep an eye on it.
[13:22:12] <dhinus>	 thanks. it was an icinga alert, more details: toolschecker: NFS read/writeable on labs instances Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Fri Jul 4 12:59:47 UTC 2025
[13:24:08] <taavi>	 now there's a few D procs alerts, I'll reboot those
[13:26:05] <taavi>	 for harbor, created T398715
[13:26:05] <stashbot>	 T398715: toolsbeta harbor disk full - https://phabricator.wikimedia.org/T398715
[14:31:47] <dhinus>	 HarborComponentDown is still firing on  https://prometheus.svc.beta.toolforge.org/tools/alerts but I don't see it in alerts.w.o
[14:31:55] <dhinus>	 I'm on my mobile so I can't check properly 
[14:33:31] <dhinus>	 ah now I see it, I was looking at the wrong cluster. it's for toolsbeta but it's paging anyway, I think this is an issue we already tracked
[14:33:45] <dhinus>	 (the fact that toolsbeta should not page)
[14:35:48] <dhinus>	 I resolved it in victorops, I think it should not page again if it remains acked in alertmanager