[10:20:03] <dhinus>	 great work arturo andrewbogott on the cloudgw migration yesterday!
[10:39:33] <arturo>	 thanks!
[10:48:44] <dhinus>	 FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down
[10:50:54] <dhinus>	 unrelated, there is also CRITICAL slave_sql_lag Replication lag: 75742.21 seconds on clouddb1013
[10:51:04] <dhinus>	 I'm looking at the clouddb alert, it looks like the lag is going down
[11:07:42] <dhinus>	 data-persistence folks say it was caused by index rebuilds
[11:07:48] <dhinus>	 tools-k8s-worker-nfs-7 is back up
[11:11:51] <dhinus>	 ...and down again :/
[11:25:18] <dhinus>	 and back up. I'm looking at the logs, and I see some OOM errors in journalctl
[11:27:43] <dhinus>	 those could simply be k8s terminating some pods exceeding their memory limit
[11:28:12] <dhinus>	 I will ignore it for now, unless it goes down again
[11:42:31] <arturo>	 ack
[13:57:11] <andrewbogott>	 dhinus: I have nudged dc people about that recurring heat alert again. So many emails!
[13:57:41] <andrewbogott>	 ooh, there's another one
[14:25:36] <dhinus>	 andrewbogott: thanks for the nudge :) if it keeps flapping, we can silence it for a week or so
[15:24:24] <dhinus>	 ^ I added a 7-day silence for that clouddump temperature alert
[15:45:29] <andrewbogott>	 thank you!
[16:03:40] <taavi>	 arturo: you have more or less hijacked T320750 from its original purpose of enabling more features for https://wikitech.wikimedia.org/wiki/Help:Using_OpenTofu_on_Cloud_VPS users ...
[16:03:41] <stashbot>	 T320750: Support managing Cloud VPS project membership via OpenTofu - https://phabricator.wikimedia.org/T320750
[16:19:46] <arturo>	 taavi: ok, I see
[16:21:58] <arturo>	 taavi: feel free to revert, I'm in a meeting, then going to go offline
[17:14:26] <dhinus>	 taavi: I agree, I would de-couple that task from the tofu-infra task
[17:22:38] <dhinus>	 taavi: I removed the parent link and rephrased again the task description
[17:32:13] <dhinus>	 andrewbogott: I found how to modify the routing of wikitech-static alerts https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117575
[17:32:50] <dhinus>	 but I'm in two minds if it makes sense to change that if we don't know who's going to maintain it :)
[17:33:08] <andrewbogott>	 if you remove us does it become a global alert of some sort? Or a no-one alert?
[17:33:48] <andrewbogott>	 oh, I see, SRE has *
[17:33:51] <dhinus>	 yep
[17:34:05] <dhinus>	 it could be easier to ignore though...
[17:34:24] <andrewbogott>	 So does that mean it's /already/ alerting SREs/
[17:34:28] <andrewbogott>	 Or is it a first match thing
[17:34:34] <dhinus>	 first match
[17:34:49] <andrewbogott>	 so no way to make it appear on both dashboards?
[17:35:30] <andrewbogott>	 I'm happy to continue to be mostly responsible, I just want it to be visible to other SREs in case I'm not around
[17:36:08] <dhinus>	 I don't think so unfortunately, it's one or the other
[17:36:19] <dhinus>	 it's visible in icinga as well, but I'm not sure many people look there
[17:36:43] <dhinus>	 I guess we can write a runbook mentioning who to escalate to when you're not around :)
[17:37:13] <dhinus>	 there are 5 alerts defined in icinga for that host
[17:38:27] <andrewbogott>	 yeah, I guess we need to leave it as is then for now.
[17:40:50] <andrewbogott>	 thank you for looking
[17:40:56] <dhinus>	 I can write the runbook if you know the answer to "who should we escalate to" :P
[17:41:40] <dhinus>	 turns out we have a runbook, but I'm not sure it's up to date: https://wikitech.wikimedia.org/wiki/Wikitech-static
[17:42:14] <andrewbogott>	 I think the other people who have worked on it are: chris danis, clément, Daniel.  And the docs there are pretty accurate.
[17:43:00] <dhinus>	 will they have access to the new AWS environment?
[17:43:35] <andrewbogott>	 the AWS setup is a ways off, will need different monitoring anyway
[17:43:44] <andrewbogott>	 Or, at least, some of the monitoring won't apply
[17:43:46] <dhinus>	 ah ok, I think it was initially a 1:1 replacement
[17:43:54] <dhinus>	 *I thought
[17:44:02] <andrewbogott>	 I started off thinking it would be but got carried away :)
[17:44:07] <dhinus>	 haha fair enough
[17:44:28] <andrewbogott>	 since it won't run mediawiki there's no need to check sw versions, for one thing
[17:46:15] <dhinus>	 makes sense yep
[17:54:56] <dhinus>	 andrewbogott: I added a note at the top of https://wikitech.wikimedia.org/wiki/Wikitech-static, let me know if you think it's not accurate
[17:55:29] <andrewbogott>	 yep, seems good. thank you
[19:10:16] <taavi>	 sort of related to T380095: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114111
[19:10:17] <stashbot>	 T380095: Do not create DNS zones for projects outside default domain - https://phabricator.wikimedia.org/T380095
[19:14:50] <andrewbogott>	 looks good, thanks taavi