[07:05:39] * dhinus paged HarborComponentDown tools
[07:08:33] <dhinus>	 alert text: "Got no data for any component, all might be down or unresponsive, toolforge might be unable to pull images"
[07:14:27] <blancadesal>	 hmm, the harbor ui ended up loading for me after a while, but extremely slowly
[07:15:08] <blancadesal>	 trying to log in, it says "Core service is not available."
[07:15:58] <blancadesal>	 there seems to be a problem with trove
[07:16:17] <blancadesal>	 https://www.irccloud.com/pastebin/a7tzSKG5/
[07:16:48] <dhinus>	 I can try restarting the trove db
[07:16:50] <blancadesal>	 (from the harbor-core logs)
[07:17:03] <blancadesal>	 didn't andrew do something on trove recently?
[07:18:03] <dhinus>	 I think he was planning to upgrade the harbor instance next Monday :)
[07:18:48] <dhinus>	 ok there are some errors in the logs that can be seen from horizon
[07:18:58] <dhinus>	 project: tools, database->instances
[07:19:39] <dhinus>	 "in create_backup raise exception.TroveError"
[07:19:58] <dhinus>	 I tried "restart instance"
[07:20:04] <dhinus>	 "Unable to restart"
[07:20:33] <blancadesal>	 not possible to ssh to the trove host either
[07:20:45] <dhinus>	 I'm logging in via ssh
[07:20:55] <dhinus>	 did it fail for you?
[07:21:17] <dhinus>	 you need a special key
[07:21:29] <dhinus>	 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Trove#Accessing_Trove_guest_VMs
[07:21:40] <blancadesal>	 ah, that might be it
[07:23:08] <dhinus>	 hmm it's not connecting, just hanging
[07:27:25] <dhinus>	 I cannot connect to any trove instances apparently
[07:29:02] <dhinus>	 I will try opening a virtsh console from the cloudvirt host
[07:29:05] <dhinus>	 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Root_console_access
[07:31:16] <dhinus>	 I can connect but it asks for a password
[07:31:27] <dhinus>	 oh I know why I cannot connect from ssh, I need to enable the security group
[07:31:58] <dhinus>	 added "ssh-from-anywhere" to the instance tools-harbordb in project trove
[07:34:13] <dhinus>	 ok I'm in the instance
[07:34:35] <dhinus>	 the docker container is in a restart loop
[07:35:04] <dhinus>	 "could not write to file "pg_wal/xlogtemp.12": No space left on device"
[07:35:31] <dhinus>	 the cinder volume /dev/sdb is full
[07:36:46] <dhinus>	 it's the postgres data dir so there's nothing we can delete easily
[07:36:54] <dhinus>	 we need to enlarge the volume I think
[07:38:24] <blancadesal>	 can it be resized via horizon?
[07:39:11] <dhinus>	 maybe? I never did it but I know it's possible :)
[07:39:54] <dhinus>	 I stopped the container and unmounted the volume
[07:40:07] <dhinus>	 https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances#Extend_a_volume
[07:40:38] <dhinus>	 this is also a Trove instance which might complicate things... because Trove is managing the volume
[07:42:30] <blancadesal>	 there's some additional info here https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Trove&section=10#Instance_is_down
[07:42:40] <dhinus>	 hmm the alert resolved by itself at 9:39 but Harbor is still down
[07:44:09] <dhinus>	 thanks for that link! there's a "resize volume" in the database->instance section of horizon
[07:44:47] <dhinus>	 re-mounted the volume and I'll try that one, I guess Trove will take care of umounting/mounting
[07:44:56] <dhinus>	 resizing from 8 to 16
[07:45:11] <dhinus>	 "Quota exceeded for resources: ['volumes']" LOL
[07:46:42] <blancadesal>	 oh no xd
[07:46:48] <dcaro>	 morning
[07:47:03] <dhinus>	 there seems to be plenty of quota though
[07:47:15] <dcaro>	 in the trove project?
[07:47:17] <dhinus>	 dcaro: o/
[07:47:21] <dcaro>	 (/me reading backlog)
[07:47:21] <dhinus>	 I checked both tools and trove
[07:47:25] <dcaro>	 ack
[07:48:36] <blancadesal>	 dcaro: morning
[07:49:31] <dcaro>	 so tools harbor db is having disk space issues and thus harbor is down in tools?
[07:50:06] <blancadesal>	 yes
[07:50:15] <dcaro>	 anyone is acting as IC? I'll do if not
[07:51:03] <blancadesal>	 i'm mostly just following along xd
[07:51:06] <dhinus>	 dcaro: thanks!
[07:51:13] <dhinus>	 I think it would be nice to create the incident doc
[07:51:19] * dcaro ic
[07:51:26] <dcaro>	 https://docs.google.com/document/d/1s1f4Lss1Znw3oUav3We6c09QXAY7ja00IsOQyVuQQtw/edit#heading=h.95p2g5d67t9q
[07:52:41] * arturo still not fully online, will be in a few minutes 
[07:52:45] <blancadesal>	 should we send an email to cloud-announce about harbor/build service being down?
[07:53:43] <dhinus>	 blancadesal: +1
[07:53:54] <dhinus>	 I think I know what the quota error is: it's trove quotas
[07:53:57] <dhinus>	 not visible from horizon
[07:54:26] <blancadesal>	 https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Trove&section=10#Adjusting_per-project_Trove_quotas
[07:54:29] * dhinus paged again, not sure why it resolved for a few mins
[07:55:47] <dcaro>	 blancadesal: I'll handle the email, feel free to keep debugging
[07:56:04] <blancadesal>	 dcaro: just started the email xd
[07:56:16] <dcaro>	 blancadesal: okok, please continue then
[07:56:49] <dhinus>	 quota upgraded
[07:58:25] <dcaro>	 \o/
[07:59:03] <dhinus>	 Success: Resizing volume 
[08:00:10] <blancadesal>	 yay!
[08:00:21] <dhinus>	 it didn't really work though :D
[08:00:27] <dhinus>	 there's no volume mounted
[08:00:44] <dhinus>	 I'll try mounting manually
[08:01:58] <dhinus>	 it mounts but it's still at the old size :/
[08:02:44] <dcaro>	 you might need to expand the fs
[08:03:45] <dhinus>	 yes, I was hoping trove would do it for me
[08:03:59] <dhinus>	 lsblk shows the new size
[08:04:53] <dhinus>	 "resize2fs /dev/sdb" fixed it
[08:05:02] <dhinus>	 restarting the trove db from horizon
[08:05:26] <dcaro>	 🤞
[08:05:37] <dhinus>	 we're back!
[08:05:42] <dhinus>	 I can log in to https://tools-harbor.wmcloud.org/
[08:06:40] <dcaro>	 \o/
[08:06:44] <blancadesal>	 harbir-core logs are back to normal
[08:06:55] <dcaro>	 let's try running a build
[08:08:16] <blancadesal>	 running the functional tests now
[08:08:25] <blancadesal>	 all good
[08:08:33] <dcaro>	 \a/
[08:09:26] <blancadesal>	 sending an update email
[08:10:35] <dcaro>	 okok, I'm resolving the incident
[08:10:39] <dcaro>	 :)
[08:10:46] <blancadesal>	 do we have a Phab ticket for this if people need to report something?
[08:11:23] <dcaro>	 take a look at https://docs.google.com/document/d/1s1f4Lss1Znw3oUav3We6c09QXAY7ja00IsOQyVuQQtw/edit and add there any stuff I might have missed/got wrong (still filling stuff up though), I'll sort all the info out after
[08:24:44] <arturo>	 good work you all, great incident kung-fu
[08:28:01] <dcaro>	 dhinus: there was a blip in the prometheus stats, probably from a restart thinking that it worked until it failed to do some db stuff
[08:28:04] <dcaro>	 https://usercontent.irccloud-cdn.com/file/dxE92KOm/image.png
[08:28:09] <dcaro>	 that might be the cause for the double alert
[08:37:15] <dhinus>	 interesting, because I stopped the container with "docker stop" and I didn't see it coming up, let me check the docker logs
[08:37:50] <dhinus>	 hmm I'm still seeing errors in the logs
[08:38:29] <dhinus>	 WARNING:  archiving write-ahead log file "0000000100000244000000B5" failed too many times, will try again later
[08:39:05] <dhinus>	 there's more details, it's apparently failing to "cp pg_wal/0000000100000244000000B5 /var/lib/postgresql/data/wal_archive/0000000100000244000000B5"
[08:41:17] <dcaro>	 that's trove itself
[08:41:40] <dcaro>	 that was supposed to be fixed in the latest version, it's used for backups and replication, but it's not working
[08:41:45] <dcaro>	 (the wal_archive thingie)
[08:42:36] <dcaro>	 https://phabricator.wikimedia.org/T343683
[08:42:43] <dcaro>	 T343683
[08:42:44] <stashbot>	 T343683: [toolsbeta.harbor] trove postrgres DB out of space, v2 - https://phabricator.wikimedia.org/T343683
[08:42:48] <dhinus>	 I see
[08:43:06] <dhinus>	 the annoying bit is that it's spamming the logs with thousands of those lines :)
[08:43:48] <arturo>	 I'm diving into prometheus metrics to see if we could alert/warn on this situation approaching, but it seems there is a metric reporting 0 always
[08:43:49] <arturo>	 https://thanos.wikimedia.org/graph?g0.expr=openstack_trove_instance_volume_size_gb%7Btenant_id%3D%22tools%22%7D&g0.tab=0&g0.stacked=0&g0.range_input=2h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g1.expr=openstack_trove_instance_volume_used_gb%7Btenant_id%3D%22tools%22%7D&g1.tab=0&g1.stacked=0&g1.range_input=2h&g1.max_source_resolution=0s&g1.deduplicate=1&g1.partial_response=0&g1.store_match
[08:43:49] <arturo>	 es=%5B%5D
[08:44:38] <arturo>	 short url: https://w.wiki/Ajv$
[08:44:54] <dcaro>	 arturo: feel free to add your notes here T354728
[08:44:54] <stashbot>	 T354728: Trove does not expose amount of disk space used - https://phabricator.wikimedia.org/T354728
[08:45:13] <arturo>	 oh, it is known
[08:46:32] <dcaro>	 dhinus: the wal was supposed to be disabled though :/ not trying to do it
[08:46:52] <dcaro>	 maybe it got re-enabled somehow (as it's not the default in trove, it was manually changed)
[08:48:08] <dhinus>	 I confirm those log errors about the wal started hours before the outage
[08:48:25] <dhinus>	 the first "disk full" was at 6:40 UTC (25 mins before the page)
[08:49:02] <dhinus>	 page was actually at 6:59 so 19 mins after the first error
[08:49:20] <dcaro>	 we have 10min buffer in the alert for the prometheus stat
[08:50:07] <dhinus>	 the logs don't show any sign of restarts or attempted restarts before 8:05 so it's a mystery why the prometheus metric had that glitch
[08:52:00] <dhinus>	 not true, there are some restarts, sorry
[08:52:53] <dhinus>	 it was in a restart loop until I did "docker stop" which was roughly at the time the alert glitched
[08:53:07] <dhinus>	 so one of the restart loops probably caused the alert to become green for a moment
[08:58:25] <dcaro>	 the db should actually be pretty small, in the order of 100MB
[09:00:20] <dcaro>	 the number of executions in 8x smaller than last time though (2M vs 17M), so maybe the wal is also taking space
[09:01:26] <dcaro>	 https://www.irccloud.com/pastebin/lT7xuYpA/
[09:01:28] <dcaro>	 yep
[09:05:04] <dcaro>	 dhinus: just manually disable the wal, and restarted the database, I don't see the wal logs anymore, will cleanup the files on disk
[09:06:08] <dcaro>	 let me investigate if we can disable at trove level somehow to avoid having to manually apply
[09:09:36] <dcaro>	 :/, wal settings are not exposed 
[09:09:37] <dcaro>	 root@cloudcontrol1006:~# wmcs-openstack database configuration parameter list b0fb1fc1-47cb-4703-9e77-2c8476ac05ee | grep wal
[09:10:11] <dcaro>	 oh, wait, wrong id
[09:10:12] <dcaro>	 https://www.irccloud.com/pastebin/oDrnQml0/
[09:10:16] <dcaro>	 \o/
[09:14:55] <dcaro>	 oh my... I edited a task instead of creating a subtask :/
[09:26:30] <dcaro>	 the volume stats are pulled from the cinder api by the openstack exporter, using https://docs.openstack.org/api-ref/block-storage/v3/#show-quota-usage-for-a-project
[09:26:50] <dcaro>	 I'm guessing that it might require some sort of openstack agent running inside the VM, but not sure :/
[09:32:42] <arturo>	 maybe because cinder sees blocks, and only from within the VM you can see actual filesystem usage. 
[09:39:59] <dcaro>	 hmpf... the cinder quotas don't seem to be exposed through the openstack cli, and the cinder cli does not seem to use the cloud file
[09:51:28] <dcaro>	 that's not what's used, it uses the trove api
[09:51:32] <dcaro>	 and it's shown on the cli
[09:51:33] <dcaro>	 https://www.irccloud.com/pastebin/ZjKTVaeN/
[09:51:58] <dcaro>	 might be a data format error somehow, it's using a different endpoint though, so maybe it's not shown in that other one
[10:02:02] <dcaro>	 hmm, I think this might be related
[10:02:05] <dcaro>	 https://www.irccloud.com/pastebin/ibDFCODk/
[10:02:19] <dcaro>	 from the trove guest agent on the VM
[10:02:50] <arturo>	 who/what is telling the agent to read /dev/vdb ?
[10:03:35] <dcaro>	 not sure, it should be reading it from the config as far as I can see in the code
[10:03:37] <dcaro>	 https://www.irccloud.com/pastebin/XD2pc6Ca/
[10:03:40] <dcaro>	 and it's there
[10:03:44] <dcaro>	 (guest-agent-venv) root@tools-harbordb:/opt/guest-agent-venv# grep vdb /etc/trove/conf.d/*
[10:03:48] <dcaro>	 that returns empty
[10:04:20] <dcaro>	 this is suspicious 
[10:04:21] <dcaro>	 lib/python3.8/site-packages/trove/common/cfg.py:    cfg.StrOpt('device_path', default='/dev/vdb',
[10:04:30] <arturo>	 ah!!
[10:05:09] <arturo>	 I have seen that before, being bitten by a default because a config was not being read/applied as expected
[10:05:25] <arturo>	 I think I even reported this to upstream openstack once, with neutron
[10:06:13] <arturo>	 maybe it was this one https://bugs.launchpad.net/neutron/+bug/2003534
[10:28:51] * dcaro lunch
[10:29:06] <dcaro>	 will have to go to the vet right after, might take a bit more time than usual
[10:29:17] <arturo>	 👍
[11:06:58] <dcaro>	 hmm... this looks ok:
[11:07:01] <dcaro>	 https://www.irccloud.com/pastebin/qB53PKie/
[11:08:07] <dcaro>	 2024-07-24 07:59:11.848 554 ERROR trove.guestagent.volume [None req-2da7ab1f-c952-4a74-9cf1-40005efe0ace fnegri trove - - - -] Device '/dev/vdb' is not ready.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
[11:08:21] <dcaro>	 I think that the device might come from the openstack db, not the trove config directly
[11:08:26] <dcaro>	 (as in an api call)
[11:20:52] * dcaro off to vet
[12:07:22] * dcaro back
[13:28:55] <blancadesal>	 dcaro: quick review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1056494
[13:29:57] <dcaro>	 blancadesal: that's fixed already xd just rebased and it shows empty
[13:30:10] <dcaro>	 sorry if I got you confused
[13:30:11] <blancadesal>	 ?
[13:30:21] <blancadesal>	 ah, you did it already?
[13:30:31] <dcaro>	 yep https://gerrit.wikimedia.org/r/c/operations/puppet/+/1056489
[13:30:45] <dcaro>	 it was blocking the creation of credentials for new accounts
[13:31:07] <blancadesal>	 thanks :) and sorry for missing that one
[13:34:59] <dcaro>	 np I missed it too, the tests were passing when we merged it, but once that endpoint was not available on the envvars side it started breaking
[13:35:57] <blancadesal>	 another quick review: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/449
[13:37:24] <dcaro>	 blancadesal: how did it pass before?
[13:37:49] <blancadesal>	 it didn't, I yolo'd it
[13:38:04] <dcaro>	 🤦‍♂️ xd
[13:43:07] <blancadesal>	 "doing it freed up some mental space to be paranoid about higher-stake changes"
[13:43:32] * blancadesal is all about the narrative
[13:43:39] <dcaro>	 xd
[13:44:28] <dcaro>	 don't hesitate to ask for a second pair of eyes on anything
[13:44:53] <blancadesal>	 like with the envvars change? :P
[13:46:12] <dcaro>	 yep (UwU)
[13:49:36] <blancadesal>	 for the calico upgrade, does anything need to be done here https://gitlab.wikimedia.org/repos/cloud/toolforge/calico or is it all through toolforge-deploy now?
[13:49:44] <blancadesal>	 I already got the new images
[13:50:58] <dcaro>	 we use that yes
[13:51:11] <dcaro>	 then in toolforge-deploy you can specify which version of the generated chart you want
[13:52:41] <dcaro>	 you will probably need to update it with the newer templates from upstream (see the notes in the current templates)
[13:53:28] <dhinus>	 is there any grafana dashboard for cloudlbs?
[13:54:15] <dhinus>	 something that shows the number of requests/bandwidth/etc. for wikireplicas as reported by haproxy
[13:55:19] <dcaro>	 blancadesal: not sure though if/why we use the templates and not the helm chart directly, I think it might install other things we don't need, arturo do you remember?
[13:55:53] <dcaro>	 dhinus: there might be something, you have one for latency https://grafana.wikimedia.org/d/UUmLqqX4k/wmcs-openstack-api-latency?forceLogin&orgId=1&refresh=30s&var-backend=keystone-admin-api_backend&var-backend=keystone-public-api_backend&var-backend=nova-api_backend&var-cloudcontrol=cloudlb1001&var-cloudcontrol=cloudlb1002
[13:56:29] <dcaro>	 this one is out of date though :/ https://grafana.wikimedia.org/d/tanisM2Zz/wmcs-openstack-api-stats-eqiad1?orgId=1&refresh=5m
[13:56:29] <blancadesal>	 dcaro: afaik, we are not installing all the components. I only pushed a subset of the images. but arturo will know more
[13:57:03] <blancadesal>	 arturo: when you have a moment to recheck https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434
[14:00:07] <arturo>	 I will check later
[14:57:28] <blancadesal>	 only 666 tasks left from that query :)
[14:57:33] <blancadesal>	 ominious
[14:57:50] <blancadesal>	 without the i 
[14:58:06] <dhinus>	 hahaha
[14:58:23] <arturo>	 is that number going up or down? or are we stuck in a triaging hell?
[14:58:24] <andrewbogott>	 'ominious' is a more ominous version of 'ominous'
[14:58:57] <blancadesal>	 sometimes more is less
[15:01:04] <dhinus>	 only 142 "needs triage" left in the cloud-services-team backlog
[15:01:20] <dhinus>	 probably doable in 2/3 meetings?
[15:01:35] <dhinus>	 I didn't count how many we triaged today
[15:03:01] <dcaro>	 I think triage will be needed regardless anyhow, probably less often, or shorter, but it can't be avoided
[15:04:58] <dhinus>	 yep I was thinking how long it will be before we can only triage "new" tasks :)
[15:12:09] <dcaro>	 technically we cleared two years of tasks today xd
[15:12:15] <dhinus>	 :D
[15:12:33] <arturo>	 not bad
[15:12:40] <arturo>	 👍
[15:15:17] <dhinus>	 related: in the clinic duties wiki there is a link to a query to be used for triaging tasks
[15:15:39] <dhinus>	 but I've always found that query to be sub-optimal, I even tried to improve it a few times
[15:16:14] <dhinus>	 this is the current version: https://phabricator.wikimedia.org/maniphest/query/_oUo8wzJaVVf/
[15:18:41] * dhinus is tempted to start a decision request on "how to triage"
[16:04:34] * arturo offline
[16:07:11] <dhinus>	 I found a nice trick to inspect the progress of long transactions in ToolsDB replication: https://phabricator.wikimedia.org/T370760#10011275
[16:07:26] <dhinus>	 tomorrow I'll add that to wikitech as well
[16:55:24] <andrewbogott>	 whoohoo Southparkfan, one more down!
[16:58:32] <Southparkfan>	 andrewbogott: yes!
[16:59:33] <Southparkfan>	 but unless is interested to provide me with assistance to move the rest over, I'm going to stop here
[17:03:17] <andrewbogott>	 that's fine, thank you for all you've done!
[17:04:18] * andrewbogott -> long lunch break
[17:09:20] <bd808>	 dhinus: does the toolsdb replica have a `heartbeat_p` setup? With the recent questions about Quarry searches I'm wondering if I can extend https://replag.toolforge.org/ to also report on ToolsDB replication. (Or make a separate tool to do that)
[17:11:05] <dhinus>	 bd808: yes it has heartbeat_p, it should work!
[17:11:24] <dhinus>	 it's already working if you use quarry, quarry will show you the replay based on that
[17:16:39] <bd808>	 dhinus: Is there a `tools.db.svc.wikimedia.cloud` equivalent service name for the replica?
[17:22:54] * dcaro off
[17:22:56] <dcaro>	 cya tomorrow!
[17:36:31] <bd808>	 I figured out that `tools-readonly.db.svc.wikimedia.cloud` is the service name needed 
[21:05:25] <bd808>	 cteam: I finally put the CSP consent tool proposal on wiki at https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Third-party_interaction_consent_tool. This has been a TODO for me for like 4 months... my brain was stuck on "is it ready to share?" and I finally got over that.