[00:04:21] <bd808>	 !log tools Building new php74-sssd-base & web images (T310435)
[00:04:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:04:24] <stashbot>	 T310435: Korean characters rendered as Unicode boxes in the SVG Translate Tool preview (due to lack of fonts) - https://phabricator.wikimedia.org/T310435
[00:14:10] <wm-bot>	 !log tools.svgtranslate <root> Restart to pick up new php7.4 container with added fonts (T310435)
[00:14:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.svgtranslate/SAL
[00:36:05] <bd808>	 !log tools Rebuilding all Toolforge docker images to pick up bug and security fix packages. (T316554)
[00:36:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:36:07] <stashbot>	 T316554: Update jdk17 toolforge image to 17.0.4 - https://phabricator.wikimedia.org/T316554
[00:39:37] <bd808>	 !log tools Image rebuild failing with debian apt repo signature issue. Will investigate tomorrow. (T316554)
[00:39:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[01:44:31] <legoktm>	 !log library-upgrader deleted stuff from upgrader-06 because it ran out of disk space oops
[01:44:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL
[09:50:06] <arturo>	 heads up, the cloud network just had an iccup
[10:14:54] <arturo>	 !log admin [codfw1dev] restart neutron-l3-agent on cloudnet2006-dev, it was dead
[10:14:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:40:06] <wm-bb>	 <lucaswerkmeister> 🤨 is running Toolforge tools on the Grid no longer a supported option? https://phabricator.wikimedia.org/T319357
[10:40:16] <wm-bb>	 <lucaswerkmeister> (that’s one of several similar tasks, I just picked the first one I saw)
[10:40:40] <wm-bb>	 <lucaswerkmeister> I thought the latest status was still “k8s is recommended, but we’re aware that there are legitimated uses for the grid that we don’t have a replacement for yet”
[10:40:57] <wm-bb>	 <lucaswerkmeister> but in the task, keeping the tool running on the grid isn’t presented as an available option
[10:42:44] <wm-bb>	 <lucaswerkmeister> (that’s me paraphrasing, not a verbatim quote, just to be clear ^^)
[10:48:55] <arturo>	 @lucaswerkmeister, that's a very legit question. I'll try to answer a bit later, currently busy with other stuff
[10:49:04] <wm-bb>	 <lucaswerkmeister> ok
[10:49:55] <wm-bb>	 <lucaswerkmeister> if the answer is “we’ve actually made a lot of progress on build packs” I’ll be very happy ;)
[10:54:54] <arturo>	 we've actually made a lot of progress on build packs :-)
[10:54:58] <arturo>	 that's a true statement
[10:55:05] <wm-bb>	 <lucaswerkmeister> oh nice!
[10:56:33] <arturo>	 I'll talk to my coworkers to see if additional communication can be made on that regard
[10:57:18] <taavi>	 arturo: i feel that over the last few months, a lot of planning and progress upgrades have moved from public venues (irc, phab, cloud-admin@l.wm.o) to somewhere else that I can't see..
[10:57:32] <taavi>	 especially for buildpacks and the grid decom stuff
[10:58:19] <arturo>	 taavi: ok. I'll capture your concern and make sure my coworkers are aware
[10:58:29] <taavi>	 thanks
[10:59:37] <arturo>	 actually,, may I suggest both you and lucas send an email to cloud@ with your concern? having something written makes for a stronger case
[11:01:20] <taavi>	 will do. I have a perfect example for this from yesterday :P
[11:01:32] <arturo>	 👍
[11:40:30] <arturo>	 !log admin [codfw1dev] rebooting both network nodes to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/839492
[11:40:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:50:31] <arturo>	 !log admin set neutron l3 agents on cloudnet1005/1006 as down `root@cloudcontrol1005:~# neutron agent-update --admin-state-down <uuid>` (T316284) 
[11:50:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:50:35] <stashbot>	 T316284: Replace cloudnet100[34] with cloudnet100[56] - https://phabricator.wikimedia.org/T316284
[11:54:14] <arturo>	 !log admin rebooting cloudnet1005/1006 to see if they have the right network config (T316284) 
[11:54:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:58:05] <arturo>	 heads up, trying to failover the l3 agent to the newer server. Brief network outage expected
[15:55:51] <arturo>	 !log admin cloudnet1005 & cloudnet1006 now in service. Secom cloudnet1003 & cloudnet1004. Drop neutron agents, etc. (T316284)
[15:55:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:55:55] <stashbot>	 T316284: Replace cloudnet100[34] with cloudnet100[56] - https://phabricator.wikimedia.org/T316284
[18:59:38] <gehel>	 andrewbogott: there is a bunch of alerts about nova-compute on cloudvirt hosts in -operations. Anything we should be concerned about?
[19:00:11] <andrewbogott>	 gehel: it's services recovering from a rabbitmq split-brain. It's not great that rabbit does this but there's nothing urgent happening.
[19:00:23] <gehel>	 ok, thanks!
[19:00:34] <andrewbogott>	 thanks for checking