[08:13:24] good morning. we can't yet default the insetup::data_persistence role to Puppet 7 until the mariadb remote access is sorted out, but I'll migrate the hosts using insetup which will eventually get a role applied which uses Puppet 7 to Puppet 7 on a per hosts basis [08:13:41] this way it's as simply as just applying the role like before [08:14:34] i.e. sessionstore2004-2006, backup[12]01[01] [08:15:07] moss-fe2002 and moss-be2002 can't get swiched over yet, since they need to get reimaged away from buster first [08:37:29] I'm doing those, but they're tied up in also fixing the disk setup on the backends [08:38:15] I'm planning to put them back into insetup::data_persistence once all the moss-* nodes are done [08:38:24] (except the two in service as ms frontends) [09:06:46] when done, I suppose the moss nodes will use a new role and not one of the existing ceph ones? then it would be best to just go ahead and create a simple stub role which only includes the base classes [09:07:04] and then switch that stub role towards Puppet 7, so that the rampup of the new servers directly happens on Puppet 7 [09:11:03] new role> that's the plan. They also need renaming :-/ [09:20:26] ack [11:23:36] eqiad seems to have disappeared from mysql graph in grafana? [11:24:46] https://grafana-rw.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&var-site=eqiad&var-group=All&var-shard=All&var-role=All [11:24:54] probably from prometheus, too? [11:29:49] what do you mean? [11:30:02] it just went back [11:30:09] we had a monitoring outage [11:30:21] see: https://grafana.wikimedia.org/goto/yuEjfJvIk?orgId=1 [11:30:51] yeah, I knew that. I thought you meant it was not selectable :) [11:31:00] it wasn't, for some time [11:31:25] now as metrics appeared again, I was able to select them [11:31:46] sustained lag alerts were on unknown also