[04:51:35] Going to start with s1 switchover [09:15:34] arnaudb: if you just want to tag things, why does https://phabricator.wikimedia.org/T372862 need a space? [09:15:55] to have its board attached to a tag [09:15:55] Surely you just need a project tag [09:16:09] I may need to make it clearer in the task though [09:16:14] arnaudb: a space is for restricting visibility [09:16:24] This doesn't feel like you want a space [09:17:09] Do you just want another project with a workboard? [09:17:28] indeed in that case just a project, with a tag and a board exactly [09:17:46] I'll edit the task accordingly [09:19:58] arnaudb: please use the template in https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects [09:20:26] ack, fixing the template [09:49:53] FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:03:39] I suspect that's my previous silence expiring, let me check [10:07:39] indeed so, silence extended [10:07:52] (this is apropos T372207 ) [10:07:53] T372207: Disk (sdc) failed on ms-be1058 - https://phabricator.wikimedia.org/T372207 [13:04:21] I'm gonna depool and reimage clouddb1014 (T365424) [13:04:22] T365424: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424 [13:33:08] well, apus testing hasn't immediately caught fire - I can upload in both codfw and eqiad at once to the same bucket (not the same object, natch), and then the results are available in both DCs [14:05:16] clouddb1014 is reimaged and repooled. I'm now reimaging clouddb1013 (last one!) T365424 [14:29:38] The bit of the day where I download cat pictures from commons and upload them to apus for testing purposes is a good bit of the day :) [22:45:48] FIRING: MysqlReplicationLag: MySQL instance db1206:9104 has too large replication lag (7m 9s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [22:45:48] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db1206:9104 has too large replication lag (6m 58s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [22:50:48] RESOLVED: MysqlReplicationLag: MySQL instance db1206:9104 has too large replication lag (5m 2s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [22:50:54] RESOLVED: MysqlReplicationLagPtHeartbeat: MySQL instance db1206:9104 has too large replication lag (5m 2s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1206&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat