[07:13:51] db1228 is going to be down for onsite maintenance [07:59:29] dhinus: I am going to be changing s2 sanitarium to a new master, so there will be abit of lag on wikireplicas but only for s2 section [08:36:07] marostegui: ack! [08:36:21] dhinus: it was done and I will start with s5 soon too [08:36:31] ok! [10:21:20] all sanitarium instances have been moved under the new HW [10:21:38] I am gonig to wait a few days or a couple of weeks before starting to move wiki replica under the new sanitariums [10:41:15] Hm, my theory that the cassandra reinstalls are sensitive to disk ordering looks like it's probably correct [10:41:59] [nothing sets a disk to install upon, to grub-install is only called on /dev/sda, and if that's not /dev/disk/by-path/pci-0000:00:11.5-ata-3.0 then no-one is going to space today] [11:19:27] marostegui: I'm putting toghether the alert for depooled-but-not-silenced hosts, when you have a second can we check some hosts in https://grafana.wikimedia.org/d/fc7sbw8/mariadb-pooling-and-silences-overview?from=now-6h&to=now&timezone=utc ? [11:21:29] federico3: so db1174 has notifications disabled and is not in dbctl, so not sure why it shows up there, db1269 and db1270 are sanitarium hosts, so they are not even in dbctl [11:21:54] federico3: db2207 has notifications disabled, db1282, let me investigate [11:22:41] db2207 should be pooled, so I will enable notirfications and pool it back (it was a host that crashed a few weeks ago) [11:24:36] hi Ceri, I'm running late. let's start 5 minutes late, please [11:24:44] federico3: I am having 500 error when using pool cookbook [11:46:04] federico3: ping [12:04:07] marostegui: you seem to have some packet loss [12:12:53] Emperor: o/ I completed the load of Docker images on the S3 buckets in apus, and we are around 2TB [12:13:44] we can prune images from that, but it will require some work, so ideally I'd say that we are looking to a +1/2TB growth during the next year [12:13:56] (1 or 2, not half :D) [12:14:01] is it something feasible? [12:14:22] (we'll free 6TB on swift as consequence :D) [12:15:52] elukey: yeah, I think so, but I'd like a ticket with the increase request. And regrettably swift and ceph capacity is not fungible like that :) [12:31:21] Emperor oook I'll open one! I was just returning something back :D [12:35:15] marostegui: can you paste the error or the command so i can run the same? [12:36:03] I'm mostly releived that apus has been holding up OK without the faster storage I was hoping to have in place by now... [12:44:43] federico3: in a meeting but essentially pooling a host i get: Error HTTP Error 500: Internal Server Error [12:53:54] somehow it seemed to have used an old credential, not sure how it happened, but it's ok after a restart [12:57:05] federico3: do we have a cheatsheet for those things in zarcillo? so we don't have to ping you all the time :-)? It'd be good to add it on wikitech [12:58:22] yes at https://wikitech.wikimedia.org/wiki/MariaDB/Zarcillo#Development but i can expand on it [12:59:32] federico3: yeah, maybe with a small section of usual maintenance operations like the one you just did