[09:01:26] fyi, I'm depooling esams to reboot the switches for https://phabricator.wikimedia.org/T437984 [11:14:44] FYI, for those who use homer outside of our networks folks; homer is now undeployed from cumin1003, please only use cumin1003 (or cumin2003) foing forward [11:15:00] cumin1004 (or cumin2003) [12:11:45] depooling magru for network maintenance [12:14:00] slyngs: https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DLibericaDiffFPCheck related to your work? [12:14:30] Yup, looking into it. Liberica can't see the "new" host for some reason [12:14:52] and moritzm maybe https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=site%3Dmagru&q=alertname%3DNodeTextfileStale (but it's 1 month ago) [12:15:16] But feel free to do your thing, I'm still trying to understand what Liberica wants from me :-) [12:16:13] slyngs: https://wikitech.wikimedia.org/wiki/Liberica#LibericaStaleConfig ? [12:17:50] Yeah, that was my thinking to, but no. I think it's due to the weight in confd being 0 [12:18:07] ok [12:20:58] that's a bug in the prom-based monitoring, it doesn't properly deal with migrations of the ganeti master [12:21:35] IIRC I made a patch, but haven't had the time to properly get it reviewed and deployed [12:49:04] magru has been repooled [14:37:58] Do we differentiate or split purge requests from MW via Kafka/purged in any way? [14:38:30] e.g. do purges always get applied by oth text and upload cp hosts, and they just no-op the majority? [14:38:48] or is there some kind of splitting or optimisation there? [14:40:17] <_joe_> Krinkle: they are definitely filtered/split [14:40:44] <_joe_> I don't recall right now how that is done, but purges on upload are like 3 or 4 orders of magnitude less [14:51:54] yeah, asking in context of thumb/upload split [14:52:04] I assume MW now emits purge for thumb, and only thumb. [14:52:07] which, if those work, is fine. [14:52:17] I guess we can treat the other as non-canonical and not care about purges as much [14:53:39] I am going to disable puppet on the hosts using pki (basically most of the bare metal infra) to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341234, should be re-enabled soon [14:54:07] ack [14:57:19] ack, thanks elukey [15:04:22] _joe_: where does that split happen? [15:05:52] https://codesearch.wmcloud.org/operations/?q=upload&files=&excludeFiles=&repos=sre%2Fpurged [15:05:53] https://codesearch.wmcloud.org/search/?q=host_prefixes [15:05:55] I don't see this set anywhere [15:05:58] <_joe_> don't remember tbh [15:06:47] I'm guessing the way it was envisioned is that all are sent to text (incl upload.wm.o despite redundant), and then upload is filtered via host_prefixes [15:07:45] ah there's a host_regex as wlel [15:07:51] host_regex [15:07:55] re-enabling puppet now [15:09:48] althoguh host_regex (for purgd) != purge_host_regex (for varnish) [15:10:32] done [15:10:36] profile::cache::purge::host_regex: '[um][pa][lp][os]' [15:11:35] I guess ^(?:upload|maps) was/is too slow or unsupported? [15:12:10] anyway, so no split on where MW sends purges or in the kafka topics, but it's on the consumer side in purged.go itself [17:20:29] jhathaway: we have pending changes from you in puppet merge [17:20:33] can we go ahead and merge them? [17:21:20] jhathaway: we are merging them, please let me know if that was a bad idea :) [17:21:32] (it blocks some other things we need to get in, so unfortunately we are not waiting) [17:39:17] so, my bad, thanks sukhe [17:39:44] stepped away for lunch, and forgot it was outstanding [17:40:01] no worries, we merged it! [17:44:47] It fills me with some dread to say this, but: If there are no objections, I am going to reimage the last remaining sessionstore host. sessionstore1005 is back up, and this node (1006) doesn't have any... history. No impact is expected. [17:46:51] godspeed! [18:04:41] urandom: I just want to tell you both good luck, and we're all counting on you [18:04:56] awesome; no pressure :) [18:07:10] * urandom reaches for a calming playlist [18:09:17] urandom: https://www.youtube.com/watch?v=a01QQZyl-_I I play this on both good and bad days [18:09:49] (because difficult to imagine a better combo) [18:09:57] :) [18:10:44] better than what I had playing: Hell Yeah (Rev Theory) [18:29:16] sukhe: big fan of the live performance with Annie Lennox singing Freddie's part, after he died [18:29:27] Freddie is irreplaceable but David Bowie and Annie Lennox is a hell of an act [18:30:20] woo thanks rzl. now playing. [18:57:08] It Begins [18:57:24] (namely, my metal week is turning into a queen week) [19:22:25] * urandom exhales [19:22:53] the sessionstore reimages are now done [19:25:37] 🙌 [19:26:28] nice! [19:37:59] nice :) [19:38:15] kinda feel like we're running sessionstore at too small redundancy given its importance