[02:37:53] I'm looking into this DNS discovery diff alert for wdqs: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=puppetmaster1001&service=DNS+Discovery+operations+diffs [02:39:26] The cause of the alert is pretty straightforward, right now it expects `false` for `eqiad` and `true` for `codfw`: https://gerrit.wikimedia.org/g/operations/puppet/+/production/modules/profile/files/configmaster/disc_desired_state.py#77 [02:39:48] Whereas we apparently have `true` for both: [02:40:00] https://www.irccloud.com/pastebin/uv5cEPLJ/confctl%20--quiet%20--object-type%20discovery%20select%20'dnsdisc%3Dwdqs'%20get [02:41:03] What I'm not sure is whether I should update the desired state or whether I should do the opposite and flip `eqiad` to `false` [02:42:24] (WDQS is `active/active` btw) [03:19:07] ryankemper: most active/active services were depooled in eqiad during the switchover, so it just depends whether you want it pooled in eqiad for another month or two [03:24:24] gotcha, was thinking it might have had something to do with the switchover but the ~6 day timing of the alert was throwing me for a loop [03:24:57] okay will talk to the rest of search team and see what state we're supposed to be in [05:36:34] <_joe_> no it's not the switchover, I think it was made active again in eqiad during the A2 switch outage [08:59:42] yup, it was pooled on eqiad on Friday 16th [09:00:07] FYI we currently have a line card down on cr2-codfw (should be replaced today, Juniper have shipped replacement). [09:00:31] This means reduced capacity between eqiad and codfw, so ideally we would not do anything that would put more traffic on the transport between those sites. [09:01:09] we depooled eqiad in DNS yesterday which helped with this as remaining transport was maxing out. [09:01:16] Unsure if it's relevant to this but just mentioning. [09:07:02] Estimated delivery: Friday, July 23 by 10:30 A.M. [09:07:05] so far so good [10:51:27] hi folks, there is an error in reprepro @ apt1001, apparently something is missing for the thirdparty/helm3 component [10:51:33] root@apt1001:~# reprepro --noskipold --component thirdparty/kubeadm-k8s-1-19 update buster-wikimedia [10:51:34] Could not find 'stable/binary-amd64/Packages' within '/srv/wikimedia/lists/thirdparty%2Fhelm3_all_InRelease' [10:51:34] There have been errors! [10:51:35] cc moritzm [10:52:21] nevermind, it is my own fault [10:54:32] no, it's my fault in copy-pasting the wrong thing [16:31:08] godog: are you in today? We have some questions for you in -cloud [16:37:41] topranks, XioNoX: if you're about, just making sure you saw T287203 [16:39:03] rzl: yep, thx [16:39:29] 👍