[02:40:11] 06Traffic, 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315632 (10MauricioGenta) I understand @Nemoralis even if a disagree, i started the thread because a user called my attention telling me... [03:14:03] 06Traffic, 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315670 (10Pppery) This "decision" was taken in the code review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1069643, where @D... [03:45:37] 06Traffic, 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315691 (10MauricioGenta) Oh cool, I didn't think we had so many domains! I trust your judgment would be the better, i do not want to bre... [04:50:17] 06Traffic, 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315775 (10Joe) My 2 cents: I think the redirect should be changed. No one expects a site with an argentinian TLD to be in arabic. [13:33:57] arnaudb: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/b94f91c4fbcc79a491fb5b611902e6834b4c059e rolled out on all A:cp-text [13:34:09] <3 [13:34:10] thanks [15:00:36] FYI, there's an alert for requestctl-credential-refresh, that is probably related to Daniel now being added to the ops group: FIRING: [3x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 [15:02:50] let me have a look [15:06:44] the step `This requires generating and adding an API token for yourself in the `profile::conftool::hiddenparma::api_tokens` array in the private puppet repo.` is crossed out on the onboarding page [15:11:33] <_joe_> yes [15:11:57] <_joe_> now it should be daniel going to https://requestctl.wikimedia.org/user-profile and generating his api token :) [15:12:27] let me update the onboarding template if isn't [15:13:07] _joe_: IIUC this is required only if the person plans to use HP APIs, correct? [15:13:19] <_joe_> fabfur: you understand incorrectly [15:13:28] <_joe_> it's required to use the requestctl cli tool as well [15:13:57] yeah, that uses the API, but is not strictly required, AKA, should it trigger an alert on alertmanager? [15:14:09] <_joe_> we can make requestctl-credentials-refresh not fail on missing tokens, but I think it's healthy that we are aware [15:14:28] dkertesz: in the meantime ^^ [15:15:34] <_joe_> fabfur: if you are volunteering to go check all the docs and remove the references to the requestctl cli, I am happy to remove the failure mode for a missing token for an ops member [15:16:19] I wouldn't remove the references, but I'll put a big warning sign saying that to use that you _need_ to generate an API token first [15:16:46] <_joe_> and that to you is better than adding an action item to the onboarding checklist? [15:16:59] I can do both [15:17:08] <_joe_> you don't just need to generate the api token, you also need to run the refresh script [15:17:29] but it's not strictly required for a new account IMHO [15:18:00] we could also update the existing node in data.yaml? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341233 [15:18:08] s/node/note [15:19:51] is there a guide to generate the token to link? [15:22:16] yeah sorry about that confusion, I am happy to update the docs as well since I was the one that removed the reference to the manual hiera during a live call with Daniel [15:22:19] please let me know [15:22:50] sukhe: I just added a couple of notes [15:22:51] https://wikitech.wikimedia.org/w/index.php?title=Requestctl&diff=2457061&oldid=2439758 [15:23:13] but if there's anything else to do happy to add a new section and link it everywhere [15:25:36] <_joe_> fabfur: yes add to the checklist that people need to generate their api token? [15:28:57] this has been done [15:29:25] which script needs to be run after ? [15:31:07] @fabfur ack about the API token (was in a meeting, sorry!) [15:53:54] 06Traffic, 06Collaboration-Services, 10Phabricator, 06SRE, 13Patch-For-Review: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#12318700 (10LSobanski) 05Open→03Declined We're not planning to work on this task. [16:27:45] 06Traffic, 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12318925 (10ssingh) @BCornwall can take care of this from Traffic, for input and the patch. [16:33:34] 06Traffic, 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12318955 (10Dzahn) Appreciate the comment from Pppery here. It's not always a big formal decision; a lot of times it's just a few people t... [16:44:39] 06Traffic, 06SRE, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12319023 (10BCornwall) 05Open→03Resolved Thanks for reporting and for the patch! Merged, and now appropriately redirecting to es.wp.o [17:01:53] 06Traffic, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Infrastructure-Foundations: wiki.gives SSL certificate expired - https://phabricator.wikimedia.org/T437914#12319088 (10greg) [17:02:24] copy/paste of what I shared in -sre-foundations: [17:02:25] heyo, the wiki.gives domain (which is a domain we own that is pointed to Acoustic's (FR's email & SMS marketing platform) servers to use as a shorturl domain in SMS sends) cert expired on Friday. It looks like a DigiCert cert. I can't find a reference in phab for who/how that cert was provisioned, but I think we (fr-tech) need ya'll's help with [17:02:25] renewing it (preferably asap! sorry!) [17:18:43] (sukhbir is responding) [17:23:49] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12319207 (10CDanis) I think we need to keep a close eye on cache hit rates for cache-text and specifically the mw-web appservers as we continue this work. Really we probably should have been alr... [17:25:07] 06Traffic, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Infrastructure-Foundations: wiki.gives SSL certificate expired - https://phabricator.wikimedia.org/T437914#12319212 (10ssingh) Hi @greg, thanks for reaching out. So as part of T379318, the domain now is actually handled by Acoustic/Azure, which is... [17:26:12] 06Traffic, 06Fundraising-Backlog, 10fundraising-tech-ops, 06Infrastructure-Foundations: wiki.gives SSL certificate expired - https://phabricator.wikimedia.org/T437914#12319215 (10greg) Thank you! [17:29:46] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12319224 (10ssingh) >>! In T436363#12319206, @CDanis wrote: > I think we need to keep a close eye on cache hit rates for cache-text and specifically the mw-web appservers as we continue this work... [17:31:01] sukhe: I didn't realize we didn't consider the weights at all ... may I ask why? [17:32:20] cdanis: I have no good answers for you unfortunately. I think the last time we did weights was way before the current CDN sandwich setup. [17:32:26] bblac.k is out till Wed otherwise we could have asked him [17:32:45] brett: do you have any recollection about this? [17:33:34] cdanis: while on that note, while we do set weights for say the DNS hosts, we also do not consider them. we put them in on a "we will use them if need be" but so far I don't recall a single use that has come up at least in Traffic. [17:33:41] interesting [17:34:06] I guess it could never have been a thing before maglev / liberica anyway [17:34:30] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12319256 (10Ladsgroup) I honestly think we should urgently start flipping upload cp hosts to text. I can take care of some of them as it's a bit of tedious work but I need the green light from tr... [17:34:33] I have no recollection [17:34:33] (is that true? I actually don't know IPVS well enough offhand to be sure) [17:34:54] wait, are we talking about dns weights or lvs weights? [17:35:00] lvs weights for each cp host [17:35:16] geodns weights I also think we should have, but that's another thing entirely [17:35:32] I think they're arbitrarily used as a tool in specific scenarios, not something implemented [17:37:03] 13:34:06 < cdanis> I guess it could never have been a thing before maglev / liberica anyway [17:37:30] my only recollection is that I last talked about this with Brandon around magru and he told me the weights were removed for a $reason I don't recall at all [17:37:39] trying to see [17:39:41] cdanis: sorry nothing I can find. I made a note for when Brandon is back and will ask him to follow up. [17:39:48] thanks :) [17:40:03] in that case, I think just do them off-peak-ish, and probably both at once [17:40:10] or whatever. [17:40:23] ok, sounds good yeah. I will flag that for slyng.s as well [17:40:34] ehhh... actually, probably one or two at a time in one datacenter can't ever be as impactful as moving a whole wiki's traffic stream [17:40:53] it's probably fine, but, we probably should have worried a little more about ^ a month or so ago [17:41:01] ha well yeah but I think we decided that even though if that worked out, we should probably be a bit more cautious [17:41:15] which is probably us being ever-more-conservative with stuff but yeah [17:41:38] just long as we aren't so cautious it takes noticeably longer, because the end state is definitely better [17:48:36] :) [17:51:58] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12319325 (10ssingh) >>! In T436363#12319256, @Ladsgroup wrote: > I honestly think we should urgently start flipping upload cp hosts to text. I can take care of some of them as it's a bit of tedio... [18:20:38] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12319436 (10BCornwall) +1 on moving some of the uploads over [23:35:44] 06Traffic, 10Beta-Cluster-Infrastructure, 06SRE: Name new CDN servers deployment-cp-(text|upload)0x instead of deployment-cache-(text|upload)0x - https://phabricator.wikimedia.org/T280393#12320486 (10BCornwall) 05Open→03Resolved a:03BCornwall deployment-cp-text09 and deployment-cp-upload09 are up a... [23:54:41] 06Traffic, 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12320526 (10BCornwall) deployment-cp-text09 and deployment-cp-upload09 are provisioned and ready for...