[08:39:12] relforge is failing with "this action would add [6] total shards, but this cluster currently has [2000]/[2000] maximum shards open" [08:41:38] somehow it has plenty of wikis there e.g. lfnwiki_content [08:43:28] created on jun 13 with an empty mapping (so most likely autocreated), might be some testing I suppose [08:43:37] going to delete them [09:53:15] lunch [12:02:57] inflatador: I'm cancelling our 1:1, conflicting meeting [12:43:49] gehel ACK [12:43:49] on wiki, is there normally a lag between updating a template and it actually displaying on the pages that use that template? [12:59:21] nm, looks like there is [13:01:25] inflatador: you can do a "null edit" on a page to force it to re-render [13:02:21] https://www.mediawiki.org/wiki/Help:Dummy_edit#A_null_edit [13:47:43] pfischer: I've done another round of review, should be close to be testable imo, please lemme know if you have questions [13:50:40] Sure, I’ll have a look right away. Thank you! [13:58:19] dcausse: Regarding https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/1033386/comment/7457a5f9_25a340ea/ - Should the pipeline fail if there is no side-output for a subgraph? [14:01:33] inflatador: thanks for the help to prep for today's switch upgrade (T365993) [14:01:33] T365993: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993 [14:02:30] unfortunately we have to postpone the works until next week [14:02:38] so I guess elastic1104, elastic1089 & elastic1090 can be re-pooled for now [14:02:39] topranks and thank you for your comms. You make it easy for me ;) [14:02:50] apologies for the inconvenience [14:03:03] np :) [14:03:09] pfischer: yes Map.apply should fail for us I think [14:03:09] np, we have plenty of capacity. Hope everything went well! [14:03:51] yeah just need to move it out to make life easier for some of the teams [14:04:25] we're still doing E2 tomorrow which has elastic1091 & elastic1092 [14:06:34] ACK, I have those on my list to ban today, along w/wdqs1018 & 1020 [14:08:11] great, thanks [14:17:14] ryankemper no rush, but https://gerrit.wikimedia.org/r/c/operations/puppet/+/1051369 is ready for review if/when you have time [14:26:47] pfischer: your opinion on T367391#9943556 would be welcomed! [14:26:48] T367391: Setup a test project to validate upload to the Gitlab package registry - https://phabricator.wikimedia.org/T367391 [16:07:01] gehel: looking [16:26:17] workout, back in ~40 [17:11:49] back [17:14:11] dinner [17:48:42] lunch, back in time for pairing in ~45 [18:32:08] back [18:34:43] ebernhardson ryankemper I'm in pairing if y'all feel like joining [18:40:33] inflatador: https://phabricator.wikimedia.org/T263073 [18:42:33] hieradata/common/profile/services_proxy/envoy.yaml [21:13:43] ryankemper posted a follow-up on T368972 . TLDR it does look like envoy can do this, but no one else seems to be using it that way [21:13:44] T368972: Use Envoy instead of LVS to route internal federation traffic for WDQS - https://phabricator.wikimedia.org/T368972 [21:15:31] inflatador: it would need some way to get etcd state to determine what hosts are depooled though right? i.e. we need some mechanism whereby if a host is depooled from pybal that it's also not going to be included in envoy's pool [21:19:48] ryankemper Ideally yes, although the health checks will catch short outages. Longer outages would require a puppet patch to remove the downed host [21:25:25] the bigger problem with this approach as that we now have 2 places to configure LB config [21:26:10] err..."2 LBs to configure" [21:27:49] ryankemper: inflatador: btw the serviceops internship project that just concluded was an Envoy control plane that reads etcd confctl [21:29:30] well, that was the intent anyway, I don't know what milestones were actually reached :) https://gitlab.wikimedia.org/repos/sre/sophroid [21:31:29] cdanis yeah, I saw that presentation and was actually thinking of that, although I didn't wanna open that can of worms quite yet ;) [21:31:35] ah ok :) nvm [21:33:00] I'd love to end up there at some point though. I guess we can ask around and see if non-k8s envoy is in scope