[06:13:40] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve Wikidata traffic via Kubernetes - https://phabricator.wikimedia.org/T347493 (10Joe) @Jdforrester-WMF no, this task is actually about that patch not having the effect we expected. [06:15:30] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve Wikidata traffic via Kubernetes - https://phabricator.wikimedia.org/T347493 (10Joe) Interestingly, I do get correct results for m.wikidata.org, but somehow not for www.wikidata.org (also, please grep for `mw-web` as we've repooled eqiad in th... [06:34:35] 10serviceops, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10Joe) [06:34:55] 10serviceops, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10Joe) p:05Triage→03High [07:23:33] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) [07:33:01] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve Wikidata traffic via Kubernetes - https://phabricator.wikimedia.org/T347493 (10Joe) I tried restarting ATS on a backend, cp1081, then made requests for wikidata's special:random to trafficserver directly: still all going to appservers on bare... [07:43:19] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Serve Wikidata traffic via Kubernetes - https://phabricator.wikimedia.org/T347493 (10Joe) Well turns out the issue was simpler: we even had a TODO in the code: ` # TODO: add mw-on-k8s once we think of moving wikidata or partial traffic. ` Sigh. T... [07:49:48] 10serviceops, 10Abstract Wikipedia team, 10MW-on-K8s, 10Patch-For-Review: Serve wikifunctions.org from mw-on-k8s at 100% - https://phabricator.wikimedia.org/T347509 (10JMeybohm) [07:50:40] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) [08:53:06] ~/26 [09:12:53] <_joe_> nope [09:13:12] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) a:03JMeybohm [09:13:36] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [09:38:42] hi folks! [09:39:04] I finally have a proposal for the CORS header change in the ingress module - https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/961379 [09:39:19] tried to follow what we discussed (base policy + custom one) [09:49:18] <_joe_> elukey: I'll take a look in a few [09:50:08] thanks! [09:54:17] 10serviceops, 10Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 6 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10jijiki) [10:15:54] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve Wikidata traffic via Kubernetes - https://phabricator.wikimedia.org/T347493 (10Lucas_Werkmeister_WMDE) Seems to be working now, thanks a lot for fixing it! `lang=shell-session $ for i in {1..100}; do curl -sIH 'User-Agent: test-Iebdc15b19b (... [10:17:17] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [10:17:37] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve Wikidata traffic via Kubernetes - https://phabricator.wikimedia.org/T347493 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert Yes, confirmed now working. Resolving. [10:26:32] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Reserve resources for system daemons on kubernetes nodes - https://phabricator.wikimedia.org/T277876 (10Clement_Goubert) [10:27:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move 10% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T346422 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert We are now serving 10% of global appserver requests from mw-on-k8s \o/... [10:51:03] <_joe_> elukey: <3 [10:51:06] <_joe_> I love it [10:51:08] <_joe_> thanks [10:51:14] <_joe_> I know I can be obnoxious [11:40:51] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [12:03:02] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [12:35:07] _joe_ <3 [12:35:30] you are not, the hint about having a sane default is a good point, I didn't think about it [12:36:21] jayme: o/ for the fixuture, should I just copy ingress_enabled? [12:36:29] I thought about it but didn't know if it was ok etc.. [12:37:08] elukey: where might as well be another fixture that enables ingress already which you yould use - if not, then make a copy, yes [12:37:28] ideally with defaults as much as possible [12:38:04] (because it will be yet another bunch of diff lines in CI output :)) [12:38:06] there is ingress_default, very basic [12:38:24] ah, yes. then I'd say just enable default cors there [12:38:40] dosn't matter really as long as both cases are rendered somewhere [12:40:17] jayme: done! [12:40:54] <3 thanks [12:41:16] sure thing! [12:43:32] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint), 10Patch-For-Review: Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Daimona) Hi @Clement_Goubert! Is there anything you n... [13:30:06] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint), 10Patch-For-Review: Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Clement_Goubert) Hi @Daimona I'm looking for a review... [13:33:45] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint), 10Patch-For-Review: Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Daimona) >>! In T339984#9206785, @Clement_Goubert wro... [13:50:55] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint), 10Patch-For-Review: Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Clement_Goubert) >>! In T339984#9206799, @Daimona wro... [13:59:34] cors deployed to ores legacy, worked perfectly [13:59:42] thanks for the brainbounce and reviews [14:00:14] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint), 10Patch-For-Review: Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Daimona) >>! In T339984#9206894, @Clement_Goubert wro... [14:11:11] nice! [14:53:40] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint): Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Daimona) 05In progress→03Stalled @Clement_Goubert Thanks! I'll mark th... [14:58:00] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint): Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Clement_Goubert) It's merged, I'm waiting for an unrelated fix to go throug... [15:02:40] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint): Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Clement_Goubert) It's now deployed in production. [16:34:20] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10Jdforrester-WMF) 05Open→03In progress