[06:11:49] <kart_>	 Hi, looking for review of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/965022 - it has restbase endpoint config update. 
[07:33:01] <elukey>	 hnowlan: o/ I have an interesting use case for CORS headers on Lift Wing, namely for kserve-based-services (not using the serviceops modules etc..) I cannot easily add the istio config to return CORS headers in the HTTP response, so I am wondering if it would be possible to add the feature to the API-Gateway in some way. Is there anything already implemented by any chance? (I checked briefly 
[07:33:08] <elukey>	 but didn't find much)
[07:33:20] <elukey>	 I'd basically need to inject base CORS headers for the various lift wing backends
[07:34:01] <elukey>	 (for everybody wondering - really Luca? Why isn't istio config injection supported in kserve? https://github.com/kserve/kserve/issues/721)
[07:45:21] <wikibugs>	 10serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team, 10MediaWiki-extensions-CentralAuth, and 5 others: MediaWiki\Extension\Notifications\Api\ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki} - https://phabricator.wikimedia.org/T342201 (10Clement_Goub...
[07:45:27] <wikibugs>	 10serviceops, 10Growth-Team, 10Growth-Team-Filtering, 10MW-on-K8s, 10Notifications: Broken (empty) cross-wiki notification when using $wgLocalHTTPProxy (e.g. on Kubernetes) - https://phabricator.wikimedia.org/T223413 (10Clement_Goubert)
[08:10:43] <elukey>	 hnowlan: self answered - I see in _apigateway_routes.tpl that we already use response_headers_to_add for cache-control, so I can add a new option for base cors headers in theory
[08:53:30] <elukey>	 of course is more complicated than this, I need to add a filter
[09:05:46] <wikibugs>	 10serviceops, 10PageViewInfo: Daily pageview/PageViewInfo errors on jobrunners - https://phabricator.wikimedia.org/T348517 (10hashar) As part of those errors, MediaWiki autoloader fails to find DBConnectionError.php or HttpStatus.php which I have filed as an independent task T348614 since I think it is an issu...
[09:59:32] <hnowlan>	 elukey: yeah or add an option of per-route headers to add in the config 
[11:03:10] <wikibugs>	 10serviceops, 10Prod-Kubernetes: KubernetesAPILatency alert fires on scap deploy - https://phabricator.wikimedia.org/T348228 (10Clement_Goubert) 05Open→03Resolved The alert thresholds were changed by @elukey in https://gerrit.wikimedia.org/r/c/operations/alerts/+/964534 Resolving this task as it is about t...
[11:26:41] <wikibugs>	 10serviceops, 10SRE, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review: Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Volans) 05Resolved→03Open FYI The service IPs in Netbox are still allocated to the service and probably needs cleanup: https://netbox.wikimedia.org/ipam...
[12:01:27] <wikibugs>	 10serviceops: k8s-ingress-aux.svc.codfw.wmnet marked as Active in Netbox - https://phabricator.wikimedia.org/T348632 (10Volans) I noticed also that `aux-k8s-ctrl.svc.eqiad.wmnet` is missing the PTR record in the operations/dns repository.
[12:11:08] <elukey>	 hnowlan: I thought about the per-route headers to add but I think that CORS are added only in certain conditions, the filter may be more inline with what envoy expects.. I'll try to file a patch today :)
[12:26:11] <wikibugs>	 10serviceops: tegola-vector-tiles SVC records missing reverse PTRs - https://phabricator.wikimedia.org/T348631 (10Clement_Goubert) a:03Clement_Goubert
[12:39:07] <elukey>	 folks I filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/965124 to "free" the ORES redis instances 
[12:39:59] <elukey>	 my idea is just to remove special configs, flush the instances allocated for ORES to drop their data and mark the ports in wikitech as "unassigned" (https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc)
[12:40:03] <elukey>	 does it make sense?
[12:42:31] <wikibugs>	 10serviceops, 10Discovery-Search, 10SRE, 10collaboration-services, and 2 others: Sunset search.wikimedia.org service - https://phabricator.wikimedia.org/T316296 (10Clement_Goubert) 05Open→03Resolved Done
[12:44:10] <wikibugs>	 10serviceops, 10SRE, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review: Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert Done
[13:07:56] <elukey>	 hnowlan: sorry I just realized that we do have a base CORS config :D The error is indeed related to an extra header to allow, filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/965153. Lemme know if it makes sense!
[13:13:43] <wikibugs>	 10serviceops, 10collaboration-services, 10GitLab (CI & Job Runners): Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10MatthewVernon) Documentation has now been updated in the main namespace - https://wikitech.wikimedia.org/wiki/Debian_packaging_with_dgit_and_CI an...
[13:23:31] <hnowlan>	 elukey: thanks! I'll give it a look later 
[13:43:46] <wikibugs>	 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, and 2 others: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10Jdforrester-WMF)
[14:23:05] <wikibugs>	 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, and 2 others: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm)
[14:29:34] <wikibugs>	 10serviceops, 10Patch-For-Review: tegola-vector-tiles SVC records missing reverse PTRs - https://phabricator.wikimedia.org/T348631 (10Clement_Goubert) 05Open→03Resolved
[14:36:08] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Create kube-state-metrics docker image - https://phabricator.wikimedia.org/T343801 (10kamila) 05In progress→03Resolved Image built and published.
[14:36:10] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10User-jijiki: Deploy kube-state-metrics - https://phabricator.wikimedia.org/T264625 (10kamila)
[14:42:51] <wikibugs>	 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[14:43:01] <wikibugs>	 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Move 25% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T348122 (10Clement_Goubert) 05Open→03In progress
[15:02:25] <elukey>	 jayme: my team is in the process of adding a new namespace and not having to mess with the puppet private is awesome. Thanks for the client-auth cert stuff <3
[15:03:27] <jayme>	 🤗
[15:06:37] <klausman>	 Wat Luca said 
[15:07:04] <jayme>	 I just added one as well and forgot about adding the service entry in deployment_server hiera :D - that needs to go away as well
[15:38:42] <wikibugs>	 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, and 2 others: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm)
[18:36:59] <wikibugs>	 10serviceops, 10Abstract Wikipedia team, 10function-evaluator, 10Patch-For-Review: Split the monolithic function-evaluator service up in production so we have differently-scalable pods for python vs. node - https://phabricator.wikimedia.org/T343388 (10Jdforrester-WMF) Deployed.
[21:38:54] <wikibugs>	 10serviceops, 10Growth-Team, 10MW-on-K8s, 10MediaWiki-Platform-Team, and 5 others: MediaWiki\Extension\Notifications\Api\ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki} - https://phabricator.wikimedia.org/T342201 (10KStoller-WMF)
[21:39:16] <wikibugs>	 10serviceops, 10Abstract Wikipedia team, 10Wikifunctions, 10Wikimedia-production-error: Wikifunctions functions that require a lookup on wikifunctions.org timing out in the orchestrator, UX instead showing 'http' - https://phabricator.wikimedia.org/T344998 (10Etonkovidova) 05Open→03Resolved Checked all...