[07:03:44] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Infrastructure-Foundations, and 5 others: Half a million of CirrusSearch jobqueue execution errors per hour since 2021-09-30 16:02 - https://phabricator.wikimedia.org/T292291 (10Aklapper) All patches merged. Is this still an issue? Should this still remai... [08:40:29] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10Wikipedia-iOS-App-Backlog: push-notifications: Validate APNS and FCM credentials on startup - https://phabricator.wikimedia.org/T307252 (10JMeybohm) >>! In T307252#7897176, @MattCleinman wrote: > Happy to help with this how... [09:03:21] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [09:03:55] 10serviceops, 10Prod-Kubernetes, 10SRE, 10Traffic, and 2 others: service::catalog entries and dnsdisc for Kubernetes services under Ingress - https://phabricator.wikimedia.org/T305358 (10JMeybohm) 05Open→03Resolved a:03JMeybohm This is now used by miscweb and documented at https://wikitech.wikimedia.... [09:10:51] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) I finally managed to verify and document the steps needed to put a service under Ingress. I did also update the general https://wikitech.wikimedia.or... [14:56:41] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) p:05Triage→03High a:03Jelto I would suggest to treat gitlab-runner hosts a little different. For the Runner hosts we can basically can //... [15:04:28] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464 (10jhathaway) @Dzahn I mentioned over email, but I t... [15:05:42] 10serviceops, 10GitLab (Infrastructure): GitLab Security Release: 14.10.1, 14.9.4, and 14.8.6 - https://phabricator.wikimedia.org/T307471 (10MoritzMuehlenhoff) [15:06:49] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) [15:08:40] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) 05Open→03Resolved This is done with miscweb being the first full Ingress service and datahub following up. Docs at https://wikitech.wikimedia.org/w... [15:08:45] 10serviceops, 10MW-on-K8s, 10SRE: Create a gateway in kubernetes for the execution of our "lambdas" - https://phabricator.wikimedia.org/T261277 (10JMeybohm) [15:15:50] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464 (10Dzahn) @jhathaway Yes and no. What I definitely d... [15:45:55] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464 (10jhathaway) @Dzahn that makes sense, so I assume i... [16:26:03] I assume new services using ingress probably shouldn't page right? I've gone and changed the default here https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress#Create_an_entry_in_the_service::catalog [16:26:53] I know the service state will protect us against blips in that regard but still [16:34:32] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464 (10Dzahn) The part that we don't (can't actually) re... [19:49:32] 10serviceops, 10SRE: Service Ops SRE support for iOS notifications update - https://phabricator.wikimedia.org/T306397 (10Tsevener) Hi team, Just a heads up - we are planning on going out on phased release with this today. Here are the rollout percentages we can expect for automatic updates. We won't be able t... [21:27:21] 10serviceops, 10SRE: Service Ops SRE support for iOS notifications update - https://phabricator.wikimedia.org/T306397 (10Dzahn) [21:27:23] 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10Wikipedia-iOS-App-Backlog: push-notifications: Validate APNS and FCM credentials on startup - https://phabricator.wikimedia.org/T307252 (10Dzahn) [21:28:59] 10serviceops, 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464 (10Dzahn) But what isn't is that there seems to be a... [22:37:42] 10serviceops, 10Generated Data Platform, 10Image-Suggestions, 10SRE, and 3 others: New Service Request Generated Datasets: Image Suggestions Service - https://phabricator.wikimedia.org/T304891 (10Dzahn) ` OK - authdns-update successful on all nodes! [authdns1001:~] $ host image-suggestion.discovery.wmnet... [22:43:39] 10serviceops, 10Generated Data Platform, 10Image-Suggestions, 10SRE, and 3 others: New Service Request Generated Datasets: Image Suggestions Service - https://phabricator.wikimedia.org/T304891 (10Dzahn) [23:36:11] 10serviceops, 10Generated Data Platform, 10Image-Suggestions, 10SRE, and 3 others: New Service Request Generated Datasets: Image Suggestions Service - https://phabricator.wikimedia.org/T304891 (10Dzahn) @WDoranWMF @hnowlan docs have been updated. https://wikitech.wikimedia.org/wiki/Kubernetes#Add_a_new_...