[05:50:17] 10serviceops, 10mwcli: Create /nonexistent direcotry for nobody user in golang images - https://phabricator.wikimedia.org/T331209 (10Joe) Let me track back one level: why are we getting the error at all? Looking at the logs, it seems that `/builds/repos/releng/cli` is not owned by the user nobody, or is world... [08:19:16] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10fgiunchedi) [08:45:55] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MoritzMuehlenhoff) [09:32:42] 10serviceops, 10MW-on-K8s, 10Wikidata, 10wdwb-tech: Migrate testwikidata to Kubernetes - https://phabricator.wikimedia.org/T331268 (10Clement_Goubert) [09:33:19] 10serviceops, 10MW-on-K8s, 10Wikidata, 10wdwb-tech: Migrate testwikidata to Kubernetes - https://phabricator.wikimedia.org/T331268 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium [09:36:36] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: etcd cluster reimage strategies to use with the K8s upgrade cookbook - https://phabricator.wikimedia.org/T330060 (10elukey) To summarize, I think that we have two options: * Add support to sp... [09:36:39] 10serviceops, 10mwcli: Create /nonexistent direcotry for nobody user in golang images - https://phabricator.wikimedia.org/T331209 (10akosiaris) For what it's worth, it's called `/nonexistent` to point out that it does not exist. it would be completely counter intuitive (as well as breaking the logic it is like... [09:42:36] 10serviceops, 10MW-on-K8s, 10Traffic, 10Wikidata, and 2 others: Migrate testwikidata to Kubernetes - https://phabricator.wikimedia.org/T331268 (10Clement_Goubert) [10:32:50] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update wikikube eqiad to k8s 1.23 - https://phabricator.wikimedia.org/T331126 (10akosiaris) [10:56:34] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update wikikube eqiad to k8s 1.23 - https://phabricator.wikimedia.org/T331126 (10akosiaris) [10:58:57] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update wikikube eqiad to k8s 1.23 - https://phabricator.wikimedia.org/T331126 (10akosiaris) [11:01:10] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update wikikube eqiad to k8s 1.23 - https://phabricator.wikimedia.org/T331126 (10akosiaris) Adding @ottomata too in case we have the same issue as T329664#8638499 [11:04:54] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update wikikube eqiad to k8s 1.23 - https://phabricator.wikimedia.org/T331126 (10Ottomata) Ping @EBernhardson @mforns @dcausse ^ [11:17:05] 10serviceops, 10SRE: kubernetes102[34] implemetation tracking - https://phabricator.wikimedia.org/T313874 (10jijiki) a:05akosiaris→03jijiki [11:24:04] akosiaris: mw24[20-51] are supposed to be what appserver role? [11:25:18] 10serviceops, 10mwcli: Create /nonexistent direcotry for nobody user in golang images - https://phabricator.wikimedia.org/T331209 (10Addshore) > the fact that user nobody has no homedir is a deliberate security measure. Ah, right, in that case let me see if I can find a work around I imagine such a work around... [11:27:10] 10serviceops, 10mwcli: Create /nonexistent direcotry for nobody user in golang images - https://phabricator.wikimedia.org/T331209 (10Addshore) > Looking at the logs, it seems that /builds/repos/releng/cli is not owned by the user nobody, or is world-writable I imagine (but didnt look) that it has the ownershi... [11:56:59] 10serviceops, 10SRE, 10Datacenter-Switchover: 28 February 2023 Service Switchover checklist - https://phabricator.wikimedia.org/T330651 (10Clement_Goubert) [11:58:13] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [12:01:09] 10serviceops, 10Traffic, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [12:01:56] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) p:05Triage→03High [12:04:05] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [12:05:31] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [12:05:41] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [12:16:43] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [12:46:27] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10jnuche) Hi @hnowlan and @MoritzMuehlenhoff I've tentatively created a patch to try to address the problem with Scap updates and added you as reviewers: https://gerr... [13:24:19] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [13:31:41] 10serviceops, 10Patch-For-Review: Redirect docker-registry URLs with tags in them to the static /tags/ HTML page - https://phabricator.wikimedia.org/T283764 (10JMeybohm) 05Open→03Declined I would rather not add another rule to the already quite complex nginx setup for docker registry if we're not strictly... [13:38:03] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: etcd cluster reimage strategies to use with the K8s upgrade cookbook - https://phabricator.wikimedia.org/T330060 (10JMeybohm) >>! In T330060#8667488, @elukey wrote: > I'd be in favor of option... [14:28:18] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10akosiaris) @jnuche, once we figure out T328033 and thus are able to complete T233196, thumbor will have nothing to do with scap. It's probably not worth solving thi... [14:34:39] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [14:42:16] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MatthewVernon) [14:57:42] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10eoghan) [14:58:53] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Decide on new Pod and Sevice IPv4 ranges for wikikube clusters - https://phabricator.wikimedia.org/T326617 (10akosiaris) 10.192.64.0/21 removed from homer and netbox. I 'll clean up tomorrow puppet too. [15:14:06] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [15:37:05] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Jelto) [16:24:49] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: March 2023 Datacenter Switchover Excluded services - https://phabricator.wikimedia.org/T329193 (10Gehel) [16:37:30] 10serviceops, 10Data-Engineering-Planning, 10Event-Platform Value Stream: k8s deployment-charts mesh module should allow use of mesh without public_port Service - https://phabricator.wikimedia.org/T326252 (10akosiaris) I guess we can close this one? [16:49:58] 10serviceops, 10Kubernetes: WMF helmfile installation does not work for ZSH users - https://phabricator.wikimedia.org/T277096 (10JMeybohm) @CDanis is this still a thing? [16:52:37] 10serviceops, 10SRE: mw2420-mw2451 service implementation tracking - https://phabricator.wikimedia.org/T326363 (10RLazarus) @akosiaris and @Clement_Goubert will come up with a cluster layout this week, and @Clement_Goubert wanted to try putting at least one or two into service themselves. Feel free to assign t... [17:24:26] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10herron) [17:34:22] 10serviceops, 10MW-on-K8s, 10Traffic: Insert a header for specific domains at the first ATS layer to redirect traffic to mw-on-k8s - https://phabricator.wikimedia.org/T331318 (10Clement_Goubert) [17:34:50] 10serviceops, 10MW-on-K8s, 10Traffic: Insert a header for specific domains at the first ATS layer to redirect traffic to mw-on-k8s - https://phabricator.wikimedia.org/T331318 (10Clement_Goubert) [17:35:00] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [17:35:27] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ssingh) [17:35:49] 10serviceops, 10MW-on-K8s, 10Traffic: Insert a header for specific domains at the first ATS layer to redirect traffic to mw-on-k8s - https://phabricator.wikimedia.org/T331318 (10Clement_Goubert) p:05Triage→03Medium [17:38:31] 10serviceops, 10MW-on-K8s, 10SRE, 10observability: Logging options for apache httpd in k8s - https://phabricator.wikimedia.org/T265876 (10Clement_Goubert) 05Open→03Resolved [17:38:41] 10serviceops, 10MW-on-K8s, 10SRE: Create the base container images for running MediaWiki in a production environment - https://phabricator.wikimedia.org/T265324 (10Clement_Goubert) [17:38:48] 10serviceops, 10Observability-Tracing: Helmchart for OpenTelemetry Collector - https://phabricator.wikimedia.org/T324117 (10Clement_Goubert) 05In progress→03Stalled [17:38:52] 10serviceops, 10Observability-Tracing: OpenTelemetry Collector running as a DaemonSet on Wikikube - https://phabricator.wikimedia.org/T320564 (10Clement_Goubert) [17:39:32] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: Post March 2023 Datacenter Switchover Tasks - https://phabricator.wikimedia.org/T328907 (10Clement_Goubert) p:05Triage→03Medium [17:46:39] 10serviceops: Migrate mediawiki_http_requests alerts to AlertManager - https://phabricator.wikimedia.org/T325277 (10Clement_Goubert) 05Open→03Resolved Done in https://gerrit.wikimedia.org/r/c/operations/alerts/+/883950 and https://gerrit.wikimedia.org/r/c/operations/alerts/+/883502/4 [18:28:14] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10akosiaris) >>! In T226931#8665381, @Brycehughes wrote: > @akosiaris absolutely no worries on respo... [19:29:11] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Jaifroid) Just to report, as promised, that the Kiwix ZIM files of Wiktionary are now reflecting t... [19:58:49] 10serviceops, 10mwcli: Create /nonexistent direcotry for nobody user in golang images - https://phabricator.wikimedia.org/T331209 (10Addshore) On the gitlab runner / CI side of things, it looks like choosing a user for the steps in CI is not possible, per https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27... [20:02:09] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10herron) [23:19:57] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=786ee8c7-4753-4e2d-96f9-8b55b691ff09) set by bking@cumin2002 for 1 day, 0:00:00 o... [23:21:02] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f9f1bd07-4af1-41e3-82b7-3ab0f2ff8672) set by bking@cumin2002 for 1 day, 0:00:00 o... [23:22:27] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10bking) [23:25:18] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10RKemper) [23:28:22] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Dzahn)