[08:34:29] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar): MediaWiki deploy servers should not be mediawiki installation targets - https://phabricator.wikimedia.org/T329857 (10Clement_Goubert) [08:48:16] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [09:26:22] 10serviceops, 10Infrastructure-Foundations, 10WikimediaDebug, 10Performance-Team (Radar): Upgrade php-excimer package from 1.0.4 to 1.1.1 - https://phabricator.wikimedia.org/T332964 (10MoritzMuehlenhoff) I've uploaded the updated Excimer package to apt.wikimedia.org and went ahead with an upgrade of the mw... [10:02:43] 10serviceops, 10Kubernetes: WMF helmfile installation does not work for ZSH users - https://phabricator.wikimedia.org/T277096 (10JMeybohm) 05Open→03Invalid I'll assume it's not. Feel free to reopen if I'm wrong. [11:26:57] 10serviceops, 10Infrastructure-Foundations, 10SRE: Deal with archival of Stretch on Debian mirrors - https://phabricator.wikimedia.org/T335282 (10MoritzMuehlenhoff) [11:41:33] 10serviceops, 10MW-on-K8s, 10Push-Notification-Service, 10Patch-For-Review: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Jgiannelos) > In normal operation, with a dummy token, does the mw api log an error? Yes [11:54:02] 10serviceops, 10MW-on-K8s, 10Push-Notification-Service, 10Patch-For-Review: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Jgiannelos) On the other hand, i see a lot of connection failures for the MW api requests: https://logstash.wikimedia.org/goto/eba59420e81c8509... [12:07:57] 10serviceops, 10MW-on-K8s, 10Push-Notification-Service, 10Patch-For-Review: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Clement_Goubert) `curl -H 'Host: en.wikipedia.org' http://localhost:6500/w/api.php` works from the pod namespace, so at least it's not an egres... [12:12:24] 10serviceops, 10MW-on-K8s, 10Push-Notification-Service, 10Patch-For-Review: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Jgiannelos) Lets merge the staging config and try the same GET request. From the config it looks like it is using localhost: ` mwapi_re... [12:18:21] 10serviceops, 10MW-on-K8s, 10Push-Notification-Service, 10Patch-For-Review: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Clement_Goubert) [[ https://logstash.wikimedia.org/goto/50f45d9fe0e53f2c739436112708ffb9 | Staging logs ]] [12:34:40] 10serviceops, 10MW-on-K8s, 10Push-Notification-Service: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Clement_Goubert) In `kubectl logs` I can see my message in queue: ` Queue (push): Added item: SingleDeviceMessage {... [13:00:43] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 12), 10Patch-For-Review: New Service Request: flink-kubernetes-operator - https://phabricator.wikimedia.org/T333464 (10JArguello-WMF) [13:00:54] 10serviceops, 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 12), 10Patch-For-Review, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10JArguello-WMF) [13:07:51] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Selected IPv6 service-cluster-up ranges are to big - https://phabricator.wikimedia.org/T335285 (10JMeybohm) [13:09:03] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Selected IPv6 service-cluster-up ranges are to big - https://phabricator.wikimedia.org/T335285 (10JMeybohm) [13:09:15] ^^ "funny" [13:10:46] ~_~ [13:33:43] 10serviceops, 10MW-on-K8s, 10Push-Notification-Service: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Clement_Goubert) 05Open→03Resolved End to end testing delivered the push notification, the test of an invalid token gave the same log as in staging. Considering... [13:33:56] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [13:34:09] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [13:38:20] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Selected IPv6 service-cluster-up ranges are to big - https://phabricator.wikimedia.org/T335285 (10JMeybohm) Given that the code reads that we should keep it "small" and the biggest IPv4 service ip range is `/20` (on the prod ml clusters on purpose. aux and dse c... [14:30:47] TIL staging-codfw? how does that work? [14:37:18] works the same :) [14:37:28] at least for admin_ng stugg [14:37:32] *stuff [14:37:42] for actual service deployments it is "hidden" [14:59:18] ottomata: o/. Happy you learned about it, now for you own sanity, forget about it. Internal implementation detail on our side. Just use staging, we make no guarantees about the state of staging-codfw at any point in time. In fact, I guarantee that if you decided to use it, you 'll find something offputting and weird as we effectively test upgrades [14:59:18] processes there. [15:00:49] but in the special case of flink-operator deployment, please don't forget about it :D [15:15:16] 10serviceops, 10Content-Transform-Team-WIP, 10RESTBase, 10SRE, and 5 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10Jgiannelos) a:03Jgiannelos [15:18:15] okay gr8 tty [15:27:54] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10LSobanski) [15:28:31] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10bking) [15:49:14] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10Clement_Goubert) [16:18:58] 10serviceops, 10Keyholder, 10SRE, 10VPS-project-Codesearch, 10Patch-For-Review: Consider using BindsTo instead of Requires to declare dependencies between systemd unit - https://phabricator.wikimedia.org/T284555 (10BCornwall) [16:39:58] 10serviceops, 10PoolCounter, 10Performance-Team (Radar): poolcounter-exporter upgrade - https://phabricator.wikimedia.org/T333947 (10akosiaris) a:03akosiaris [18:34:16] 10serviceops, 10Shellbox, 10SyntaxHighlight, 10Patch-For-Review, 10User-bd808: Install pygments in Shellbox container with pip, not a Debian package - https://phabricator.wikimedia.org/T320848 (10Iniquity) Hi :) Any progress here? Or is there any information when it will be possible to implement new schema? [19:59:40] 10serviceops, 10Keyholder, 10VPS-project-Codesearch, 10Patch-For-Review: Consider using BindsTo instead of Requires to declare dependencies between systemd unit - https://phabricator.wikimedia.org/T284555 (10BCornwall) 05Open→03Resolved