[04:38:47] 10serviceops, 10MediaWiki-General, 10SRE, 10MW-1.35-notes (1.35.0-wmf.28; 2020-04-14), and 5 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Izno) The change was noticed at https://en.wikipedia.org/wiki/WP:VPT#Aut... [09:07:00] 10serviceops, 10MediaWiki-General, 10SRE, 10MW-1.35-notes (1.35.0-wmf.28; 2020-04-14), and 5 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10TheDJ) https://en.wikipedia.org/wiki/WP:VPT#Automatically_renamed_users... [10:03:39] 10serviceops, 10MediaWiki-General, 10SRE, 10MW-1.35-notes (1.35.0-wmf.28; 2020-04-14), and 5 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Joe) >>! In T219279#7488442, @TheDJ wrote: > https://en.wikipedia.org/wi... [14:12:45] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10Jgiannelos) [14:17:26] 10serviceops, 10Patch-For-Review, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dbrant) @Dzahn Does this kind of change get merged/deployed in the same way as other config chan... [14:18:28] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10Jgiannelos) @Jelto Could this be related to https://phabricator.wikimedia.org/T292390 ? [14:38:13] 10serviceops, 10Community-Tech, 10SRE, 10wikidiff2, and 2 others: Deploy wikidiff2 1.13.0 - https://phabricator.wikimedia.org/T285857 (10NRodriguez) 05Open→03Resolved a:03NRodriguez 🪄🧞‍♀️ Thanks everyone for your great work on this cross-departmental coordination and deployment! [14:56:16] 10serviceops, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech, and 2 others: Deploy Flink (rdf-streaming-updater) to kubernetes (k8s) - https://phabricator.wikimedia.org/T264006 (10Gehel) [15:09:45] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10JMeybohm) From looking at the envoy logs from that particular Pod I'd assume that envoy was not up/ready when tegola tried to connect. You should be able to wait for it to be up b... [15:13:34] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10Joe) >>! In T295290#7489538, @JMeybohm wrote: > From looking at the envoy logs from that particular Pod I'd assume that envoy was not up/ready when tegola tried to connect. > You... [15:15:36] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10JMeybohm) >>! In T295290#7489559, @Joe wrote: > I'm not sure you are allowed to reach `/healthz` via that port, which is the public one and is restricted to reach `/stats` IIRC. T... [15:40:18] 10serviceops, 10MediaWiki-General, 10SRE, 10MW-1.35-notes (1.35.0-wmf.28; 2020-04-14), and 5 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Tol) (I was the one at the Village Pump.) It looks like the renames didn... [15:42:22] <_joe_> Pchelolo: ^^ [15:59:03] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10Jgiannelos) Is it something that can be fixed in the k8s level or the job script should be orchestrated to wait for some time for envoy to be ready and fail in case of timeout? [17:01:11] 10serviceops, 10Data-Engineering, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10odimitrijevic) p:05Triage→03Low [17:18:35] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10JMeybohm) >>! In T295290#7489726, @Jgiannelos wrote: > Is it something that can be fixed in the k8s level or the job script should be orchestrated to wait for some time for envoy... [17:28:47] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10Jgiannelos) Yeah i was wondering if there is some sort of dependency between containers in a pod. Thanks will add a check on the script. [17:30:24] 10serviceops, 10Data-Engineering, 10Platform Engineering, 10Wikibase change dispatching scripts to jobs: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620 (10Michael) >>! In T291620#7483230, @Ottomata wrote: > @Ladsgroup @Michael let us know if having the MW... [17:45:25] 10serviceops, 10Maps: tegola-vector-tiles doesnt execute new tile pregeneration jobs - https://phabricator.wikimedia.org/T295290 (10Jgiannelos) Meanwhile, can somebody help me force exit the hung pod? I don't think I have access. We only allow one job instance to run (to avoid overlaps) and k8s wont let new jo... [18:20:17] 10serviceops, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Legoktm) [18:20:48] 10serviceops, 10SRE, 10MW-1.38-notes (1.38.0-wmf.6; 2021-10-26), 10Patch-For-Review, 10Sustainability: Jobrunner timeouts on cross-DC file uploads because of HTTP/2 - https://phabricator.wikimedia.org/T275752 (10Legoktm) 05Open→03Resolved I think all the technical work is done, please re-open if I mi... [19:17:42] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) Here's the plan to migrate codfw Kubernetes to `helm3` as well: * Announce maintenance some days ahead on ops list * Downtime Kubernetes services in codfw ` is this needed? is t... [20:06:04] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Legoktm) >>! In T251305#7490456, @Jelto wrote: > Some things which are not clear to me: > * Is this the right approach to use the `sre.switchdc.services` cookbook? Kind of, do note tha... [22:34:19] 10serviceops, 10SRE, 10Traffic, 10Performance-Team (Radar): Reconcile MediaWiki POST timeout and Varnish/ATS timeouts - https://phabricator.wikimedia.org/T294800 (10colewhite) p:05Triage→03Medium [22:35:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Doing): Reduce latency of new Scap releases - https://phabricator.wikimedia.org/T292646 (10colewhite) p:05Triage→03Medium