[01:38:18] 10serviceops, 10MediaWiki-Core-Snapshots, 10Wikimedia-General-or-Unknown: Reproducible HTTP 503 error trying to import from Telugu wikipedia to Telugu Wikibooks - https://phabricator.wikimedia.org/T283472 (10Arjunaraoc) I tried again today and got the following error "If you report this error to the Wikime... [02:13:27] 10serviceops, 10MediaWiki-Core-Snapshots, 10Wikimedia-General-or-Unknown: Reproducible HTTP 503 error trying to import from Telugu wikipedia to Telugu Wikibooks - https://phabricator.wikimedia.org/T283472 (10Platonides) So you are trying to import https://te.wikipedia.org/wiki/%E0%B0%B8%E0%B0%BE%E0%B0%AE%E0%... [03:58:50] 10serviceops, 10MediaWiki-Core-Snapshots, 10Wikimedia-General-or-Unknown: Reproducible HTTP 503 error trying to import from Telugu wikipedia to Telugu Wikibooks - https://phabricator.wikimedia.org/T283472 (10Arjunaraoc) Yes, Afterwards I was able to import with full history about 40 pages. Again I am faci... [07:17:50] 10serviceops, 10SRE, 10docker-pkg: Refresh all images in production-images - https://phabricator.wikimedia.org/T284431 (10Joe) 05Open→03Resolved I don't think the changelog-creating part should really be part of this task. What we've done now should be enough for the original intended scope of the task.... [10:24:07] 10serviceops, 10MW-on-K8s, 10SRE: Create a gateway in kubernetes for the execution of our "lambdas" - https://phabricator.wikimedia.org/T261277 (10Joe) >>! In T261277#7161333, @Legoktm wrote: > @joe is everything in this ticket now covered by Shellbox? No, this ticket is about adding an Ingress controller i... [12:59:05] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate contour as an ingress - https://phabricator.wikimedia.org/T286196 (10Joe) [13:01:22] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate nginx-controller as an Ingress - https://phabricator.wikimedia.org/T286197 (10Joe) [13:02:32] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate nginx-controller as an Ingress - https://phabricator.wikimedia.org/T286197 (10Joe) [13:53:03] 10serviceops, 10MW-on-K8s, 10SRE: Benchmark performance of MediaWiki on k8s - https://phabricator.wikimedia.org/T280497 (10Joe) >>! In T280497#7194633, @wkandek wrote: > Is this the dashboard? https://grafana.wikimedia.org/d/U7JT--knk/joe-k8s-mwdebug?viewPanel=70&orgId=1&from=1625227688488&to=1625246654342... [13:54:15] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate contour as an ingress - https://phabricator.wikimedia.org/T286196 (10Joe) a:03Joe **Architecture:** Contour works decoupling the management layer (contour itself) from the proxying one (using envoy): the first is deployed as a deployment in kubernetes, and needs t... [14:10:31] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate contour as an ingress - https://phabricator.wikimedia.org/T286196 (10Joe) **Docker images needed**: We get to pick two directions: * If we use the helm chart, we would need a recent version of envoy (easy) and one image for contour. * If we want to use the operatior... [14:15:29] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate contour as an ingress - https://phabricator.wikimedia.org/T286196 (10Joe) **Deployment** There are two options, similar to istio: * use contour-operator, which also allows to control the runtime status of the cluster (for instance allowing zero-downtime envoy upgrad... [16:46:00] The docker-registry.wikimedia.org/nodejs10-devel container pulled from the registry seems to suddenly have no `npm` binary installed. I'm not sure when this happened, but sometime between 2021-07-02 and now. https://gerrit.wikimedia.org/r/c/wikimedia/toolhub/+/702451/ shows CI working on 2021-07-02 and now failing due to the base image contents. [16:47:04] it looks like `0.0.3-20210704` is the timestamped version of `latest` there [16:48:26] * legoktm looks [16:49:09] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/nodejs10-devel/stretch/Dockerfile.template [16:49:33] I guess nodesource's "nodejs" no longer recommends/requires npm [16:51:43] no, it used to be in the package [16:53:08] legoktm: should I start a phab bug for this, or have you already? [16:53:27] yes please [16:58:47] 10serviceops, 10Release Pipeline: docker-registry.wikimedia.org/nodejs10-devel container after 0.0.2 does not include `npm` - https://phabricator.wikimedia.org/T286212 (10bd808) [17:02:25] oh wow it's even worse [17:02:34] root@7f402e8aff72:/# nodejs --version [17:02:34] v6.11.0 [17:04:10] 10serviceops, 10Release Pipeline: docker-registry.wikimedia.org/nodejs10-devel container after 0.0.2 does not include `npm` - https://phabricator.wikimedia.org/T286212 (10Legoktm) a:03Legoktm For some reason it's now installing nodejs from stretch-wikimedia (`6.11.0~dfsg-1+wmf5`) instead of nodesource. Shoul... [17:04:34] 10serviceops, 10Release Pipeline: docker-registry.wikimedia.org/nodejs10-devel container after 0.0.3 does not include `npm` - https://phabricator.wikimedia.org/T286212 (10Legoktm) [17:09:59] 10serviceops, 10Release Pipeline: docker-registry.wikimedia.org/nodejs10-devel container after 0.0.3 does not include `npm` - https://phabricator.wikimedia.org/T286212 (10bd808) [17:18:10] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate nginx-controller as an Ingress - https://phabricator.wikimedia.org/T286197 (10Legoktm) a:03Legoktm [17:23:01] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate nginx-controller as an Ingress - https://phabricator.wikimedia.org/T286197 (10bd808) @aborrero may be able to provide some information from his past work to setup ingress-nginx for Toolforge. [17:25:28] 10serviceops, 10Release Pipeline, 10Toolhub: docker-registry.wikimedia.org/nodejs10-devel container after 0.0.3 does not include `npm` - https://phabricator.wikimedia.org/T286212 (10bd808) [17:32:44] bd808: btw you may want to migrate to the `buster-nodejs10-devel` image unless there's a reason to be on stretch [17:33:41] as soon as T284112 is done :() [17:34:00] arg. no stashbot [17:34:15] oh uh [17:34:22] Production buster-nodejs10-devel image has npm 5.x, which is not actually compatible with node 10.x -- https://phabricator.wikimedia.org/T284112 [17:43:04] 10serviceops, 10Release Pipeline, 10Toolhub, 10Patch-For-Review: docker-registry.wikimedia.org/nodejs10-devel container after 0.0.3 does not include `npm` - https://phabricator.wikimedia.org/T286212 (10Legoktm) 05Open→03Resolved ` km@cashew ~> podman run --rm -it --entrypoint=bash docker-registry.wikim... [17:44:36] thanks for the super quick fix legoktm :) [17:44:48] yw :) [17:45:29] I don't undersand how all these services are using buster-nodejs10-devel https://codesearch.wmcloud.org/search/?q=buster-nodejs10-devel&i=nope&files=&excludeFiles=&repos= if it's using the wrong npm [17:46:08] I would guess most of them are only using nodejs for banana-checker and similar tests [17:48:28] 10serviceops, 10Release Pipeline: Production buster-nodejs10-devel image has npm 5.x, which is not actually compatible with node 10.x - https://phabricator.wikimedia.org/T284112 (10Legoktm) Is npm 5.x flat out incompatible with node10, or is it more subtle than that? Wondering how all these services work prope... [17:52:48] 10serviceops, 10Release Pipeline: Production buster-nodejs10-devel image has npm 5.x, which is not actually compatible with node 10.x - https://phabricator.wikimedia.org/T284112 (10MoritzMuehlenhoff) >>! In T284112#7198157, @Legoktm wrote: > Is npm 5.x flat out incompatible with node10, or is it more subtle th...