[00:03:23] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [00:09:07] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 7 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [02:37:32] 10Continuous-Integration-Config, 10Sonarqubebot: SonarQube is unhelpfully suggesting ES6 feautures in ES5 code - https://phabricator.wikimedia.org/T289957 (10Krinkle) [06:23:57] (03CR) 10Jeena Huneidi: WIP Adds kubernetes cluster (035 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [06:54:41] (03PS2) 10Jeena Huneidi: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) [06:58:29] (03CR) 10Jeena Huneidi: WIP Adds kubernetes cluster (034 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [07:52:06] 10Deployments, 10Math, 10Mathoid, 10Platform Team Workboards (Clinic Duty Team): Deploy new Mathoid version to production - https://phabricator.wikimedia.org/T289674 (10Physikerwelt) The urgency was resolved, without the patch. However, in the future people doing the deployment should be able to figure out... [08:03:35] 10Gerrit, 10Release-Engineering-Team, 10GitLab, 10Icinga, and 4 others: RelEng access to downtime alerts in Icinga for gitlab, gerrit, possibly other services? - https://phabricator.wikimedia.org/T289746 (10fgiunchedi) Unless there are objections let's go with (b), do you need command line access or web in... [08:09:06] 10Gerrit, 10Release-Engineering-Team, 10GitLab, 10Icinga, and 4 others: RelEng access to downtime alerts in Icinga for gitlab, gerrit, possibly other services? - https://phabricator.wikimedia.org/T289746 (10fgiunchedi) p:05Triage→03Medium [08:52:06] 10Continuous-Integration-Config, 10Sonarqubebot, 10Developer Productivity: SonarQube is unhelpfully suggesting ES6 feautures in ES5 code - https://phabricator.wikimedia.org/T289957 (10kostajh) [08:56:20] 10Continuous-Integration-Config, 10Sonarqubebot, 10Developer Productivity: SonarQube is unhelpfully suggesting ES6 feautures in ES5 code - https://phabricator.wikimedia.org/T289957 (10kostajh) I don't know if this is still accurate or not, but doesn't seem possible to tell SonarQube that files in a certain d... [09:27:38] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-API, 10MediaWiki-File-management, and 3 others: Usage and linking of a page or file through multiple levels of redirects is not reported by API query for linkshere and fileusage - https://phabricator.wikimedia.org/T289792 (10Aklapper) @AlexisJazz Feel f... [11:12:32] (03PS1) 10Ladsgroup: Add EXT_DEPENDENCIES to quibble-daily jobs [integration/config] - 10https://gerrit.wikimedia.org/r/715470 (https://phabricator.wikimedia.org/T285049) [11:45:16] Hi, I'm struggling to find out why set_parameters() is not being ran on these jobs https://integration.wikimedia.org/ci/job/quibble-daily-ArticlePlaceholder-vendor-mysql-php72-docker/ and thus the dependencies are not set (and they are failing because of it). Is it because its trigger is "daily" and not zuul? [11:45:35] hashar: if/when you have some free time ^ [11:47:20] Amir1: yes, the job is broken [11:47:38] set_parameter is invoked by the Zuul scheduled before triggering the job [11:48:04] and pass some extra parameters to the build [11:49:23] those quibble-daily-* jobs are trigger by Jenkins [11:49:29] and thus do not have any parameters injected [11:49:30] https://gerrit.wikimedia.org/r/c/integration/config/+/703785/8/jjb/mediawiki.yaml [11:50:46] yeah, I just realized that was why I had to set ZUUL parameters there [11:51:10] how can I get it to actually trigger set_parameters (if it's possible?) [11:51:46] to be clear: I won't mess with actual jobs like this, this is a new job so I played with it [12:00:24] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-API, 10MediaWiki-File-management, and 3 others: Usage and linking of a page or file through a redirect is not reported by API query for linkshere and fileusage - https://phabricator.wikimedia.org/T289792 (10AlexisJazz) [12:20:09] !log foreachwikiindblist wikisource refreshImageMetadata.php --mediatype=OFFICE --batch-size=10 --verbose --split --sleep 5 [12:20:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:34:28] 10Release-Engineering-Team (Doing), 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work), 10Release Pipeline (Blubber): PipelineBot for flink-rdf-streaming-updater does not generate proper deployment-chart patches - https://phabricator.wikimedia.org/T287374 (10Gehel) 05Open→03Resolv... [14:45:14] 10Gerrit, 10Release-Engineering-Team, 10GitLab, 10Icinga, and 4 others: RelEng access to downtime alerts in Icinga for gitlab, gerrit, possibly other services? - https://phabricator.wikimedia.org/T289746 (10brennen) > Unless there are objections let's go with (b), do you need command line access or web int... [14:58:25] 10Deployments, 10Math, 10Mathoid, 10Platform Team Workboards (Platform Engineering Reliability): Deploy new Mathoid version to production - https://phabricator.wikimedia.org/T289674 (10WDoranWMF) p:05Triage→03Medium [14:59:26] 10Deployments, 10Math, 10Mathoid, 10Platform Team Workboards (Platform Engineering Reliability): Deploy new Mathoid version to production - https://phabricator.wikimedia.org/T289674 (10WDoranWMF) @hnowlan Is going to take a look at the docs and fix them this week. After discussing with @thcipriani, this... [15:02:57] (03CR) 10Ahmon Dancy: [C: 03+1] Ensure dependencies from `includes` are present in the CopiesDepGraph [blubber] - 10https://gerrit.wikimedia.org/r/715276 (https://phabricator.wikimedia.org/T289880) (owner: 10Dduvall) [15:13:20] (03PS3) 10Jforrester: jjb, Zuul: [mediawiki/services/apple-search] Add pipeline CI [integration/config] - 10https://gerrit.wikimedia.org/r/715317 (https://phabricator.wikimedia.org/T289224) (owner: 10Majavah) [15:14:16] (03CR) 10Jforrester: [C: 03+2] "JJB part deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/715317 (https://phabricator.wikimedia.org/T289224) (owner: 10Majavah) [15:15:32] (03Merged) 10jenkins-bot: jjb, Zuul: [mediawiki/services/apple-search] Add pipeline CI [integration/config] - 10https://gerrit.wikimedia.org/r/715317 (https://phabricator.wikimedia.org/T289224) (owner: 10Majavah) [15:16:45] !log Zuul: [mediawiki/services/apple-search] Add pipeline CI for T289224 [15:16:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:16:49] T289224: Split search.wikimedia.org out of ops/mediawiki-config into separate service - https://phabricator.wikimedia.org/T289224 [15:20:09] (03PS1) 10Jforrester: Docker: Drop quibble-stretch-php72 and quibble-stretch, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/715539 [15:44:50] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:07:00] (03PS1) 10Subramanya Sastry: Coerce numeric titles to strings [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715549 [16:07:02] (03PS1) 10Subramanya Sastry: More Cite CSS support for plwiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715550 [16:23:36] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:24:17] (03CR) 10Legoktm: jjb, Zuul: [mediawiki/services/apple-search] Add pipeline CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/715317 (https://phabricator.wikimedia.org/T289224) (owner: 10Majavah) [16:25:05] (03PS1) 10Majavah: zuul: [mediawiki/services/apple-search] Remove composer-test-package-php72-or-later [integration/config] - 10https://gerrit.wikimedia.org/r/715553 [16:25:31] (03CR) 10Majavah: jjb, Zuul: [mediawiki/services/apple-search] Add pipeline CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/715317 (https://phabricator.wikimedia.org/T289224) (owner: 10Majavah) [16:26:31] (03CR) 10jerkins-bot: [V: 04-1] zuul: [mediawiki/services/apple-search] Remove composer-test-package-php72-or-later [integration/config] - 10https://gerrit.wikimedia.org/r/715553 (owner: 10Majavah) [16:29:40] (03PS2) 10Majavah: zuul: [mediawiki/services/apple-search] Remove composer-package [integration/config] - 10https://gerrit.wikimedia.org/r/715553 [16:33:36] (03CR) 10Jforrester: [C: 03+2] "I was surprised. :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/715553 (owner: 10Majavah) [16:34:11] majavah: :-) [16:35:29] thanks and sorry James_F [16:35:39] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/apple-search] Remove composer-package [integration/config] - 10https://gerrit.wikimedia.org/r/715553 (owner: 10Majavah) [16:35:54] No worries. My fault for not actually whining but just going "huh, that's funny… I guess majavah knows what they're doing". ;-) [16:36:30] !log Zuul: [mediawiki/services/apple-search] Remove composer-package [16:36:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:01] as a fun fact, ever since I got +2 I've been more careful before sending mw things to review because I assume others assume that (in reality I mostly don't) [16:37:12] legoktm: I think we'll need to force merge https://gerrit.wikimedia.org/r/c/mediawiki/services/apple-search/+/715314/3, as the pipeline is only added on the next patch that patch will never pass it [16:37:54] * James_F grins. [16:38:01] ack [16:38:29] also, I've definitely been assuming that for way longer than when you got +2 [16:38:37] Indeed. [16:42:08] (03PS2) 10Jforrester: Docker: Drop quibble-stretch-php72 and quibble-stretch, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/715539 (https://phabricator.wikimedia.org/T252432) [16:42:27] (03CR) 10Jforrester: [C: 03+2] Docker: Drop quibble-stretch-php72 and quibble-stretch, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/715539 (https://phabricator.wikimedia.org/T252432) (owner: 10Jforrester) [16:43:57] (03Merged) 10jenkins-bot: Docker: Drop quibble-stretch-php72 and quibble-stretch, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/715539 (https://phabricator.wikimedia.org/T252432) (owner: 10Jforrester) [16:55:41] (03PS1) 10Jforrester: dockerfiles: Provide node12-test-browser-php{72,80}-composer images [integration/config] - 10https://gerrit.wikimedia.org/r/715556 [17:02:32] majavah: oh, you also need a postmerge -publish job so the build image is pushed to the docker registry [17:02:39] built* [17:02:54] I thought I added that? [17:03:19] no, just test/gate-and-submit [17:03:39] you added the publish job to jjb, but didn't add it as postmerge in zuul [17:04:41] (03PS1) 10Majavah: zuul: [mediawiki/services/apple-search] Add postmerge publish [integration/config] - 10https://gerrit.wikimedia.org/r/715564 [17:04:45] argh, there [17:23:53] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Platform Engineering Roadmap, 10User-Daniel: Convert static mediawiki configuration to form more suitable for containers - https://phabricator.wikimedia.org/T263166 (10daniel) Pinging @hnowlan for contextual awareness. [17:58:32] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 2 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10J... [18:23:39] !log gitlab-test: associating floating IP to primary test box [18:23:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:57:25] (03CR) 10Arlolra: [V: 03+2 C: 03+2] Coerce numeric titles to strings [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715549 (owner: 10Subramanya Sastry) [18:57:52] (03CR) 10Arlolra: [V: 03+2 C: 03+2] More Cite CSS support for plwiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715550 (owner: 10Subramanya Sastry) [19:09:19] (03CR) 10Jforrester: [C: 03+2] zuul: [mediawiki/services/apple-search] Add postmerge publish [integration/config] - 10https://gerrit.wikimedia.org/r/715564 (owner: 10Majavah) [19:10:10] !log gitlab-test: upgrading gitlab on gitlab-ansible-test to 13.12.9; re-associating floating IP to gitlab-ansible-test [19:10:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:10:38] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/apple-search] Add postmerge publish [integration/config] - 10https://gerrit.wikimedia.org/r/715564 (owner: 10Majavah) [19:11:57] !log Zuul: [mediawiki/services/apple-search] Add postmerge publish [19:11:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:16:22] !log gitlab-test: powering off gitlab (former main test instance) [19:16:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:26:59] !log deployment-prep: reboot deployment-eventgate-3 (T289029) [19:27:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:27:02] T289029: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 [19:27:43] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10Urbanecm) So, I tried to just reboot that host, and...the URI sounds to work again now? [19:28:47] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible, 10User-Urbanecm: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10Urbanecm) 05Open→03Resolved a:03Urbanecm Boldly closing this, eventlogging_HomepageVisit events... [19:33:32] (03PS1) 10Ottomata: Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/715520 [19:35:08] (03CR) 10Gergő Tisza: [C: 03+1] Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/715520 (owner: 10Ottomata) [19:36:40] (03CR) 10Urbanecm: [C: 03+1] Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/715520 (owner: 10Ottomata) [19:47:39] (03PS3) 10Ahmon Dancy: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [19:48:37] (03CR) 10Ahmon Dancy: WIP Adds kubernetes cluster (033 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [19:53:16] !log urbanecm@deployment-deploy01:/srv/mediawiki-staging$ git submodule update portals # to clear dirty staging dir at beta [19:53:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:53:48] 10Scap: Scap should be clearer about the need for a revert after a failed canary check - https://phabricator.wikimedia.org/T290037 (10Tgr) [19:57:30] (03CR) 10Mholloway: [C: 03+1] Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/715520 (owner: 10Ottomata) [19:57:44] 10Scap: scap sync-file --force warns "sudo: no tty present and no askpass program specified" - https://phabricator.wikimedia.org/T290038 (10Tgr) [19:59:11] (03CR) 10Ahmon Dancy: WIP Adds kubernetes cluster (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [20:01:10] 10Deployments, 10EventStreams, 10Wikimedia-Site-requests: Structure tests for stream settings in operations/mediawiki-config - https://phabricator.wikimedia.org/T290039 (10Tgr) [20:01:38] (03CR) 10Ottomata: [C: 03+2] Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/715520 (owner: 10Ottomata) [20:01:41] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/715520 (owner: 10Ottomata) [20:02:38] (03CR) 10Jeena Huneidi: WIP Adds kubernetes cluster (034 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [20:04:17] (03CR) 10Ahmon Dancy: WIP Adds kubernetes cluster (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [20:06:31] (03CR) 10Jeena Huneidi: WIP Adds kubernetes cluster (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [20:10:05] (03PS4) 10Ahmon Dancy: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [20:11:50] 10Deployments, 10EventStreams, 10Wikimedia-Site-requests: Structure tests for stream settings in operations/mediawiki-config - https://phabricator.wikimedia.org/T290039 (10Ottomata) {T269774} [20:48:05] majavah: If you'd want to hack on T276650 with me I'd be interested, but I appreciate it'd be a bunch of work, and I'm not an expert. [20:48:05] T276650: Re-consider setting up a Kubernetes cluster on the Beta cluster - https://phabricator.wikimedia.org/T276650 [20:52:11] (03CR) 10Jeena Huneidi: "It's working for me in mac also. I'll do the remaining tasks of fixing the checksum thing, moving the setup function, and updating the rea" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [21:12:13] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible, 10User-Urbanecm: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10Ottomata) Hm, I can't reproduce anymore. https://intake-analytics.wikimedia.beta.wmflabs.org/_info [21:12:36] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible, 10User-Urbanecm: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10Ottomata) Oh, oops, commented before seeing your comments. OoooK! [21:18:06] (03CR) 10Ahmon Dancy: WIP Adds kubernetes cluster (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [21:21:48] 10Release-Engineering-Team (Next), 10GitLab, 10serviceops, 10User-brennen: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 (10brennen) [21:21:52] 10Gerrit, 10Release-Engineering-Team (Doing), 10GitLab, 10Icinga, and 4 others: RelEng access to downtime alerts in Icinga for gitlab, gerrit, possibly other services? - https://phabricator.wikimedia.org/T289746 (10brennen) [21:26:07] (03CR) 10Ahmon Dancy: WIP Adds kubernetes cluster (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [21:30:59] (03CR) 10Ahmon Dancy: WIP Adds kubernetes cluster (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [23:38:30] (03PS1) 10Subramanya Sastry: Bump puppeteer to 9.1.1 to use a more recent Chrome browser [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715626 [23:38:32] (03PS1) 10Subramanya Sastry: More Cite CSS fixes to use custom counter style symbols [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715627