[02:22:55] 10Scap, 10Icinga, 10SRE, 10SRE Observability: expose hosts in maintenance state so we can prevent scap from running on them - https://phabricator.wikimedia.org/T100777 (10lmata) [02:23:01] 10Beta-Cluster-Infrastructure, 10SRE Observability, 10Wikimedia-Logstash: [_field_stats] endpoint is deprecated! Use [_field_caps] instead or run a min/max aggregations on the desired fields. - https://phabricator.wikimedia.org/T241485 (10lmata) [02:23:40] 10VPS-project-Codesearch, 10SRE Observability, 10Graphite, 10User-Addshore: Discuss adding grafana dashboard json to codesearch - https://phabricator.wikimedia.org/T244132 (10lmata) [02:29:27] 10Beta-Cluster-Infrastructure, 10SRE Observability, 10Wikimedia-Logstash, 10User-Majavah: logstash-beta.wmflabs.org does not receive any mediawiki events - https://phabricator.wikimedia.org/T233134 (10lmata) [04:59:30] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/703953 [04:59:32] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/703953 (owner: 10TrainBranchBot) [05:01:27] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/703953 (owner: 10TrainBranchBot) [07:57:00] (03PS1) 10Hashar: Increase timeout for Pywikibot deeptest [integration/config] - 10https://gerrit.wikimedia.org/r/704066 (https://phabricator.wikimedia.org/T286336) [08:00:43] (03CR) 10Hashar: [C: 03+2] "It is trivial enough that I just did it" [integration/config] - 10https://gerrit.wikimedia.org/r/704066 (https://phabricator.wikimedia.org/T286336) (owner: 10Hashar) [08:01:15] 10Continuous-Integration-Infrastructure, 10Pywikibot, 10Patch-For-Review, 10Pywikibot-tests: Increase test time for Pywikibot deeptest - https://phabricator.wikimedia.org/T286336 (10hashar) I have updated the build timeout from 15 to 30 minutes. [08:01:51] (03Merged) 10jenkins-bot: Increase timeout for Pywikibot deeptest [integration/config] - 10https://gerrit.wikimedia.org/r/704066 (https://phabricator.wikimedia.org/T286336) (owner: 10Hashar) [08:18:47] (03CR) 10R4356thwiki: "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/703938 (owner: 10Zabe) [08:26:30] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/704067 [08:26:32] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/704067 (owner: 10TrainBranchBot) [08:27:31] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/704067 (owner: 10TrainBranchBot) [08:35:55] 10Release-Engineering-Team (Doing), 10MediaWiki-Docker, 10dev-images, 10mwcli, and 3 others: XDebug not present in docker-registry.wikimedia.org/dev/stretch-php72-fpm:2.0.0 - https://phabricator.wikimedia.org/T273682 (10Physikerwelt) [10:26:45] 10Continuous-Integration-Infrastructure, 10Pywikibot, 10Pywikibot-tests: Increase test time for Pywikibot deeptest - https://phabricator.wikimedia.org/T286336 (10Xqt) 05Open→03Resolved a:03hashar Thanks a lot hashar! [11:05:43] 10Project-Admins: Create project tag for Slow-DB-Query - https://phabricator.wikimedia.org/T277124 (10LSobanski) 05Declined→03Open The main difference I see between the four is that #wikimedia-database-error and #performance_issue focus on the "what" (an error or low performance), while #mariadb-optimizer-bu... [11:52:46] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/704098 [11:52:48] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/704098 (owner: 10TrainBranchBot) [11:53:45] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/704098 (owner: 10TrainBranchBot) [12:05:04] (03PS1) 10Xqt: [i18n] Add gate-and-submit-l10n pipeline to pywikibot-i18n [integration/config] - 10https://gerrit.wikimedia.org/r/704100 (https://phabricator.wikimedia.org/T286207) [12:07:33] James_F_: Thanks! [12:07:48] If we want to make things faster we can do https://gerrit.wikimedia.org/r/c/integration/quibble/+/702909 [12:09:25] I will probably cut a new version of quibble this week and rebuild the CI images [12:09:33] not sure when cause I have to ugprade gerrit as well :D [12:09:40] and July 14th is an holidayhere [12:10:13] hashar: That sounds good. [12:10:23] (Amir1: What did I do?) [12:10:54] James_F_: merged the jquery ui deprecation fixes patch [12:11:23] Oh, yes, that. [12:11:28] E_TOOMANYTHINGS [12:11:35] Amir1: Thanks for writing it. :-) [12:11:51] Though I'd dearly love to delete jQUI, at least we can make it less spammy for now. [12:12:00] hashar: let me know when you want to deploy, I'm already overly excited about it [12:12:15] yes, that thing should just burn in hell, next to mailman2 [12:12:16] it is not going to speed up things that much [12:12:59] well [12:13:08] the --prefer-offline when having a lock file would :D [12:13:43] are you sure? it seems in quibble we run npm install for each subsystem, like wikibase multiple times (one for bridge, one for termbox, one for the main one) plus each extension and skin that has browser test [12:13:53] oh men [12:14:01] it takes 8 minutes just to start selenium in the first part [12:14:09] contint2001 seems to have every single Docker images from /releng/ pulled in :-\ [12:15:27] oh btw, I did my first deployment of jjb last week when you all were away but it was straightforward: https://gerrit.wikimedia.org/r/c/integration/config/+/703212 [12:15:41] If things are broken, it's probably my fault [12:18:31] Amir1: last time I checked a good chunk of it is due to compiling stuff like fibers [12:18:43] result which is not stored in the npm cache [12:18:52] :( [12:19:09] an idea we had ages ago was to just save to our cache ./node_modules and restore it [12:19:17] then run "npm prune && npm install" [12:19:47] so that we would have compiled stuff saved between builds. But I guess that proven a bit too compilcated to handle [12:30:27] (03CR) 10Hashar: npm: Use cache for npm ci and prefer offline (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/702909 (https://phabricator.wikimedia.org/T211705) (owner: 10Kosta Harlan) [12:31:57] oh [12:35:32] It’d be nice to stop using fibers and similar low-performance things if we could. Oh well. [12:45:14] (03PS3) 10David Caro: tox-poetry-buster: Added a new image for poetry based builds [integration/config] - 10https://gerrit.wikimedia.org/r/701121 [12:47:18] (03CR) 10Hashar: [C: 04-1] "I don't think --prefer-offline has any benefit when there is a package-lock.json present see my latest comment on ps3." (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/702909 (https://phabricator.wikimedia.org/T211705) (owner: 10Kosta Harlan) [12:47:21] (03CR) 10David Caro: "I've been able to only need the cache directory, I have to end up setting the "HOME" env var in the tox.ini though, will send the patch fo" [integration/config] - 10https://gerrit.wikimedia.org/r/701121 (owner: 10David Caro) [12:48:35] James_F_: I think fibers comes from wdio-sync to have commands run synchronously [12:48:46] it used to be a hard dependency of wdio-testrunner-mocha in version 4 [12:48:49] Yeah. [12:48:52] but that is decoupled now [12:48:57] But still used. [12:49:02] then I suspect folks like to use wdio-sync [12:49:10] Indeed. [12:49:19] cause the tests might be easier to write when in sync mode [12:49:32] else you get to browser.wait and add async/await all around [12:50:20] or maybe one can just await $('element') and magic happens [12:50:58] https://v5.webdriver.io/docs/sync-vs-async.html has some good examples [12:51:09] and the sync version: $('body').click() is rather straightforward [12:52:11] when in async one has to el = await $('body'); await el.click(); that is slightly more boilerplate [13:01:04] (03PS2) 10Jforrester: Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/704100 (https://phabricator.wikimedia.org/T286207) (owner: 10Xqt) [13:03:16] (03CR) 10Jforrester: [C: 03+2] Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/704100 (https://phabricator.wikimedia.org/T286207) (owner: 10Xqt) [13:04:17] (03Merged) 10jenkins-bot: Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/704100 (https://phabricator.wikimedia.org/T286207) (owner: 10Xqt) [13:05:07] !log Zuul: [pywikibot/i18n] Add gate-and-submit-l10n pipeline T286207 [13:05:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:05:10] T286207: Move pywikibot-i18n CI from travis-ci.com to jenkins - https://phabricator.wikimedia.org/T286207 [13:06:11] (03CR) 10Jforrester: [C: 03+2] Add R4356th to the CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/703938 (owner: 10Zabe) [13:06:23] (03CR) 10Jforrester: [C: 03+2] Add Voidwalker to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/702953 (owner: 10Universal Omega) [13:07:43] (03Merged) 10jenkins-bot: Add R4356th to the CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/703938 (owner: 10Zabe) [13:08:46] (03Merged) 10jenkins-bot: Add Voidwalker to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/702953 (owner: 10Universal Omega) [13:19:24] !log Zuul: Add R4356th to the CI allow list [13:19:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:19:39] !log Zuul: Add Voidwalker to the CI allow list [13:19:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:20:46] Ty James_F [13:20:53] Happy to help. [13:55:12] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech: wikidata-query-gui-build job broken (unable to resolve dependency tree) - https://phabricator.wikimedia.org/T286479 (10Lucas_Werkmeister_WMDE) [13:57:28] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech: wikidata-query-gui-build job broken (unable to resolve dependency tree) - https://phabricator.wikimedia.org/T286479 (10Lucas_Werkmeister_WMDE) Note that the general job / container (not sure which tbh) already installs packages:... [13:59:12] (03PS1) 10Lucas Werkmeister (WMDE): Change wikidata-query-gui-build command [integration/config] - 10https://gerrit.wikimedia.org/r/704119 (https://phabricator.wikimedia.org/T286479) [13:59:24] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech, 10Patch-For-Review: wikidata-query-gui-build job broken (unable to resolve dependency tree) - https://phabricator.wikimedia.org/T286479 (10Lucas_Werkmeister_WMDE) a:03Lucas_Werkmeister_WMDE [14:07:12] (03PS2) 10Jforrester: jjb: [wikidata-query-gui-build] Use only_build command, not create_build [integration/config] - 10https://gerrit.wikimedia.org/r/704119 (https://phabricator.wikimedia.org/T286479) (owner: 10Lucas Werkmeister (WMDE)) [14:08:45] (03CR) 10Jforrester: "Because of the hundreds of different teams' and virtual teams' repos the CI config can touch, we're somewhat prescriptive about commit tit" [integration/config] - 10https://gerrit.wikimedia.org/r/704119 (https://phabricator.wikimedia.org/T286479) (owner: 10Lucas Werkmeister (WMDE)) [14:24:08] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech, 10Patch-For-Review: wikidata-query-gui-build job broken (unable to resolve dependency tree) - https://phabricator.wikimedia.org/T286479 (10Ladsgroup) Form me it was this: ` amsa@amsa-Latitude-7480:~/gui$ npm run deploy > wiki... [14:44:55] !log fix merge conflict on deployment-puppetmaster04 [14:44:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:44:59] (03CR) 10Ladsgroup: [C: 03+2] "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/704119 (https://phabricator.wikimedia.org/T286479) (owner: 10Lucas Werkmeister (WMDE)) [14:46:02] (03Merged) 10jenkins-bot: jjb: [wikidata-query-gui-build] Use only_build command, not create_build [integration/config] - 10https://gerrit.wikimedia.org/r/704119 (https://phabricator.wikimedia.org/T286479) (owner: 10Lucas Werkmeister (WMDE)) [14:46:05] Amir1: jjb jobs are pushed first and merged second, unlike the other two kinds of thing in the int/config repo. [14:46:15] (Yay standardisation.) [14:46:32] ugh, I'm sorry [14:46:45] 10Beta-Cluster-Infrastructure, 10User-Majavah: Puppet failing on deployment-parsoid12.deployment-prep.eqiad1.wikimedia.cloud due to cfssl signing failure - https://phabricator.wikimedia.org/T286375 (10Majavah) 05Open→03Resolved a:03Majavah Fixed by resolving merge conflicts on deployment-puppetmaster04.... [14:46:47] I read it but I forgot because it's the exact opposite everywhere [14:46:56] No worries, it is indeed unusual. [14:47:04] Most of the time it doesn't matter. [14:47:13] But occasionally it really does. [14:48:36] !log ran $ ./jjb-update 'wikidata-query-gui-build' (T286479) [14:48:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:48:45] T286479: wikidata-query-gui-build job broken (unable to resolve dependency tree) - https://phabricator.wikimedia.org/T286479 [14:48:55] Oh good, those jjb scripts are helpful. [15:04:30] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech, 10Patch-For-Review: wikidata-query-gui-build job broken (unable to resolve dependency tree) - https://phabricator.wikimedia.org/T286479 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved Looks like the jjb change successfully re... [15:36:21] (03CR) 10Hashar: "After my investigation on the source repo ( https://gerrit.wikimedia.org/r/c/cloud/wmcs-backups/+/701380/2..3 ) it seems we need Poetry i" (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/701121 (owner: 10David Caro) [16:15:46] !log hard reboot deployment-cache-text06, refusing to let me log in and console full of errors [16:15:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:19:55] (03CR) 10Ahmon Dancy: "> Patch Set 1:" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/702778 (owner: 10Dduvall) [16:37:36] (03CR) 10Hashar: [C: 03+1] "I would happily +2 it :] Thank you Kosta" [integration/quibble] - 10https://gerrit.wikimedia.org/r/692372 (https://phabricator.wikimedia.org/T90875) (owner: 10Kosta Harlan) [16:44:38] 10Phabricator, 10User-zeljkofilipin: Decide guidelines for creation and how to distribute phabricator badges - https://phabricator.wikimedia.org/T196390 (10zeljkofilipin) [17:10:14] hi releng! I'll be taking over c.danis's seat deploying for the puppet request window while he's on baby leave -- I just updated the calendar, could you update the recurring window please? (attn thcipriani) [17:11:33] rzl: sure thing. For future reference, the calendar gets generated from https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/release/+/refs/heads/master/make-deployment-calendar/deployments-calendar.json [17:12:07] oh rad, happy to send a patch instead if you prefer -- just following instructions from the top of the page :) [17:13:30] heh, I should probably mention this is now maintained by a bot at the top of the page :P [17:13:55] rzl: if you could send a patch, that'd be welcome. I can update this/next week's calendar. [17:14:24] will do -- and don't worry about updating the current page, just did it [17:14:25] thanks! [17:14:30] <3 [17:19:07] (03PS1) 10RLazarus: deployment-calendar: rzl takes over cdanis's place in puppet request windows [tools/release] - 10https://gerrit.wikimedia.org/r/704146 [17:21:11] (03CR) 10Thcipriani: [C: 03+2] deployment-calendar: rzl takes over cdanis's place in puppet request windows [tools/release] - 10https://gerrit.wikimedia.org/r/704146 (owner: 10RLazarus) [17:22:05] (03Merged) 10jenkins-bot: deployment-calendar: rzl takes over cdanis's place in puppet request windows [tools/release] - 10https://gerrit.wikimedia.org/r/704146 (owner: 10RLazarus) [18:10:30] 10Beta-Cluster-Infrastructure: Figure out why deployment-cache-text06 keeps crashing - https://phabricator.wikimedia.org/T286502 (10Majavah) p:05Triage→03High [18:23:36] !log hard reboot deployment-cache-text06 once I got in using a root ssh key [18:23:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:28:14] do we have a direction around what we are going to keep doing with phabricator? keep using the community maintained version? [18:36:30] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Design m8s deployment workflows and tooling - https://phabricator.wikimedia.org/T279322 (10dduvall) p:05Triage→03Medium a:03jeena @jeena assigning this to reflect that you have a WIP patch for `... [18:42:31] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline: Support incremental multiversion MW image builds - https://phabricator.wikimedia.org/T286505 (10dduvall) [18:42:59] I'm guessing https://we.phorge.it [18:43:17] 10Beta-Cluster-Infrastructure: Varnish not starting on deployment-cache-text06 - https://phabricator.wikimedia.org/T286506 (10Majavah) [18:43:52] !log deployment-cache-text06 varnish not starting, T286506, causing an outage on text traffic on deployment-prep [18:43:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:43:55] T286506: Varnish not starting on deployment-cache-text06 - https://phabricator.wikimedia.org/T286506 [18:45:33] !log upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, T286506 [18:45:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:46:13] 10Beta-Cluster-Infrastructure, 10User-Majavah: Varnish not starting on deployment-cache-text06 - https://phabricator.wikimedia.org/T286506 (10Majavah) 05Open→03Resolved a:03Majavah Fixed by upgrading Varnish component on hiera and running `apt-get upgrade`. [19:03:53] 10Release-Engineering-Team, 10User-kalle: Release new binary of blubber - https://phabricator.wikimedia.org/T285853 (10dduvall) We stopped building and publishing new binary releases of the `blubber` CLI a while back after migrating to blubberoid (its microservice counterpart)... and I see now that the docs on... [19:21:12] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline: Support incremental multiversion MW image builds - https://phabricator.wikimedia.org/T286505 (10dduvall) p:05Triage→03Medium a:03dancy @dancy, since you volunteered to take a crack at one possible implementation of this idea I'll tent... [19:32:38] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline: Failed PipelineLib based jobs resulting in pileup of old images/containers - https://phabricator.wikimedia.org/T286511 (10dduvall) [19:33:05] 10Release-Engineering-Team (Next), 10MW-on-K8s, 10Release Pipeline: Move .pipeline/config.yaml out of mediawiki-config - https://phabricator.wikimedia.org/T286510 (10dduvall) [19:37:17] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline: PipelineLib system tests should support testing job failure - https://phabricator.wikimedia.org/T286512 (10dduvall) [19:37:57] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline: PipelineLib system tests should support testing job failure - https://phabricator.wikimedia.org/T286512 (10dduvall) p:05Triage→03Medium [19:48:12] addshore: I don't think it's been fully decided but I'm contributing to phorge and I will probably switch upstream to that as long as it remains compatible and stable [19:48:30] cool! [19:49:44] so far the phorge team is coming together with several highly qualified folks who care about phabricator. [19:49:53] nice [20:55:29] (03PS1) 10Mholloway: Add a Swift language dev image for CI testing [releng/dev-images] - 10https://gerrit.wikimedia.org/r/704162 (https://phabricator.wikimedia.org/T284195) [20:57:05] 10Continuous-Integration-Infrastructure, 10Metrics-Platform, 10Patch-For-Review: Add a Swift image to Wikimedia's docker registry for testing - https://phabricator.wikimedia.org/T284195 (10Mholloway) p:05Triage→03High [20:57:17] 10Continuous-Integration-Infrastructure, 10Metrics-Platform, 10Patch-For-Review: Add a Swift image to Wikimedia's docker registry for testing - https://phabricator.wikimedia.org/T284195 (10Mholloway) a:03Mholloway [21:05:01] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T281155 (10Pchelolo) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/697641 * **Summary**: ** This patch is trivial but it touches... [21:10:46] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10SRE, 10serviceops, 10Patch-For-Review: Check out www-portals repo in the mediawiki-webserver and in the mediawiki-multiversion images - https://phabricator.wikimedia.org/T285325 (10jeena) a:05jeena→03None [21:51:01] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments, 10User-brennen: 1.37.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T281153 (10Krinkle)