[01:20:03] Project beta-update-databases-eqiad build #51021: 04FAILURE in 2.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/51021/ [01:55:28] Looks like that was the CategoryTree update [02:02:57] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T281152 (10Reedy) [02:03:44] !log beta-update-databases-eqiad seemingly broken by CategoryTree fix for T271011. Comment left on gerrit patch and task, not reverting patch in master at this stage [02:03:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:03:47] T271011: Update CategoryTree to use the new HookContainer/HookRunner system - https://phabricator.wikimedia.org/T271011 [02:20:01] Project beta-update-databases-eqiad build #51022: 04STILL FAILING in 0.83 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/51022/ [03:20:01] Project beta-update-databases-eqiad build #51023: 04STILL FAILING in 0.99 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/51023/ [04:20:01] Project beta-update-databases-eqiad build #51024: 04STILL FAILING in 1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/51024/ [05:20:01] Project beta-update-databases-eqiad build #51025: 04STILL FAILING in 0.86 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/51025/ [06:20:01] Project beta-update-databases-eqiad build #51026: 04STILL FAILING in 0.87 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/51026/ [07:18:16] gehel: bonmatin [07:18:31] hashar: bijour! [07:19:50] hashar: je me prend un kawa et je suis à toi [07:20:01] pas bete, pareil [07:21:40] Yippee, build fixed! [07:21:41] Project beta-update-databases-eqiad build #51027: 09FIXED in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/51027/ [07:24:25] gehel: https://meet.google.com/iwz-aiko-mwc ;] [07:27:21] 10Continuous-Integration-Config, 10Patch-For-Review: Replace all the npm* images with node10* ones - https://phabricator.wikimedia.org/T207087 (10hashar) That was a great sprint :] [07:48:01] (03CR) 10Hashar: [C: 03+2] python-jobs: Point sonarscanner at expected branch names [integration/config] - 10https://gerrit.wikimedia.org/r/696586 (https://phabricator.wikimedia.org/T283778) (owner: 10Ebernhardson) [07:49:16] (03Merged) 10jenkins-bot: python-jobs: Point sonarscanner at expected branch names [integration/config] - 10https://gerrit.wikimedia.org/r/696586 (https://phabricator.wikimedia.org/T283778) (owner: 10Ebernhardson) [07:56:28] Question for ops/deployers: how risky would it be for me to do an l10n-sync next Monday? [08:04:00] awight: probably not that risky? :] [08:05:21] hashar: :-) I just vaguely remembered, I think the last time I asked this I was instructed to just use the full scap. [08:05:40] Which... makes me think I should not do this on a Monday before All Hands. [08:05:50] depend on the use case [08:05:57] Monday is business as usual afaik [08:06:17] O_O [08:06:19] though probably we got asked to lightened the load on monday/friday since we will get super busy for the 3 days of all hands [08:06:30] it is a 72 hours straight marathon of events ;] [08:07:11] +1 we will avoid melting down the wikis next week, we've decided. [08:11:07] no dpeloys next week. [08:11:16] https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar [08:11:26] makes it even easier to decide :-) [08:11:32] 8) [08:12:13] on another topic, awight where are WMDE engineers chatting beside IRC? [08:12:29] I think you used to have mattermost , wmf does have some slack channels [08:12:46] and I am thinking about having a common place for wmf/wmde/volunteers folks to be able to talk to each others [08:13:17] hashar: +1 still mattermost. We used to have a mattermost-IRC bridge but took that down just recently. [08:13:45] IRC seems like a good common ground, I regret that we all have a walled garden now. [08:18:45] awight: no worries :] [08:29:26] (03PS3) 10Martaannaj: Zuul: Add EventLogging to phan dependencies of PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/698584 [08:33:21] * hashar fights with eslint and jsdoc [08:34:41] (03CR) 10Hashar: [C: 03+2] Zuul: Add EventLogging to phan dependencies of PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/698584 (owner: 10Martaannaj) [08:35:45] (03Merged) 10jenkins-bot: Zuul: Add EventLogging to phan dependencies of PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/698584 (owner: 10Martaannaj) [08:49:25] ./node_modules/.bin/eslint .jsdoc.json [08:49:30] 0:0 warning File ignored by default. Use a negated ignore pattern (like "--ignore-pattern '!'") to override [08:49:39] who knows why it is ignored by default [08:52:22] looks like eslint ignores . files great [09:09:41] (03CR) 10Hashar: "I have now deployed the change" [integration/config] - 10https://gerrit.wikimedia.org/r/698584 (owner: 10Martaannaj) [09:33:18] 10Phabricator: Disable "Browse Gerrit Projects" on https://phabricator.wikimedia.org/r/ - https://phabricator.wikimedia.org/T228507 (10Aklapper) @mmodell: Any guidance, please? :) [10:30:57] !log deployment-prep: cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 to add a motd on all instances # T100837 [10:31:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:31:01] T100837: On deployment-prep, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837 [10:39:48] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [10:45:07] (03PS2) 10Zfilipin: jjb: Fix failing selenium-daily-* Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/699244 (https://phabricator.wikimedia.org/T284706) [10:50:39] (03CR) 10Zfilipin: "The commit is ready for review. I've updated the jobs and started them. See related task for the list of jobs. No jobs fail with `MW_SERVE" [integration/config] - 10https://gerrit.wikimedia.org/r/699244 (https://phabricator.wikimedia.org/T284706) (owner: 10Zfilipin) [11:24:50] (03CR) 10WMDE-Fisch: [C: 03+1] jjb: Fix failing selenium-daily-* Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/699244 (https://phabricator.wikimedia.org/T284706) (owner: 10Zfilipin) [12:25:08] (03CR) 10Hashar: [C: 03+2] "From the task: The updated webdriverio configuration no longer has MW_SCRIPT_PATH environment variable hard-coded." [integration/config] - 10https://gerrit.wikimedia.org/r/699244 (https://phabricator.wikimedia.org/T284706) (owner: 10Zfilipin) [12:26:34] (03Merged) 10jenkins-bot: jjb: Fix failing selenium-daily-* Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/699244 (https://phabricator.wikimedia.org/T284706) (owner: 10Zfilipin) [12:31:15] !log deployment-prep: removed deployment-shellbox puppet certificate and regenerated it. Ran puppet and it passes all fine. [12:31:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:42:00] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [14:14:31] !log deployment-imagescaler03: delete local mwdeploy user with uid 497 # T73480 [14:14:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:14:34] T73480: Prevent puppet from creating local user when they are defined in LDAP - https://phabricator.wikimedia.org/T73480 [14:41:11] hmm [14:41:26] I think I might have bricked deployment-imagescaler03.deployment-prep.eqiad1.wikimedia.cloud [14:41:31] can't login to it anymore :-\ [14:44:16] hashar: there is a ssh key for the root user on all deployment-prep instances with "hashar@postwater_labs" as its comment, if you have that try logging in as root [14:44:39] OH [14:45:00] that reminds me that I should generate a root key of my own too :P [14:45:32] I added some exit 0 to nslcd to bring it down [14:45:35] and that locked me out [14:45:46] but cant still reach it via cumin [14:45:51] so I have fixed it \o/ [14:47:37] !log generate and add my (taavi) own root key to deployment-prep [14:47:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:50:56] o/ [14:51:25] i don't really want to upgrade an npm package, but it looks like CI using a newer version of nodejs/npm that may not be compatible wth my package's version [14:51:27] https://integration.wikimedia.org/ci/job/generic-node12-docker/39/console [14:51:54] CI is failing i thiink because it is now using generic-node12-docker [14:53:42] looks like that was changed in https://gerrit.wikimedia.org/r/c/integration/config/+/698246 [15:02:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10MW-1.37-notes (1.37.0-wmf.9; 2021-06-07), 10Patch-For-Review, 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10Ottomata) Hi, I think this broke... [15:03:07] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10MW-1.37-notes (1.37.0-wmf.9; 2021-06-07), 10Patch-For-Review, 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10Ottomata) cc @razzi ^ [15:04:09] ottomata: I guess your code doesn't compile against node12 [15:04:42] hashar yeah [15:04:44] https://phabricator.wikimedia.org/T284345#7151622 [15:04:59] ... do we have to use node12? [15:05:04] can I switch this back to node10? [15:05:05] or [15:05:11] James_F: ^ :D [15:05:19] can/should I use service-pipeline blubber stuff to use my own docker images for CI? [15:05:29] and james dropped the generic node10 jobs apparently [15:08:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10MW-1.37-notes (1.37.0-wmf.9; 2021-06-07), 10Patch-For-Review, 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10Jdforrester-WMF) >>! In T284345#7... [15:08:59] ottomata: I am restoring it [15:10:19] James_F: making sure I understand your comment: using service-pipeline for node-rdkafka-factory (a library, not a service repo) would be a good idea? [15:12:18] ottomata: and I am guessing node-rdkafka-statsd.git has the same issue [15:12:31] yes [15:12:55] (03PS1) 10Hashar: Restore generic-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/699431 (https://phabricator.wikimedia.org/T284345) [15:12:59] ^^ :] [15:13:00] using pipeline here is actually nice; for the service packages we don't build librdkafka, instead we use the .deb package in the docker image [15:13:26] hashar: i'm ok with using serviice-pipeline instead if that would work [15:13:30] would that be bettter? [15:13:35] (03CR) 10Hashar: [C: 03+2] Restore generic-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/699431 (https://phabricator.wikimedia.org/T284345) (owner: 10Hashar) [15:13:50] ottomata: I guess yes ;) [15:14:09] ok, i'm making a .pipeline now [15:14:14] ottomata: and you should then be able to have a branch for node10 and another for node12 , each with slightly different images [15:14:19] ottomata: Yes. [15:14:20] oh right cool [15:14:35] i'll leave it at node10 now, but when we upgrade node-rdkafka we can do that [15:14:39] or maybe have the master branch to create a node10 and a node12 images in case you can have the code support multiple versions [15:14:40] ottomata: Sorry about this. It'd be nice if SRE let people use a non-EOL'ed node. [15:14:49] (03Merged) 10jenkins-bot: Restore generic-node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/699431 (https://phabricator.wikimedia.org/T284345) (owner: 10Hashar) [15:15:31] ottomata: you can recheck now [15:15:56] k [15:17:55] and maybe we can rename that generic-node10-docker job to a name like "compile-node10" for node-rdkafka-statsd and node-rdkafka-factory [15:18:04] + provide one for node12 [15:18:11] (03PS1) 10Ottomata: Use service-pipeline-test for node-rdkafka-factory [integration/config] - 10https://gerrit.wikimedia.org/r/699435 (https://phabricator.wikimedia.org/T284345) [15:18:27] or well focus on switching to pipeline lib which lets you define the base Debian image / node version [15:18:49] anyway, it is late and friday. So I am going out ;) [15:19:00] See you! [15:19:19] James_F: beside that issue, not much has been reported about the node12 upgrade. So well done! :] [15:19:20] oh [15:19:43] and before I leave, I found an issue with eslint wikimedia config, it doesn't link .jsdoc.json file [15:19:51] seems there is a default pattern to ignore all dot files [15:20:04] I'm going to make the quibble-bullseye images this weekend so that we can offer node12 quibble selenium jobs for zeljko. [15:20:28] hashar: Could be? grunt-eslint isn't controlled by us. [15:20:34] James_F: that is like 80% chance to break the whole world :-\ [15:20:43] hashar: Just for selenium. :-) [15:21:26] aybe we can add some extra pattern like !.*.json to at least lint those. Anyway the task is https://github.com/wikimedia/eslint-config-wikimedia/issues/374 [15:21:34] Yes. [15:21:40] James_F: you get a new apache/mysql as well, new libicu, whatever php version [15:22:00] hashar: thank you! [15:22:16] James_F: https://gerrit.wikimedia.org/r/c/integration/config/+/699435 look right? [15:22:17] along with [15:22:22] hashar: What could go wrong?! ;-) [15:22:35] https://gerrit.wikimedia.org/r/c/node-rdkafka-factory/+/699422 [15:22:46] James_F: well everything. I have very very bad memories about the jessie > stretch switch [15:23:04] ottomata: Yeah, that looks roughly right [15:23:09] ottomata: I'll deploy and we can test. [15:23:12] ty [15:23:24] hashar: Yes, but we did the stretch -> buster change earlier this year without breaking things. [15:23:45] (03PS2) 10Jforrester: Zuul: [node-rdkafka-factory] Switch to service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/699435 (https://phabricator.wikimedia.org/T284345) (owner: 10Ottomata) [15:23:50] (03CR) 10Jforrester: [C: 03+2] Zuul: [node-rdkafka-factory] Switch to service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/699435 (https://phabricator.wikimedia.org/T284345) (owner: 10Ottomata) [15:24:06] cause we had ops providing us php7.2 packages and the libicu upgrade got dealt with. [15:24:18] True. [15:24:40] ideally we would backport node 12 to Buster but well .. [15:24:52] (03Merged) 10jenkins-bot: Zuul: [node-rdkafka-factory] Switch to service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/699435 (https://phabricator.wikimedia.org/T284345) (owner: 10Ottomata) [15:25:10] hashar: SRE vetoed. [15:25:13] hashar: So… [15:25:27] Hence all the work. :-( [15:25:30] !log Zuul: [node-rdkafka-factory] Switch to service-pipeline-test T284345 [15:25:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:25:33] T284345: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 [15:26:14] ottomata: Test run at https://integration.wikimedia.org/ci/job/service-pipeline-test/9123/console [15:26:22] James_F: I am inclined to veto the migration for CI :D but it is friday so I am off!! :] [15:26:26] have fun and happy hacking [15:26:32] and a good Pipeline! [15:26:32] And you! [15:29:25] coool [15:29:41] watching (tbh i didn't run the docker test locally yet..fingers crossed) [15:29:47] Ah! [15:30:10] SUCCESS! [15:30:22] that's great stuff James_F :) [15:30:29] Happy to help. [15:30:34] Couple of comments inbound. [15:33:10] James_F: re config.yaml....i mostly just copied that from another repo [15:33:20] no real idea what it is for, i guess i'll remove it! [15:33:27] Yeah, should be a no-op. [15:33:38] (03PS1) 10Jforrester: Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/699437 (https://phabricator.wikimedia.org/T284345) [15:33:49] ^ There's the other one for you. No rush though. :-) [15:33:52] k [15:35:51] hmm, one day we shouldn' need that statsd one anymore...i think eventgate still uses it hough [15:35:58] :-( [15:36:22] https://phabricator.wikimedia.org/T272714 [15:37:49] * James_F subscribes. [15:37:53] Always love tracking tech debt. [15:39:52] oh, actually... node-rdkafka-statsd does not depend on node-rdkafka, its just a plugin [15:40:13] i betcha it works with docker-node12 [15:40:29] James_F: would it still be better to use service-pipeline? [15:40:58] ottomata: As it's prod-deployed, yes. Also that'll let you upgrade to node12 (or node14 or whatever) at your own pace. [15:41:04] ok [15:41:15] Once we move to GitLab, all repos' CI will be configured in-repo like the pipeline. [15:42:05] nice [15:44:27] James_F: lets try it, ssh://otto@gerrit.wikimedia.org:29418/node-rdkafka-statsd [15:44:32] we can merge the zuul one [15:47:23] (03CR) 10Jforrester: [C: 03+2] Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/699437 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [15:48:54] (03Merged) 10jenkins-bot: Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test [integration/config] - 10https://gerrit.wikimedia.org/r/699437 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [15:50:31] !log Zuul: [node-rdkafka-statsd] Switch to service-pipeline-test T284345 [15:50:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:50:34] T284345: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 [16:18:33] 10Scap (Scap3-MediaWiki-MVP), 10releng-201516-q3, 10releng-201617-q4, 10scap2, and 2 others: [EPIC] Migrate the MW weekly train deploy to scap3 - https://phabricator.wikimedia.org/T114313 (10thcipriani) 05Open→03Declined [16:18:41] 10Release-Engineering-Team (Seen), 10Scap: Use git as transport mechanism for MediaWiki scap deploys - https://phabricator.wikimedia.org/T147938 (10thcipriani) [16:18:43] 10Scap (Scap3-MediaWiki-MVP), 10releng-201617-q3, 10scap2: Bring co-master / fanout capabilities to scap3 deployments - https://phabricator.wikimedia.org/T121276 (10thcipriani) 05Open→03Declined [16:18:51] 10Release-Engineering-Team (Seen), 10Scap: Use git as transport mechanism for MediaWiki scap deploys - https://phabricator.wikimedia.org/T147938 (10thcipriani) 05Open→03Declined [16:18:53] 10Scap (Scap3-MediaWiki-MVP), 10releng-201516-q3, 10releng-201617-q4, 10scap2, and 2 others: [EPIC] Migrate the MW weekly train deploy to scap3 - https://phabricator.wikimedia.org/T114313 (10thcipriani) [16:44:32] 10Release-Engineering-Team (Deployment Training Requests): Deployment credentials for @egardner - https://phabricator.wikimedia.org/T282061 (10thcipriani) 05Open→03Resolved [17:22:23] 10Release-Engineering-Team (Seen), 10MobileFrontend, 10Readers-Web-Backlog (Kanbanana-FY-2020-21): Add tests to verify initial collapsing of sections - https://phabricator.wikimedia.org/T263843 (10Jdlrobson) [17:23:55] (03PS2) 10Nikki Nikkhoui: rehearse stage for image-suggestion-api [integration/config] - 10https://gerrit.wikimedia.org/r/698774 (https://phabricator.wikimedia.org/T281132) [17:27:18] 10Release-Engineering-Team (Seen), 10MobileFrontend, 10Readers-Web-Backlog (Tracking): Add tests to verify initial collapsing of sections - https://phabricator.wikimedia.org/T263843 (10Jdlrobson) [17:28:36] 10Release-Engineering-Team (Logspam), 10MediaWiki-Page-diffs, 10Wikidata, 10wdwb-tech, 10Wikimedia-production-error: Assert.php: Bad value for parameter $oldContent: must be a TextContent|null [Story Points 5] - https://phabricator.wikimedia.org/T231084 (10Krinkle) Chatted with @addshore, and here is wha... [17:31:27] 10Release-Engineering-Team (Logspam), 10MediaWiki-Page-diffs, 10Wikidata, 10wdwb-tech, 10Wikimedia-production-error: Assert.php: Bad value for parameter $oldContent: must be a TextContent|null [Story Points 5] - https://phabricator.wikimedia.org/T231084 (10Ladsgroup) Yes. {T265524} is an example that doe... [17:33:33] 10Release-Engineering-Team (Logspam), 10Contributors-Team, 10MediaWiki-Page-diffs, 10Wikidata, and 2 others: Assert.php: Bad value for parameter $oldContent: must be a TextContent|null [Story Points 5] - https://phabricator.wikimedia.org/T231084 (10Krinkle) @marcella The [maintainers](https://www.mediawiki... [17:36:20] 10Release-Engineering-Team, 10SRE, 10serviceops-radar, 10Release Pipeline (Blubber): build and import blubber package for buster and bullseye (which supports v4) - https://phabricator.wikimedia.org/T283891 (10thcipriani) [17:41:11] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] New PropertyMatcher class and example usage in boardmetrics.py (033 comments) [releng/ddd] - 10https://gerrit.wikimedia.org/r/698961 (owner: 1020after4) [18:00:23] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Can't use git master in blubber for dependencies without caching issues - https://phabricator.wikimedia.org/T283981 (10thcipriani) [18:10:55] 10Release-Engineering-Team (Doing), 10Instrument-ClientError, 10Documentation, 10User-brennen: Deploy process should be updated so JavaScript errors block the train - https://phabricator.wikimedia.org/T274000 (10thcipriani) >>! In T274000#7139385, @Jdlrobson wrote: > @thcipriani I am going to assume resolv... [18:19:46] 10Release-Engineering-Team (Radar), 10Analytics, 10Event-Platform: Stop using puppet + git pull for auto deployment of schema repos - https://phabricator.wikimedia.org/T274901 (10Ottomata) p:05Triage→03Low [18:20:07] (03PS3) 10Ahmon Dancy: Add 'trigger' step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699313 [18:20:09] (03PS1) 10Ahmon Dancy: Typo fix [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699456 [18:20:11] (03PS1) 10Ahmon Dancy: remove obsolete comment [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699457 [18:20:13] (03PS1) 10Ahmon Dancy: added additional entries to .gitignore [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699458 [18:20:54] (03CR) 10Ahmon Dancy: [C: 03+2] Typo fix [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699456 (owner: 10Ahmon Dancy) [18:21:07] (03CR) 10Ahmon Dancy: [C: 03+2] remove obsolete comment [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699457 (owner: 10Ahmon Dancy) [18:21:30] (03CR) 10Ahmon Dancy: [C: 03+2] added additional entries to .gitignore [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699458 (owner: 10Ahmon Dancy) [18:21:37] (03Merged) 10jenkins-bot: Typo fix [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699456 (owner: 10Ahmon Dancy) [18:22:09] (03Merged) 10jenkins-bot: remove obsolete comment [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699457 (owner: 10Ahmon Dancy) [18:22:11] (03Merged) 10jenkins-bot: added additional entries to .gitignore [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699458 (owner: 10Ahmon Dancy) [18:23:43] (03PS4) 10Ahmon Dancy: Add 'trigger' step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699313 [18:25:05] (03CR) 10Ahmon Dancy: Add 'trigger' step (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699313 (owner: 10Ahmon Dancy) [18:39:29] 10Release-Engineering-Team, 10Diffusion-Repository-Administrators, 10Projects-Cleanup, 10Services (watching): Archive mediawiki/event-schemas repository from gerrit - https://phabricator.wikimedia.org/T250113 (10Ottomata) [18:47:23] 10Release-Engineering-Team: Improvements for the process of "WMF Releases" pages on MW - https://phabricator.wikimedia.org/T284615 (10thcipriani) > Is the "branch cut" (which occurs at 02:00 UTC Tuesday) the same thing as (or essentially connected to) running the "make-deploy-notes/uploadChangelog.php" script?... [18:48:14] 10Release-Engineering-Team (Next): Improvements for the process of "WMF Releases" pages on MW - https://phabricator.wikimedia.org/T284615 (10thcipriani) p:05Triage→03Low a:03thcipriani [18:48:38] 10Release-Engineering-Team (Next): Improvements for the process of "WMF Releases" pages on MW - https://phabricator.wikimedia.org/T284615 (10Jdforrester-WMF) Yup, currently //ad hoc//. The bot doing it would be great. [18:50:03] 10Release-Engineering-Team (Next): Improvements for the process of "WMF Releases" pages on MW - https://phabricator.wikimedia.org/T284615 (10thcipriani) Thanks for confirming. We've got a bot building the deployment calendar. Maybe it could make these pages... I'll give this a think. [19:10:32] 10Release-Engineering-Team (Doing): Document automatic MW branching time in Deployment Calendar - https://phabricator.wikimedia.org/T284487 (10thcipriani) p:05Medium→03Low [19:35:43] (03CR) 10Ahmon Dancy: [C: 04-2] "holding" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699313 (owner: 10Ahmon Dancy) [19:44:09] 10Release-Engineering-Team, 10Abstract Wikipedia team, 10Release Pipeline: Service pipeline publish job creates and then immediately deletes its tags - https://phabricator.wikimedia.org/T284495 (10thcipriani) I don't think this is unexpected. We delete builds locally from that machine after they're pushed to... [19:47:00] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:40:52] (03PS1) 10Thcipriani: Calendar: Add train branch time [tools/release] - 10https://gerrit.wikimedia.org/r/699467 (https://phabricator.wikimedia.org/T284487) [20:42:56] (03CR) 10Thcipriani: [C: 03+2] "Tested. Works." [tools/release] - 10https://gerrit.wikimedia.org/r/699467 (https://phabricator.wikimedia.org/T284487) (owner: 10Thcipriani) [20:43:50] 10Release-Engineering-Team (Doing), 10Patch-For-Review: Document automatic MW branching time in Deployment Calendar - https://phabricator.wikimedia.org/T284487 (10thcipriani) 05Open→03Resolved [20:43:57] (03Merged) 10jenkins-bot: Calendar: Add train branch time [tools/release] - 10https://gerrit.wikimedia.org/r/699467 (https://phabricator.wikimedia.org/T284487) (owner: 10Thcipriani) [20:47:39] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:49:21] !log gitlab1001: resetting application data, re-running ansible playbook [20:49:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:24:14] (03CR) 10Tosfos: [C: 03+2] "Thanks everyone!" [extensions/GoogleCustomWikiSearch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/662682 (https://phabricator.wikimedia.org/T284655) (owner: 10Zoranzoki21) [21:24:38] (03CR) 10Tosfos: [V: 03+2 C: 03+2] Add WikiTeq group in this repository [extensions/GoogleCustomWikiSearch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/662682 (https://phabricator.wikimedia.org/T284655) (owner: 10Zoranzoki21) [21:53:25] (03PS5) 10Ahmon Dancy: Add 'trigger' step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699313 [21:53:27] (03PS1) 10Ahmon Dancy: Makefile: add kill target [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699473 [21:53:29] (03PS1) 10Ahmon Dancy: Makefile: give systemtest temp file a self-identifying name [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699474 [21:53:31] (03PS1) 10Ahmon Dancy: Fail systemtest if repo1 job is not successful [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699475 [21:54:14] (03PS2) 10Ahmon Dancy: Makefile: add kill target [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699473 [21:54:16] (03PS2) 10Ahmon Dancy: Makefile: give systemtest temp file a self-identifying name [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699474 [21:54:18] (03PS2) 10Ahmon Dancy: Fail systemtest if repo1 job is not successful [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699475 [21:54:20] (03PS6) 10Ahmon Dancy: Add 'trigger' step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699313 [21:55:18] (03PS3) 10Ahmon Dancy: Makefile: give systemtest temp file a self-identifying name [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699474 [21:55:38] (03CR) 10Ahmon Dancy: [C: 03+2] Makefile: give systemtest temp file a self-identifying name [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699474 (owner: 10Ahmon Dancy) [21:56:06] (03Merged) 10jenkins-bot: Makefile: give systemtest temp file a self-identifying name [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699474 (owner: 10Ahmon Dancy) [21:58:28] (03CR) 10Ahmon Dancy: [C: 03+2] Makefile: add kill target [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699473 (owner: 10Ahmon Dancy) [22:00:33] (03PS3) 10Ahmon Dancy: Makefile: add kill target [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699473 [22:01:29] (03CR) 10Ahmon Dancy: Makefile: add kill target [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699473 (owner: 10Ahmon Dancy) [22:01:33] (03CR) 10Ahmon Dancy: [C: 03+2] Makefile: add kill target [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699473 (owner: 10Ahmon Dancy) [22:02:10] (03Merged) 10jenkins-bot: Makefile: add kill target [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699473 (owner: 10Ahmon Dancy) [22:02:42] (03PS3) 10Ahmon Dancy: Fail systemtest if repo1 job is not successful [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699475 [22:02:44] (03PS7) 10Ahmon Dancy: Add 'trigger' step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/699313 [23:49:57] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook