[04:03:31] (03CR) 10Thcipriani: [C: 03+2] feat: Add script to dump bundle as wikitext [tools/release] - 10https://gerrit.wikimedia.org/r/723012 (owner: 10Thcipriani) [04:04:45] (03Merged) 10jenkins-bot: feat: Add script to dump bundle as wikitext [tools/release] - 10https://gerrit.wikimedia.org/r/723012 (owner: 10Thcipriani) [07:44:03] 10Beta-Cluster-Infrastructure: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679 (10Chlod) [07:47:39] 10Beta-Cluster-Infrastructure: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679 (10Chlod) Very likely related to T285527. [08:05:17] 10Beta-Cluster-Infrastructure, 10Domains: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679 (10Chlod) [08:33:29] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10fgiunchedi) [08:33:33] 10Release-Engineering-Team (Radar), 10SRE, 10serviceops, 10Patch-For-Review: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10fgiunchedi) 05Resolved→03Open I don't think this is resolved, see T275752 for jobrunner on buster slowness in upload [08:33:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop MediaWiki testing in stretch and instead test only in buster - https://phabricator.wikimedia.org/T252432 (10fgiunchedi) [09:05:21] 10Beta-Cluster-Infrastructure: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679 (10Majavah) Fun question: why is that being sent out from cloudinfra mx:es instead of deployment-prep's own mx boxes? [09:10:02] (03CR) 10Hashar: [C: 03+2] "As discussed with James ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/723256 (https://phabricator.wikimedia.org/T267888) (owner: 10Krinkle) [09:12:00] (03Merged) 10jenkins-bot: node14: Update from bundled npm 6 to pinned npm 7 [integration/config] - 10https://gerrit.wikimedia.org/r/723256 (https://phabricator.wikimedia.org/T267888) (owner: 10Krinkle) [09:41:19] (03CR) 10Hashar: "Successfully published image docker-registry.discovery.wmnet/releng/node14:0.0.2" [integration/config] - 10https://gerrit.wikimedia.org/r/723256 (https://phabricator.wikimedia.org/T267888) (owner: 10Krinkle) [09:50:57] (03CR) 10Hashar: [V: 03+2 C: 03+2] Merge branch 'stable-3.3' into wmf/stable-3.3 [software/gerrit/plugins/gitiles] (wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/716484 (owner: 10Hashar) [09:56:19] (03PS2) 10Hashar: Merge tag 'v3.3.6' into wmf/stable-3.3 [software/gerrit] (wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/716485 (https://phabricator.wikimedia.org/T290236) [09:58:43] (03CR) 10jerkins-bot: [V: 04-1] Merge tag 'v3.3.6' into wmf/stable-3.3 [software/gerrit] (wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/716485 (https://phabricator.wikimedia.org/T290236) (owner: 10Hashar) [10:04:16] 10Release-Engineering-Team (Seen), 10Data³, 10Quality-and-Test-Engineering-Team (QTE): Release Engineering Data Collection and Retention (aka Data³) - https://phabricator.wikimedia.org/T216085 (10zeljkofilipin) [10:07:34] 10Gerrit, 10SRE: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Marostegui) @Dzahn did this ever happen? Or should we maybe ping specific people to make it happen? [10:20:33] (03PS1) 10Hashar: dockerfiles: add python3-distutils to gerrit image [integration/config] - 10https://gerrit.wikimedia.org/r/723477 [10:20:53] (03CR) 10Hashar: [C: 03+2] dockerfiles: add python3-distutils to gerrit image [integration/config] - 10https://gerrit.wikimedia.org/r/723477 (owner: 10Hashar) [10:21:20] (03PS1) 10Hashar: jjb: update Gerrit job for python3-distutils [integration/config] - 10https://gerrit.wikimedia.org/r/723479 [10:23:04] (03Merged) 10jenkins-bot: dockerfiles: add python3-distutils to gerrit image [integration/config] - 10https://gerrit.wikimedia.org/r/723477 (owner: 10Hashar) [10:39:05] (03CR) 10Hashar: "Successfully published image docker-registry.discovery.wmnet/releng/gerrit:1.1.4" [integration/config] - 10https://gerrit.wikimedia.org/r/723477 (owner: 10Hashar) [10:39:48] (03CR) 10Hashar: "recheck after adding python3-distutils to provide distutils.spawn ( https://gerrit.wikimedia.org/r/c/integration/config/+/723477 )" [software/gerrit] (wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/716485 (https://phabricator.wikimedia.org/T290236) (owner: 10Hashar) [10:48:52] (03CR) 10Hashar: [C: 03+2] "That fixed the build" [integration/config] - 10https://gerrit.wikimedia.org/r/723479 (owner: 10Hashar) [10:50:52] (03Merged) 10jenkins-bot: jjb: update Gerrit job for python3-distutils [integration/config] - 10https://gerrit.wikimedia.org/r/723479 (owner: 10Hashar) [12:32:34] (03PS2) 10Hashar: Gerrit v3.3.6 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/716317 (https://phabricator.wikimedia.org/T290236) [12:32:47] (03CR) 10Hashar: [C: 03+2] Merge tag 'v3.3.6' into wmf/stable-3.3 [software/gerrit] (wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/716485 (https://phabricator.wikimedia.org/T290236) (owner: 10Hashar) [12:39:17] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Performance-Team, 10SRE, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10akosiaris) >>! In T290536#7371552, @jijiki wrote: >>>! In T290536#7364817, @Joe wrote: >> I have some alternative ideas. Specifical... [12:40:06] (03Merged) 10jenkins-bot: Merge tag 'v3.3.6' into wmf/stable-3.3 [software/gerrit] (wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/716485 (https://phabricator.wikimedia.org/T290236) (owner: 10Hashar) [12:44:01] (03CR) 10Hashar: "I have rebuild all the plugins and pushed the .jar to Archiva. Gerrit boots locally and the plugin list looks correct ;)" [software/gerrit] (deploy/wmf/stable-3.3) - 10https://gerrit.wikimedia.org/r/716317 (https://phabricator.wikimedia.org/T290236) (owner: 10Hashar) [13:20:27] (03CR) 10Hashar: [C: 03+2] "I have build the new releng/tox-buster so we can migrate the various job to it." [integration/config] - 10https://gerrit.wikimedia.org/r/721885 (https://phabricator.wikimedia.org/T291292) (owner: 10Jforrester) [13:20:36] (03CR) 10jerkins-bot: [V: 04-1] Docker: Drop tox-(censorshipmonitoring|conftool|eventlogging|homer|ldap) [integration/config] - 10https://gerrit.wikimedia.org/r/721885 (https://phabricator.wikimedia.org/T291292) (owner: 10Jforrester) [13:26:17] (03PS1) 10Hashar: jjb: update to latest releng/tox-buster image [integration/config] - 10https://gerrit.wikimedia.org/r/723523 [13:27:45] (03PS2) 10Hashar: Docker: Drop tox-(censorshipmonitoring|conftool|eventlogging|homer|ldap) [integration/config] - 10https://gerrit.wikimedia.org/r/721885 (https://phabricator.wikimedia.org/T291292) (owner: 10Jforrester) [13:28:15] (03CR) 10Hashar: [C: 03+2] "Rebased due to conflict with shellcheck being added to the images I10dd786464096f115fc3d3c8cf2163ac99004764" [integration/config] - 10https://gerrit.wikimedia.org/r/721885 (https://phabricator.wikimedia.org/T291292) (owner: 10Jforrester) [13:30:36] (03Merged) 10jenkins-bot: Docker: Drop tox-(censorshipmonitoring|conftool|eventlogging|homer|ldap) [integration/config] - 10https://gerrit.wikimedia.org/r/721885 (https://phabricator.wikimedia.org/T291292) (owner: 10Jforrester) [13:51:39] (03PS2) 10Hashar: jjb: update to latest releng/tox-buster image [integration/config] - 10https://gerrit.wikimedia.org/r/723523 (https://phabricator.wikimedia.org/T291292) [13:51:41] (03PS1) 10Hashar: jjb: drop custom tox images in favor of tox-buster [integration/config] - 10https://gerrit.wikimedia.org/r/723532 (https://phabricator.wikimedia.org/T291292) [13:52:36] James_F: bunch of tox cleanup ^ :] [14:44:00] (03Abandoned) 10Hashar: experimental test with mediawiki/tools/release [integration/config] - 10https://gerrit.wikimedia.org/r/589336 (owner: 10Hashar) [14:55:49] (03CR) 10Hashar: Automatically mention train blocker task in deploy-promote (032 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/608936 (owner: 1020after4) [14:55:56] (03PS3) 10Hashar: Automatically mention train blocker task in deploy-promote [tools/release] - 10https://gerrit.wikimedia.org/r/608936 (owner: 1020after4) [14:56:11] (03CR) 10Hashar: [C: 03+1] Automatically mention train blocker task in deploy-promote [tools/release] - 10https://gerrit.wikimedia.org/r/608936 (owner: 1020after4) [14:59:07] (03CR) 1020after4: [C: 03+1] "Nicely done, thanks Antoine! So should we merge this now? I think that my only concern is now addressed." [tools/release] - 10https://gerrit.wikimedia.org/r/608936 (owner: 1020after4) [15:01:10] (03CR) 1020after4: [C: 03+1] "👍" [tools/release] - 10https://gerrit.wikimedia.org/r/608936 (owner: 1020after4) [15:05:45] (03CR) 10Hashar: [C: 03+2] "Lets fly!" [tools/release] - 10https://gerrit.wikimedia.org/r/608936 (owner: 1020after4) [15:08:20] (03Merged) 10jenkins-bot: Automatically mention train blocker task in deploy-promote [tools/release] - 10https://gerrit.wikimedia.org/r/608936 (owner: 1020after4) [15:33:04] (03PS3) 10Ahmon Dancy: WIP: Access train-dev git server instead of gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 [15:40:11] (03PS4) 10Ahmon Dancy: Access train-dev git server instead of gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 [15:40:41] (03CR) 10Ahmon Dancy: [C: 03+1] "Tested on Linux and Mac, but could use another tester." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723267 (owner: 10Ahmon Dancy) [15:41:56] !log Cherry-picking https://gerrit.wikimedia.org/r/c/performance/coal/+/722948 and latest https://gerrit.wikimedia.org/r/c/operations/puppet/+/721047 in deployment-prep. Should only affect deployment-webperf11. [15:42:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:44:02] (03CR) 10Ahmon Dancy: "Happy to see this!" [tools/release] - 10https://gerrit.wikimedia.org/r/608936 (owner: 1020after4) [15:46:30] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:51:27] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10ARamirez_WMF) Hi. I would like to ask to add @madalina to #acl*Project-Admins because she is the program manager for Trust and Safety Tools. Madalina will need to frequently create n... [15:55:18] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Ladsgroup) >>! In T706#7376791, @ARamirez_WMF wrote: > Hi. I would like to ask to add @madalina to #acl*Project-Admins because she is the program manager for Trust and Safety Tools.... [16:01:26] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:15:59] 10Gerrit, 10SRE: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn) @Marostegui No, it's still open. I'll follow-up on it soon. [16:16:09] 10Gerrit, 10SRE: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10Dzahn) a:03Dzahn [17:24:54] for some reason it feels like things that should be train blockers or cause for immediate backports are slipping through the cracks, e.g. https://phabricator.wikimedia.org/T291675, https://phabricator.wikimedia.org/T291677 [17:25:03] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Fold most of the tox images into tox-buster - https://phabricator.wikimedia.org/T291292 (10Jdforrester-WMF) [17:25:48] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Fold most of the tox images into tox-buster - https://phabricator.wikimedia.org/T291292 (10Jdforrester-WMF) I didn't fold in tox-pywikibot given what it's doing to the image is very specialised. Resolved? [17:25:54] (03CR) 10Jforrester: "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/721885 (https://phabricator.wikimedia.org/T291292) (owner: 10Jforrester) [17:26:38] also https://phabricator.wikimedia.org/T291272 [17:38:34] (03PS3) 10Jforrester: jjb: Update jobs to latest releng/tox-buster image [integration/config] - 10https://gerrit.wikimedia.org/r/723523 (https://phabricator.wikimedia.org/T291292) (owner: 10Hashar) [17:38:36] (03PS2) 10Jforrester: jjb: Switch jobs from custom tox images to tox-buster [integration/config] - 10https://gerrit.wikimedia.org/r/723532 (https://phabricator.wikimedia.org/T291292) (owner: 10Hashar) [17:49:46] kostajh: what did you want to discuss about https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PageTriage/+/723581 ? [17:50:22] (03PS1) 10Ahmon Dancy: build-mw-image-loop.py: Add --build-at-start option [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723603 [17:51:13] legoktm: if we can fix instead is reverting. There are some ideas in the patch. I mistakenly thought this was merged months ago, I didn’t realize it was just in wmf.1 [17:51:40] so, a revert is fine. I’ll comment on the patch. [17:51:57] thanks [17:52:12] (03PS2) 10Ahmon Dancy: build-mw-image-loop.py: Add --build-at-start option [tools/train-dev] - 10https://gerrit.wikimedia.org/r/723603 [17:52:23] happy to review/merge a fix if one exists, but given how important this is I don't think leaving it like this is acceptable [18:04:45] legoktm: ack [18:24:57] legoktm: at a quick glance, i don't think any of those issues mention this week's train blocker task or a current train deployer (or other relenger), and i'm not sure any of them would have manifested as errors in logs. [18:25:54] as a practical matter, regressions that manifest in user-facing UI aren't very likely to get marked train blockers unless someone brings them to our attention. [18:26:31] (or just marks them as a blocker directly.) [18:27:08] yeah...I feel there's a disconnect here, I don't think end users filing bugs should be expected to know enough about the train to mark blockers like that [18:27:29] I just found out about these by reading WP:VPT mostly [18:28:07] yeah, i think you're right - though i think this is an outlier on this class of issue slipping through compared to most weeks. [18:29:15] i don't think there's a stated expectation anywhere that people should know to block the train, but i was trying to think through what actually tends to happen. [18:29:32] I don't remember another week like this either [18:29:49] I do know Andre is on vacation this week, I'm not sure if anyone is doing bug triage in his absence [18:31:02] it does happen. i can think of regressions that probably should have been train blockers coming up the following week a couple of times in my last handful of trains. [18:33:13] i very much doubt we want to commit train deployers to monitoring any more channels, but i wonder if we should work on some general "hey, please surface these things to deployers" messaging / training. [18:35:02] English Wikipedians are generally pretty good at recognizing "WP:ITSTHURSDAY" bugs, https://en.wikipedia.org/wiki/Wikipedia:Bug_reports_and_feature_requests#Software_deployment_schedule [18:36:06] maybe that page should advise leaving a comment on the current train blocker ticket [18:36:55] seems like a decent idea - could use https://train-blockers.toolforge.org/ to link [18:55:44] TIL about WP:ITSTHURSDAY [18:59:49] 10Phabricator: give visibility for "in progress" tasks on a work board - https://phabricator.wikimedia.org/T291593 (10mmodell) 05Open→03In progress [18:59:51] 10Phabricator: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10mmodell) [19:00:39] https://en.wikipedia.org/w/index.php?title=Wikipedia:Bug_reports_and_feature_requests&diff=next&oldid=1046261162 [19:04:00] 10Phabricator, 10Data³: Phab feature request: Cycle time for a task entering a column to resolution, with support for wildcards - https://phabricator.wikimedia.org/T148805 (10mmodell) 05Open→03In progress [19:16:02] I am disconnected from user bug reports. It was a recent surprise to realize that many backports are not for production errors, but from bug reports: https://gitlab.wikimedia.org/thcipriani/train-stats#train-bugfixes [19:17:32] pushing this to the edges as much as possible would be ideal (I guess that's what normally happens with bug reports and less with production errors) [19:19:15] I'd be interested if there are good tacts to pushing this kind of thing closer to folks who have the context. Adding the blurb about the train blocker to bug reports seems like a good idea, but I have a lot of trouble knowing where production errors are coming from, folks spotting bugs who don't have an org-chart in their heads are likely less equipped. [19:19:36] s/but/and [19:20:53] (where bugs are coming from is important for knowing who to tag so that a bug gets attention) [19:22:28] tl;dr: want to push all bugs/errors to the edges where the knowledge is, but how? [19:26:18] * legoktm nods [19:26:49] there's also https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Paalika_Keka which is an issue in en-gb translations that no one has reported to phab yet AFAICT [19:28:35] they're already reverted on twn, so not sure if a phab report woulf help anything (except statistics) [19:29:01] Again [19:29:09] it's in wmf.1, so it needs a backport + scap [19:29:15] We had that not too many trains ago [19:29:20] > Really, no one should use en-gb, it is a monstrosity. [19:29:21] huh [19:30:14] the fallback chain with local overrides is complicated and somewhat broken [19:30:31] and since we don't do translations nightly, that'd need someone to do some scapping [19:31:04] yeah, with l10nupdate it would've been automatically taken care of [19:31:52] if there are overrides in place, I'd rather keep this backport for Monday, but seems like it needs an explicit backport [19:32:53] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/e343ada8ab8267a29793a671c4e9da7d6e188cb8%5E%21/#F7 the fix is already in master, so yeah [19:32:55] thcipriani: we didn't backport for the last time that happened [19:33:23] does that happen often? [19:33:32] RhinosF1: a mistake IMO [19:33:33] from the comment it seems like it may [19:33:37] It wasn't that long back [19:34:31] thcipriani: https://phabricator.wikimedia.org/T286679 [19:34:34] I wouldn't say this falls into an emergency category of backports, but ¯\_(ツ)_/¯ seems broken :) [19:35:48] personally I'm over my quota/enthusaism for doing more backports today so I agree with doing it on Monday [19:36:34] is this the backport for T286679? https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/705750/ [19:36:35] T286679: Some interface messages are in Welsh when British English is selected as the interface language - https://phabricator.wikimedia.org/T286679 [19:37:08] backport enthusiasm quotient is an important stat [19:38:26] thcipriani: one of many [19:39:04] anyway: I'm pro fixing broken stuff. I'm also pro folks who get paged having a weekend. Sometimes there arise conflicts in these positions, but this isn't one of those times :) [19:50:33] 10Phabricator, 10Release-Engineering-Team: The "Project report" links to individual weeks don't work - https://phabricator.wikimedia.org/T291710 (10Krinkle) [19:52:32] no-stupid-questions: i'm pulling a list of users from gitlab. i'd like to query on their e-mails for uid in LDAP... [19:53:08] what is a less-ridiculous way to do this than shelling into mwmaint1002 and running `ldapsearch`? [19:53:43] ldapsearch, FWIW, would be my way of doing this [20:33:35] brennen: sql query against labswiki `user` table? [20:34:11] hashar: i think i got it figured out, but i am making a note of that idea for future use. [20:34:22] hashar: technically not every ldap user has a wikitech account [20:34:40] true [20:35:01] and `mail` is not enforced to be unique accross users [20:35:42] i just noticed that part, but i think i can modify the list by hand for the few places it matters. [20:36:06] this is just a one-off to rename accounts which have already logged into gitlab. [20:37:12] ouch [20:40:14] sleep time, I was merely catching up with backlogs. Have all a good week-end! [20:45:29] * bd808 is in favor of uid instead of cn for gitlab usernames [20:47:44] yeah, i think we've got it figured out. i'm going to do the migration on monday morning just so i don't get mired in whatever inevitably breaks before the weekend. [20:49:13] and then we can turn off the group restriction on logins. [20:54:20] woot! [21:02:39] 10Release-Engineering-Team (Doing), 10GitLab (Auth & Access), 10Patch-For-Review, 10Privacy, 10User-brennen: GitLab uses 'real name' as username (rather than 'shell name' or an user-specified name) - https://phabricator.wikimedia.org/T288392 (10brennen) A script for resetting both `extern_uid` and `usern... [22:50:36] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:56:52] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1