[00:40:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10User-Ladsgroup: image sizes not displayed on beta - https://phabricator.wikimedia.org/T285125 (10thcipriani) >>! In T285125#7163537, @Majavah wrote: > @dancy Hi, do you imagine the recent `scap` python3 migration might be causing these?... [02:00:47] (03CR) 10Krinkle: "> Patch Set 3: Verified+2 Code-Review+2" (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/699905 (https://phabricator.wikimedia.org/T255671) (owner: 1020after4) [02:03:59] (03CR) 10Krinkle: "> […] I don't have time for this. […]" [releng/phatality] - 10https://gerrit.wikimedia.org/r/699905 (https://phabricator.wikimedia.org/T255671) (owner: 1020after4) [02:40:05] 10MediaWiki-Codesniffer, 10MediaWiki-Documentation, 10Patch-For-Review: Standardize return type hint spacing - https://phabricator.wikimedia.org/T220719 (10Krinkle) I've voted for the "without a space" variant like `function ( Aa $a, Bb $b ): Type`. My thinking: * PSR standards relating to coding style don'... [02:46:22] 10MediaWiki-Codesniffer, 10MediaWiki-Documentation, 10Patch-For-Review: Standardize return type hint spacing - https://phabricator.wikimedia.org/T220719 (10Krinkle) >>! In T220719#7164351, @RobinHood70 wrote: > I'm not seeing anything that looks like a vote […] Fixed. Try again now :) [02:48:45] 10MediaWiki-Codesniffer, 10MediaWiki-Documentation, 10Patch-For-Review: Standardize return type hint spacing - https://phabricator.wikimedia.org/T220719 (10RobinHood70) >>! In T220719#7164464, @Krinkle wrote: > Fixed. Try again now :) That's got it. Thanks! [07:11:36] 10MediaWiki-Codesniffer, 10MediaWiki-Documentation, 10Patch-For-Review: Standardize return type hint spacing - https://phabricator.wikimedia.org/T220719 (10JeffreyWang) Am I eligible to vote? [08:12:15] Hello, when https://phabricator.wikimedia.org/T284797 is deployed??? [09:01:15] 10Release-Engineering-Team (Logspam), 10MassMessage, 10Malayalam-Sites, 10Wikimedia-production-error: MassMessage not working for delivery to Suwikiquote or Mlwikiquote: "CAS update failed on user_touched" - https://phabricator.wikimedia.org/T171003 (10abi_) Still seeing this on wikiquote & wikibooks as of... [09:59:46] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T281152 (10Ladsgroup) [11:00:42] 10Deployments, 10Release-Engineering-Team (Radar), 10Language-Team: Spot check on-wiki messages against prior day's translatewiki updates - https://phabricator.wikimedia.org/T93308 (10Nikerabbit) 05Open→03Resolved a:03Nikerabbit [11:02:36] 10Release-Engineering-Team (Logspam), 10MediaWiki-File-management, 10Platform Team Workboards (Clinic Duty Team): High frequency production warning: FormatMetadata::formatNum with non-numeric value - https://phabricator.wikimedia.org/T285213 (10daniel) [11:08:17] 10Release-Engineering-Team (Logspam), 10Growth-Team, 10MediaWiki-Watchlist, 10Performance-Team (Radar), 10Platform Team Workboards (Clinic Duty Team): WatchedItemStore should not write MapCacheLRU instances to a BagOStuff - https://phabricator.wikimedia.org/T282105 (10daniel) Tagging as "log spam", since... [11:14:09] (03PS1) 10Hashar: Delete lock file on SIGTERM/SIGINT [tools/scap] - 10https://gerrit.wikimedia.org/r/700520 (https://phabricator.wikimedia.org/T192703) [11:14:55] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Scap should release lock on SIGTERM - https://phabricator.wikimedia.org/T192703 (10hashar) a:03hashar [11:14:57] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Scap should release lock on SIGTERM - https://phabricator.wikimedia.org/T192703 (10hashar) I gave it a shot and even found a bug in `scap.lock.Locker` :-] [11:15:36] 10Release-Engineering-Team (Logspam), 10MediaWiki-File-management, 10Platform Team Workboards (Clinic Duty Team): High frequency production warning: FormatMetadata::formatNum with non-numeric value - https://phabricator.wikimedia.org/T285213 (10Reedy) Dupe of {T284269} [11:22:15] 10Release-Engineering-Team (Logspam), 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Platform Team Workboards (Clinic Duty Team): High frequency production warning: Couldn't find a global ID for user {user} - https://phabricator.wikimedia.org/T285216 (10daniel) [12:07:05] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10User-Ladsgroup: image sizes not displayed on beta - https://phabricator.wikimedia.org/T285125 (10LarsWirzenius) We'v been porting Scap to Python3 latetly, and this _might_ be Unicode string confusion problem somewhere. I don't know that i... [12:23:31] (03CR) 10Hashar: [C: 03+2] Remove MediaUploader's dependency on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/700436 (https://phabricator.wikimedia.org/T275157) (owner: 10Ostrzyciel) [12:24:55] (03Merged) 10jenkins-bot: Remove MediaUploader's dependency on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/700436 (https://phabricator.wikimedia.org/T275157) (owner: 10Ostrzyciel) [13:06:02] 10Release-Engineering-Team (Logspam), 10MediaWiki-Interface (active libraries), 10Readers-Web-Backlog (Tracking), 10Wikimedia-production-error: TypeError: str is undefined at byteLength - https://phabricator.wikimedia.org/T279301 (10matmarex) The issue is caused by this script: https://ar.wikipedia.org/wik... [13:06:25] (03PS1) 10David Caro: operations-puppet: wrap rake to allow trapping signals [integration/config] - 10https://gerrit.wikimedia.org/r/700602 [13:12:44] (03PS1) 10Hashar: Add skins for composer merge plugin [integration/quibble] - 10https://gerrit.wikimedia.org/r/700605 (https://phabricator.wikimedia.org/T280506) [13:16:26] (03CR) 10jerkins-bot: [V: 04-1] Add skins for composer merge plugin [integration/quibble] - 10https://gerrit.wikimedia.org/r/700605 (https://phabricator.wikimedia.org/T280506) (owner: 10Hashar) [13:16:56] (03CR) 10Hashar: "A gem found by Lens0021 while using Quibble on GitHub actions :]" [integration/quibble] - 10https://gerrit.wikimedia.org/r/700605 (https://phabricator.wikimedia.org/T280506) (owner: 10Hashar) [13:17:35] (03CR) 10Kosta Harlan: Add skins for composer merge plugin (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/700605 (https://phabricator.wikimedia.org/T280506) (owner: 10Hashar) [13:28:41] (03CR) 10Hashar: Add skins for composer merge plugin (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/700605 (https://phabricator.wikimedia.org/T280506) (owner: 10Hashar) [13:28:51] (03PS2) 10Hashar: Add skins for composer merge plugin [integration/quibble] - 10https://gerrit.wikimedia.org/r/700605 (https://phabricator.wikimedia.org/T280506) [14:00:03] 10Release-Engineering-Team, 10MW-on-K8s: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) [14:10:06] (03CR) 10Jbond: [C: 03+1] "LGTM" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/700602 (owner: 10David Caro) [14:20:57] Project mediawiki-core-doxygen-docker build #25826: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25826/ [14:21:13] (03CR) 10David Caro: operations-puppet: wrap rake to allow trapping signals (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/700602 (owner: 10David Caro) [14:31:44] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: When jenkins kills a build due to max execution time the docker containers stay running - https://phabricator.wikimedia.org/T176747 (10hashar) The issue I have discovered at T176747#3749436 was that D... [14:32:48] (03CR) 10Hashar: operations-puppet: wrap rake to allow trapping signals (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/700602 (owner: 10David Caro) [14:34:32] 10Release-Engineering-Team (Logspam), 10MediaWiki-Interface (active libraries), 10Readers-Web-Backlog (Tracking), 10Wikimedia-production-error: TypeError: str is undefined at byteLength - https://phabricator.wikimedia.org/T279301 (10Jdlrobson) 05Open→03Invalid Since this relates to a gadget, I'm closin... [14:42:40] (03CR) 10David Caro: operations-puppet: wrap rake to allow trapping signals (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/700602 (owner: 10David Caro) [14:53:56] (03CR) 10Hashar: [C: 03+2] "Indeed I have missed the capture_logs trap and wait is the way to go :]" [integration/config] - 10https://gerrit.wikimedia.org/r/700602 (owner: 10David Caro) [14:55:38] (03Merged) 10jenkins-bot: operations-puppet: wrap rake to allow trapping signals [integration/config] - 10https://gerrit.wikimedia.org/r/700602 (owner: 10David Caro) [15:18:42] 10Phabricator: Enable slowvote application - https://phabricator.wikimedia.org/T89906 (10Aklapper) 05Declined→03Resolved For reasons I am not aware of, "Slowvote" got enabled on this instance at some point. :-/ [15:18:51] 10Phabricator: Enable slowvote application - https://phabricator.wikimedia.org/T89906 (10Aklapper) a:05chasemp→03None [15:23:17] Project mediawiki-core-doxygen-docker build #25827: 04STILL FAILING in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25827/ [15:24:27] (03PS1) 10Hashar: jjb: update image for operations-puppet-tests-buster-docker [integration/config] - 10https://gerrit.wikimedia.org/r/700648 [15:35:06] (03CR) 10Hashar: [C: 03+2] jjb: update image for operations-puppet-tests-buster-docker [integration/config] - 10https://gerrit.wikimedia.org/r/700648 (owner: 10Hashar) [15:36:22] (03Merged) 10jenkins-bot: jjb: update image for operations-puppet-tests-buster-docker [integration/config] - 10https://gerrit.wikimedia.org/r/700648 (owner: 10Hashar) [15:41:06] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10MW-1.37-notes (1.37.0-wmf.9; 2021-06-07), 10Patch-For-Review, 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10hashar) [15:46:00] (03PS1) 10Hashar: Restore generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/700654 (https://phabricator.wikimedia.org/T285239) [15:46:16] (03CR) 10Hashar: [C: 03+2] Restore generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/700654 (https://phabricator.wikimedia.org/T285239) (owner: 10Hashar) [15:51:36] (03Merged) 10jenkins-bot: Restore generic-node10-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/700654 (https://phabricator.wikimedia.org/T285239) (owner: 10Hashar) [16:14:11] Yippee, build fixed! [16:14:12] Project mediawiki-core-doxygen-docker build #25828: 09FIXED in 8 min 40 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/25828/ [16:26:56] have we done failovers for Gerrit/Phab during the datacenter switchover in the past? [16:30:55] legoktm: for Gerrit that was done once back in 2019 as part of hardware upgrade. There is some doc at https://wikitech.wikimedia.org/wiki/Gerrit/Operations#Switch_over but it might not be accurate [16:31:19] CI master got moved a year ago as part of OS upgrade iirc [16:31:34] doc.wm.o once a year ago for os upgrade as well [16:32:04] they all free suffer from a huge issue due to rsync messing up files ownership and I haven't dig into that rabbit hole yet [16:32:20] Phabricator I can't tell, twentyafterfour will know whether it got switched over any recently [16:33:20] we have a backup machine and we have done failover before but not as part of the datacenter switch as far as I can remember [16:33:43] dc center switch exercise is solely for mediawiki isn't it? [16:33:56] no, it's for as much stuff as possible [16:33:59] or is there a plan to switch everything next time ? [16:34:20] it would probably be good to exercise the process but it's not hard to failover phabricator [16:34:21] in theory we switch over more stuff than last time, which is why I'm asking :) [16:34:26] mainly just a dns switch I think [16:35:56] is it documented anywhere? the next step would be to get it into a cookbook [16:36:37] also I don't want to put pressure on getting ready for next week's switchover, but if it's straightforward and you have some time, it would be great to do. if not, let's at least try to plan for next time [16:37:04] hashar: doc.wm.o should be straightforward to switchover since it's just static files right? [16:37:28] gerrit/ci/doc all lsuffer from the same problem: files changes to random uid/gid [16:38:03] not owned by www-data or smth? [16:39:04] "The reason is the quickdata copy puppet classes that spawns rsync within a chroot which in turns prevent uid/gid mapping." [16:39:19] ah [16:39:22] runbook for CI is https://www.mediawiki.org/wiki/Continuous_integration/Data_center_switch [16:39:59] tentatively running rsync over ssh as root / root would let rsync use the proper uid/gid username/groupname but I haven't tested it [16:40:22] the switch over doc for Gerrit probably needs some updating. We have upgraded from 2.15 to 3.2 since then [16:40:55] gotcha, let me file tasks for these since it doesn't seem straightforward [16:41:21] for CI master runbook we have https://phabricator.wikimedia.org/T256396#6506919 [16:42:08] I guess it all boils down to find out a way to safely move files frmo a host to another, or we go to use some distributed file system that span both DCs [16:44:55] we have swift :) [16:45:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Datacenter-Switchover: Create a runbook for switching CI master - https://phabricator.wikimedia.org/T256396 (10Legoktm) [18:15:15] why is my cursor/hand so large on phab right now when hovering over a task card? https://phabricator.wikimedia.org/F34518964 [18:16:54] reupload due to private tab leak: https://phabricator.wikimedia.org/F34518968 [18:17:01] * greg-g looked first, promise [19:08:17] greg-g: the css is just `cursor: grab` so if it looks weird maybe that's some update in your browser? [19:16:54] I knew Firefox was making weird design choices in this latest release but oh my [19:20:22] huh, yeah, I can get it to happen in the gmail webui if I find this one pixel-width area next to a message subject [19:20:40] ok, more like 15, but still [19:33:40] (03CR) 10Jeena Huneidi: WIP: Xdebug: always install xdebug module, move config to core (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/692737 (https://phabricator.wikimedia.org/T273682) (owner: 10Brennen Bearnes) [20:32:06] (03PS6) 10Jeena Huneidi: WIP: Xdebug: always install xdebug module, move config to core [releng/dev-images] - 10https://gerrit.wikimedia.org/r/692737 (https://phabricator.wikimedia.org/T273682) (owner: 10Brennen Bearnes) [21:03:11] 10Project-Admins: Create project tag for SRE Infrastructure Foundations - https://phabricator.wikimedia.org/T285143 (10Aklapper) 05Open→03Resolved a:03Aklapper * Requested public project #infrastructure-foundations has been created: https://phabricator.wikimedia.org/project/view/5387/ ** Interested people... [23:27:05] !log Jobs for `deployment-deploy01` are waiting for executor but host is completely idle for over 10min. Disconnecting and relaunching. [23:27:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL