[02:06:05] 10Phabricator: Make it clearer/easier how to silence task batch edits - https://phabricator.wikimedia.org/T386147#10542454 (10Pppery) For archeolgy's sake: https://secure.phabricator.com/T13042 > However, since you can "bulk edit" a single task, it's a policy issue to just add a [√] Don't tell anyone I did thi... [03:14:53] 10MediaWiki-Releasing, 10TemplateStyles: Bundle Extension:TemplateStyles with MediaWiki core - https://phabricator.wikimedia.org/T327006#10542543 (10Bugreporter2) Why is the checklist vague? Why not make it less vague? Then you can't hide behind it as an excuse to avoid doing something - or have I discovered... [03:24:32] 10MediaWiki-Releasing, 10MobileFrontend (Tracking): Bundle MobileFrontend extension with MediaWiki - https://phabricator.wikimedia.org/T191734#10542545 (10Izno) [03:24:33] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10542546 (10Izno) [03:24:33] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146#10542547 (10Izno) [03:24:39] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146#10542549 (10Izno) [03:24:45] 10MediaWiki-Releasing, 10MediaWiki-extensions-OATHAuth: Bundle WebAuthn extension with MediaWiki - https://phabricator.wikimedia.org/T258007#10542551 (10Izno) [03:24:51] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146#10542553 (10Izno) [03:24:57] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10542552 (10Izno) [03:25:12] 10MediaWiki-Releasing: Bundle Popups extension with MediaWiki - https://phabricator.wikimedia.org/T231797#10542554 (10Izno) [03:25:18] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10542555 (10Izno) [03:25:20] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146#10542556 (10Izno) [03:25:35] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146#10542559 (10Izno) [03:25:36] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10542558 (10Izno) [03:25:44] 10MediaWiki-Releasing, 10AntiSpoof, 06Trust and Safety Product Team: Bundle AntiSpoof extension with MediaWiki - https://phabricator.wikimedia.org/T191736#10542560 (10Izno) [03:25:52] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146#10542562 (10Izno) [03:25:58] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10542561 (10Izno) [03:26:25] 10MediaWiki-Releasing, 10TemplateStyles: Bundle Extension:TemplateStyles with MediaWiki core - https://phabricator.wikimedia.org/T327006#10542564 (10Izno) [03:26:28] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 07Epic, 05MW-1.44-release: Expand the set of bundled extensions and skins in MediaWiki 1.44 - https://phabricator.wikimedia.org/T333405#10542565 (10Izno) [04:15:11] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T382369#10542584 (10thcipriani) p:05Triageβ†’03Medium a:03dduvall [05:27:49] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for JainLakshita28 - https://phabricator.wikimedia.org/T384247#10542603 (10Jainlakshita28) Sorry for creating same task again. I want developer access so that i can start contributing. Can you please guide me how can... [06:38:03] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for JainLakshita28 - https://phabricator.wikimedia.org/T384247#10542613 (10Anoop) >>! In T384247#10542603, @Jainlakshita28 wrote: > Sorry for creating same task again. I want developer access so that i can start cont... [06:38:50] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for JainLakshita28 - https://phabricator.wikimedia.org/T384247#10542614 (10Anoop) 05Openβ†’03Resolved a:03Anoop [06:50:23] Not a train blocker but very noisy in the logs: https://phabricator.wikimedia.org/T386162#10542616 [08:31:39] 14Release-Engineering-Team (Watching / External), 10CirrusSearch, 13Patch-For-Review: Figure out why browser tests can't create suggestion box - https://phabricator.wikimedia.org/T162966#10542788 (10Gehel) [08:34:14] 14Scap (Scap3-Adoption-Phase1), 14scap2, 10Elasticsearch: Deploy elasticsearch plugins with scap3 (Trebuchet elasticsearch/plugins) - https://phabricator.wikimedia.org/T151996#10542875 (10Gehel) [08:34:18] 10Continuous-Integration-Config, 14Release-Engineering-Team (Backlog), 10CirrusSearch: Make browsertests for CirrusSearch run on every submitted patch with proper CI infrastructure rather than a bot - https://phabricator.wikimedia.org/T98374#10542877 (10Gehel) [08:37:33] 06Release-Engineering-Team, 06Data-Engineering, 06Java-Scala-Standardization, 10Metrics Platform: Adapt gitlab pipelines for the new wmf-jvm-parent-pom - https://phabricator.wikimedia.org/T358841#10542978 (10Gehel) [08:38:01] 10GitLab (CI & Job Runners), 14Release-Engineering-Team (Blocking 🧱), 10CirrusSearch: Consider adding "official" docker.elastic.co images to the list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T327519#10542991 (10Gehel) [08:38:59] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 07git-lfs: wikimedia/discovery/analytics: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316877#10543014 (10Gehel) [09:55:36] urbanecm, tgr|away: Hi, if any of you is around, could someone look into https://phabricator.wikimedia.org/T386171 ? I'm wondering whether to roll back group1 or not [09:56:10] jnuche: fyi ^ (and feel free to share opinions based on your larger experience) [09:56:18] andre: hey, just saw your comment in phab. i have no idea why something would look for `centralauth.logging`, but i can take a look [09:56:45] urbanecm: thanks if you have some time. because I'm torn whether this justifies rolling back or not [10:03:25] andre: hmmm, it looks like the actual failures are coming from the job runners, which would explain why the actual logging hasn't been affected I suppose [10:04:10] to me it looks like a blocker but maybe not worth a rollback? [10:04:23] input from someone else would definitely be helpful [10:05:14] thanks, had the same feeling [10:08:06] I pinged Dreamy_Jazz. I have a feeling this might be causing a regression in Temp account's IP reveal, but I'm not 100% certain. [10:08:53] thank you [10:09:15] I don't believe this is the fault or a regression caused by CheckUser [10:09:34] The code triggering this is through the CentralAuthCreateLocalAccountJob from CentralAuth [10:10:20] As to whether this should cause a rollback, this means that account creation entries are not being created in CheckUser on metawiki [10:10:59] Dreamy_Jazz, ah, so you think we should roll back IIUC? [10:11:32] i'm also confused at actually seeing CU entries on loginwiki, which is also wmf.16 [10:12:02] It's confusing because I can't see why this is the cause [10:12:14] It could lead to more issues, so maybe a rollback would be good until we can work out why [10:14:39] Essentially it seems that somewhere something is overriding the wiki DB returned by `IConnectionProvider::getPrimaryDatabase` to return `centralauth`. [10:14:42] it's leading to some accs not being autocreated at all [10:15:08] andre: i'd rollback as well. [10:15:59] CheckUser gets that from the "ConnectionProvider" service, so I don't think (AFAIK) this is a problem caused by CheckUser. [10:16:04] okay, thanks everyone, let me roll back [10:16:47] jnuche: I'm following https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Rollback and will make "scap train" go back to group0 but I may have stupid questions for you in the next 30min [10:17:17] andre: I'll be around πŸ‘ [10:18:40] jnuche: so if I interpret https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Rollback correctly it's just "scap train"? I wonder where the "Alternatively" starts with those block of commands above [10:19:49] andre: yeah, that's not great wording, using "scap train" to roll back should be the default, not the alternative [10:20:00] "scap train" and then selecting group0 is the way to go [10:20:00] thanks. I can fix docs later [10:20:08] thanks, doing that now [10:25:45] 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T382367#10543579 (10Aklapper) Rolling back group1 from `1.44.0-wmf.16` to `1.44.0-wmf.15` due to T386171 after IRC conversations in `#wikimedia-rel... [10:37:34] (03CR) 10MSantos: [C:03+1] Add bash script to generate confidence report table [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1118874 (https://phabricator.wikimedia.org/T383166) (owner: 10Subramanya Sastry) [11:48:59] 10Continuous-Integration-Infrastructure, 10Castor: Running `cypress` in Wikimedia CI requires unusual env variables - https://phabricator.wikimedia.org/T361624#10543921 (10zeljkofilipin) >>! In T361624#10536042, @thcipriani wrote: > @zeljkofilipin have you ever seen that before? Sorry, beyond a few quick loo... [13:14:03] T386171 fix has been backported to wmf.16 [13:14:04] T386171: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'centralauth.logging' doesn't exist - https://phabricator.wikimedia.org/T386171 [13:16:38] mszabo: Thanks so much for the quick fix! [13:16:49] thanks for flagging this! [13:16:57] Alright, looks like I can use the remaining 43min to run the train again [13:39:55] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10544343 (10Aklapper) Got this again when deploying `1.44.0-wmf.16` to `group1` for the second time today, but with a 302: ` 13:32:3... [14:27:02] 06Release-Engineering-Team: logspam-watch doesn’t handle normalized exceptions well - https://phabricator.wikimedia.org/T347064#10544564 (10Lucas_Werkmeister_WMDE) FTR, this is still happening with other messages – for instance, T386162 is looking extremely spammy in `logspam-watch`. {F58391294} [16:11:07] 10Phabricator: Make it clearer/easier how to silence task batch edits - https://phabricator.wikimedia.org/T386147#10545222 (10bd808) The upstream task dug up by @Pppery perfectly illustrates my `"admin auth controls are too hard; just use shell access"` snark. [17:18:28] (03PS2) 10Jforrester: Zuul: Add Dragoniez to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1118850 (owner: 10Pppery) [17:18:48] (03PS4) 10Jforrester: Zuul: Add User:Michi j to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1118833 (owner: 10Neriah) [17:18:53] (03CR) 10Jforrester: [C:03+2] Zuul: Add Dragoniez to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1118850 (owner: 10Pppery) [17:18:54] (03CR) 10Jforrester: [C:03+2] Zuul: Add User:Michi j to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1118833 (owner: 10Neriah) [17:21:01] (03Merged) 10jenkins-bot: Zuul: Add Dragoniez to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1118850 (owner: 10Pppery) [17:21:13] !log Zuul: Add Dragoniez to CI allowlist [17:21:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:22:25] (03Merged) 10jenkins-bot: Zuul: Add User:Michi j to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1118833 (owner: 10Neriah) [17:22:47] !log Zuul: Add User:Michi j to CI allowlist [17:22:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:50:03] maintenance-disconnect-full-disks build 675422 integration-agent-docker-1050 (/: 27%, /srv: 96%, /var/lib/docker: 36%): OFFLINE due to disk space [17:55:02] maintenance-disconnect-full-disks build 675423 integration-agent-docker-1050 (/: 27%, /srv: 38%, /var/lib/docker: 35%): RECOVERY disk space OK [19:40:03] maintenance-disconnect-full-disks build 675444 integration-agent-docker-1051 (/: 27%, /srv: 96%, /var/lib/docker: 34%): OFFLINE due to disk space [19:45:03] maintenance-disconnect-full-disks build 675445 integration-agent-docker-1051 (/: 27%, /srv: 11%, /var/lib/docker: 32%): RECOVERY disk space OK [19:56:23] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for creynolds - https://phabricator.wikimedia.org/T386260 (10creynolds) 03NEW [19:57:00] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for creynolds - https://phabricator.wikimedia.org/T386260#10546514 (10creynolds) also https://meta.wikimedia.org/wiki/User:CReynolds_(WMF) [20:09:51] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for creynolds - https://phabricator.wikimedia.org/T386260#10546586 (10bd808) a:03bd808 I just added @creynolds to the #trusted-contributors group. The #tool-gitlab-account-approval bot will handle the actual authori... [21:38:11] (03PS2) 10Subramanya Sastry: Reinstate more aggressive false-positive suppression [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1118875 [21:39:05] (03PS3) 10Subramanya Sastry: Add bash script to generate confidence report table [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1118874 (https://phabricator.wikimedia.org/T383166) [21:39:05] (03PS3) 10Subramanya Sastry: Reinstate more aggressive false-positive suppression [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1118875