[00:38:59] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Chinese-Sites: Incorrect create/edit time displayed in the curation toolbar - https://phabricator.wikimedia.org/T323648 (10Novem_Linguae) cc @SD0001, who has done some date() work on PageTriage before. [01:38:56] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“„), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) files by user 497 should be owned by phd: `find /srv/repos -uid 497 -exec chown phd {} \; ` files by... [02:36:04] ^demon: left some late comments at https://docs.google.com/document/d/1VwZqZTVec_buCci-VcqosUDwZf0msl_KK9DkXdMFcfQ/edit per the email request. Happy to talk or chat anytime if it's still on and you want to hear or share more. [02:37:21] note that the suggestions are mostly aimed at my own past self, I suggested a lot of what we do now, so I'm mostly saying we should stop listening to what I suggested years ago :P [03:36:51] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Chinese-Sites: Incorrect create/edit time displayed in the curation toolbar - https://phabricator.wikimedia.org/T323648 (10SD0001) The i18n strings controlling date formats can be edited from translatewiki.net. During the migration from date.js... [06:28:14] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage: PHP Notice: Undefined index: afc_state - https://phabricator.wikimedia.org/T323647 (10MPGuy2824) I'd guess that the relevant row is missing from the pagetriage_tags table. [06:33:42] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage: PHP Notice: Undefined index: afc_state - https://phabricator.wikimedia.org/T323647 (10TheresNoTime) >>! In T323647#8415866, @MPGuy2824 wrote: > I'd guess that the relevant row is missing from the pagetriage_tags table. Agreed, though the question w... [06:39:49] 10Beta-Cluster-Infrastructure: Enable ContentTranslation for Wikivoyage on the Beta Cluster - https://phabricator.wikimedia.org/T322325 (10TheresNoTime) a:05TheresNoTime→03None unlick, not //currently// working on this [07:27:39] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage: PHP Notice: Undefined index: afc_state - https://phabricator.wikimedia.org/T323647 (10Novem_Linguae) afc_state is also missing in the testwiki2 SQL database. T224705 The relevant code that is supposed to add afc_state appears to be here: https://gi... [07:32:30] 10Continuous-Integration-Config, 10GrowthExperiments, 10Quibble, 10Growth-Team (Current Sprint), and 3 others: api-testing tests break for GrowthExperiments - https://phabricator.wikimedia.org/T320935 (10EUdoh-WMF) Hi @kostajh, can you kindly point me in the direction of the API test runs for GrowthExperim... [08:40:09] 10Beta-Cluster-Infrastructure: admin right to beta commons for testing uploads - https://phabricator.wikimedia.org/T323674 (10Zache) [09:20:18] 10Beta-Cluster-Infrastructure: admin right to beta commons for testing uploads - https://phabricator.wikimedia.org/T323674 (10Zache) @Multichill or somebody, if you have time to check this before tomorrow (I have an IRL meeting with the students) it would be nice. [09:45:02] 10Continuous-Integration-Config, 10GrowthExperiments, 10Quibble, 10Growth-Team (Current Sprint), and 3 others: api-testing tests break for GrowthExperiments - https://phabricator.wikimedia.org/T320935 (10kostajh) >>! In T320935#8415972, @EUdoh-WMF wrote: > Hi @kostajh, can you kindly point me in the direct... [10:36:43] (03PS1) 10Hashar: Makefile: drop reference to blubber [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859988 [10:36:45] (03PS1) 10Hashar: Send Gerrit commits with tag autogenerated:pipelinelib [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 [10:38:17] (03CR) 10Hashar: "That uses the Blubber BuildKit since 41d41be634b98a1f31295f74fb063f71f7248b01 / T318290" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859988 (owner: 10Hashar) [10:39:27] (03CR) 10Hashar: Use Blubber BuildKit frontend instead of blubberoid (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/842543 (https://phabricator.wikimedia.org/T318290) (owner: 10Dduvall) [10:42:02] (03CR) 10Hashar: "The Pipeline Bot got added by 43b64f2c0b1c380636d5878999e07b3aafa69bbf . We have most/all bots tagging their gerrit message with `tag: aut" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 (owner: 10Hashar) [10:42:14] (03PS2) 10Hashar: Send Gerrit review with tag autogenerated:pipelinelib [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 [11:01:51] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Make mw-web and mw-api-ext available behind LVS - https://phabricator.wikimedia.org/T323621 (10Clement_Goubert) p:05Triage→03Medium [11:14:54] 10Release-Engineering-Team (Radar), 10Pywikibot: Share Appveyor account credentials with Release Engineering - https://phabricator.wikimedia.org/T104306 (10Xqt) @jayvdb: are you willing to share your account for https://ci.appveyor.com/project/wikimedia/pywikibot-core e.g. via pm or give admin access to others... [12:55:58] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wdwb-tech, and 6 others: Wikibase test failures on REL1_39 - https://phabricator.wikimedia.org/T322467 (10Reedy) 05Resolved→03Open Any chance of a backport to #mw-1.35-release please? https://gerrit.wikimedia.org... [14:24:41] (03CR) 10Jforrester: layout.yaml: Add doxygen-publish and phpunit-coverage for IPAValidator (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/859615 (https://phabricator.wikimedia.org/T322744) (owner: 10Reedy) [14:24:51] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Wikimedia-Site-requests, 10Patch-For-Review: Update favicon and CentralAuthLoginIcon for wikifunctionswiki - https://phabricator.wikimedia.org/T323627 (10Jdforrester-WMF) >>! In T323627#8415080, @Stang wrote: >>>! In T323627#8415054, @gerritbot wr... [14:27:45] (03CR) 10Reedy: [C: 03+2] layout.yaml: Add doxygen-publish and phpunit-coverage for IPAValidator (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/859615 (https://phabricator.wikimedia.org/T322744) (owner: 10Reedy) [14:31:54] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wdwb-tech, and 6 others: Wikibase test failures on REL1_39 - https://phabricator.wikimedia.org/T322467 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved 1.35 backport: https://gerrit.wikimedia.org/r/c/mediawiki/extensi... [14:51:07] (03PS1) 10Jforrester: Zuul: Provide a mediawiki-php-library template [integration/config] - 10https://gerrit.wikimedia.org/r/860058 [14:51:22] (03CR) 10Jforrester: layout.yaml: Add doxygen-publish and phpunit-coverage for IPAValidator (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/859615 (https://phabricator.wikimedia.org/T322744) (owner: 10Reedy) [14:57:36] 10Continuous-Integration-Config, 10WikiLambda, 10Abstract Wikipedia team (Phase Îș – Clean-up), 10MW-1.39-notes (1.39.0-wmf.22; 2022-07-25), and 2 others: Run some full end-to-end browser integration tests on Beta Cluster Wikifunctions now it exists - https://phabricator.wikimedia.org/T294388 (10Jdforrester-... [14:58:33] (03CR) 10Jforrester: [C: 03+2] Zuul: Provide a mediawiki-php-library template [integration/config] - 10https://gerrit.wikimedia.org/r/860058 (owner: 10Jforrester) [15:00:26] (03Merged) 10jenkins-bot: Zuul: Provide a mediawiki-php-library template [integration/config] - 10https://gerrit.wikimedia.org/r/860058 (owner: 10Jforrester) [15:01:25] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [15:01:35] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Make mw-web and mw-api-ext available behind LVS - https://phabricator.wikimedia.org/T323621 (10Clement_Goubert) 05In progress→03Resolved [15:02:21] (03CR) 10Jforrester: [C: 03+2] zuul: configure CI for operations/debs/varnish-modules [integration/config] - 10https://gerrit.wikimedia.org/r/855991 (https://phabricator.wikimedia.org/T321309) (owner: 10Ssingh) [15:04:09] (03Merged) 10jenkins-bot: zuul: configure CI for operations/debs/varnish-modules [integration/config] - 10https://gerrit.wikimedia.org/r/855991 (https://phabricator.wikimedia.org/T321309) (owner: 10Ssingh) [15:07:38] !log Zuul: configure CI for operations/debs/varnish-modules for T321309 [15:07:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:07:40] T321309: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 [15:12:13] (03PS1) 10Jforrester: Zuul: [avro-php] Use lib template (add doxygen and coverage) [integration/config] - 10https://gerrit.wikimedia.org/r/860061 [15:12:15] (03PS1) 10Jforrester: Zuul: [mediawiki/services/jobrunner] Use lib template (add doxygen and coverage) [integration/config] - 10https://gerrit.wikimedia.org/r/860062 [15:12:17] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/codesniffer] Use lib template (add doxygen) [integration/config] - 10https://gerrit.wikimedia.org/r/860063 [15:12:19] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/phpunit-patch-coverage] Use lib template (add doxygen) [integration/config] - 10https://gerrit.wikimedia.org/r/860064 [15:12:21] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/minus-x] Use lib template (add doxygen and coverage) [integration/config] - 10https://gerrit.wikimedia.org/r/860065 [15:26:41] (03CR) 10Jforrester: [C: 03+2] doc: Link to the Codex docs for the latest release, not the main branch [integration/docroot] - 10https://gerrit.wikimedia.org/r/859627 (https://phabricator.wikimedia.org/T311097) (owner: 10Catrope) [15:27:20] (03Merged) 10jenkins-bot: doc: Link to the Codex docs for the latest release, not the main branch [integration/docroot] - 10https://gerrit.wikimedia.org/r/859627 (https://phabricator.wikimedia.org/T311097) (owner: 10Catrope) [16:16:22] PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:17:59] (03PS1) 10Zfilipin: jjb: Delete selenium-daily-beta-MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/860076 (https://phabricator.wikimedia.org/T280652) [16:59:57] duesen: phan does support `@deprecated` markup but we have them disabled in our `wikimedia/mediawiki-phan-config` at https://github.com/wikimedia/mediawiki-phan-config/blob/ef8c6b086e2c9399125bd18cdde08ccde6ad76ee/src/config.php#L140-L148 [17:00:14] they have been suppressed since for ever https://github.com/wikimedia/mediawiki-phan-config/blob/6616fa4baf15fc5f9a1e78f5f249a2e62d211967/HISTORY.md#030--2018-06-08 [17:00:29] I guess one can potentially send a patch that tweak the phan config to enable those [17:00:43] and maybe we can have a daily job that runs against everything with the deprecation turned on [17:13:28] RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:58:13] (03PS1) 10Hashar: eslint: switch to es2018 [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/860086 [17:59:54] (03CR) 10Hashar: "Gerrit has a few JavaScript plugins in ./plugins which are passed through eslint. I found out I could use the little bit modern es2018 whe" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/860086 (owner: 10Hashar) [18:07:01] (03PS2) 10Hashar: eslint: switch to es2018 [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/860086 [18:10:43] (03CR) 10Hashar: eslint: switch to es2018 (031 comment) [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/860086 (owner: 10Hashar) [19:42:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10serviceops-collab: contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Papaul) [19:49:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10serviceops-collab: contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10LSobanski) [19:49:51] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10LSobanski) 05Stalled→03Open a:05LSobanski→03None [19:51:09] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10Dzahn) also see T313830#8418218 [19:51:22] 10GitLab (Infrastructure), 10serviceops-collab: ensure Gitlab logs end up in logstash - https://phabricator.wikimedia.org/T322261 (10LSobanski) p:05Triage→03Medium [19:52:03] 10Continuous-Integration-Infrastructure, 10serviceops-collab: migrate doc hosts to bullseye - https://phabricator.wikimedia.org/T319477 (10LSobanski) p:05Triage→03Medium [20:12:58] 10Release-Engineering-Team (Priority Backlog đŸ“„), 10Release, 10Train Deployments: 1.40.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T320517 (10Jdlrobson) [20:56:58] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“„), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) After these commands have run we can verify by either: `tree -upfg` (and grep -v for lines without ph... [21:33:26] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Wikimedia-Site-requests: Update favicon and CentralAuthLoginIcon for wikifunctionswiki - https://phabricator.wikimedia.org/T323627 (10Stang) 05Open→03Resolved {F35816514} It works [21:37:21] 10Release-Engineering-Team: Prompted for sudo password during deployment - https://phabricator.wikimedia.org/T323735 (10cjming) [21:38:46] 10Release-Engineering-Team: Prompted for sudo password during deployment - https://phabricator.wikimedia.org/T323735 (10cjming) [21:43:24] 10Release-Engineering-Team, 10Scap, 10User-brennen: Prompted for sudo password during deployment - https://phabricator.wikimedia.org/T323735 (10brennen) Confirming this is reproducible for the mentioned users. I was able to run `scap backport` without incident. I'm guessing something like a missing group f... [21:51:09] 10Release-Engineering-Team, 10Scap, 10User-brennen: Prompted for sudo password during deployment - https://phabricator.wikimedia.org/T323735 (10bd808) The sudoer rule allowing 'ALL = (mwbuilder) NOPASSWD: /usr/local/bin/update-mediawiki-tools-release' is attached to the deployment-ci-admins group in ops/pupp... [21:51:46] brennen: ^ y'all made deploying require deployment-ci-admins membership accidentally [21:53:22] that'll do it [21:56:04] that script looks pretty boring -- https://gerrit.wikimedia.org/g/operations/puppet/+/bb46a57262c553cf5822fd7bd06bffd9252d2399/modules/profile/manifests/kubernetes/deployment_server/mediawiki/builder.pp#34 -- so I'd guess the sudoers rule can be attached to the deployers group without a lot of controversy [21:59:37] (aside: scap doesn't seem to be indexed by codesearch anymore? At least an "everything" scoped search for "update-mediawiki-tools-release" isn't hitting on anything in https://gitlab.wikimedia.org/repos/releng/scap) [22:00:12] https://gitlab.wikimedia.org/repos/releng/scap isn't mirrored to github anymore either which is unfortunate for quick code searches too [22:04:40] https://phabricator.wikimedia.org/T268196 :\ [22:06:05] I think K.rinkle's comment about discovery during "the time of transition" is spot on [22:08:08] 10Release-Engineering-Team (Priority Backlog đŸ“„), 10Release, 10Train Deployments: 1.40.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T320517 (10Jdlrobson) [22:08:58] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“„), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [22:09:30] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“„), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) created migration plan v2: {P40847} [22:21:57] 10Release-Engineering-Team, 10Scap, 10User-brennen: Prompted for sudo password during deployment - https://phabricator.wikimedia.org/T323735 (10brennen) Yep, that sure seems like it'd do it. @jnuche I'm guessing this is a side effect of {77bcab70}? From `#wikimedia-releng`: ` that script looks pre... [22:23:35] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review, 10User-brennen: Prompted for sudo password during deployment - https://phabricator.wikimedia.org/T323735 (10brennen) p:05Triage→03High [22:30:18] bd808: yeah, it should be a priority. codesearch is way too load bearing in our process to have stuff like this missing from it. [22:31:53] the search built into gitlab seems to be not super fancy either [22:32:52] it is observably hamstrung for sales funnel reasons. [22:33:52] ugh [22:34:01] does anything in deployment-prep rely on the sessionstore instance there? [22:34:25] meanwhile, i suppose there is a sense in which no mere-mortal deployers being able to deploy constitutes an UBN, isn't there. [22:35:12] brennen: it does seem like a bad way to leave things over a long weekend [22:36:43] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“„): Ensure phab1004/phab2002 are scap3 targets - https://phabricator.wikimedia.org/T313954 (10brennen) 05Open→03Resolved [22:36:47] 10Phabricator, 10Release-Engineering-Team (Priority Backlog đŸ“„), 10Patch-For-Review, 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10brennen) [22:37:09] urandom: MediaWiki? (/me has not looked at the config) [22:39:30] urandom: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/LabsServices.php#L98 [22:40:27] bd808: that's interesting, because the echostore instance was shutdown some time ago (it had been running Stretch), and I haven't heard any complaints [22:40:42] *shrug* I don't know exactly how that is plumbed in, but I would expect that key being set means that MediaWiki is using that sessionstore instance. [22:40:51] bd808: I just accidentally the whole sessionstore instance and am trying to figure out how big a deal that is [22:41:10] I'd just recreated it, making that moot, but that is failing spectacularly [22:42:30] urandom: I can't login to https://meta.wikimedia.beta.wmflabs.org/ so I would guess this is a sign [22:42:35] s/recreated/recreate/ [22:42:39] "There seems to be a problem with your login session; this action has been canceled as a precaution against session hijacking. Please resubmit the form." [22:42:40] * urandom sighs [22:44:11] I would just (re)create it, but puppet blows up on my vms [22:44:22] I'm going to move this to -cloud tho [22:45:30] 10Continuous-Integration-Config, 10GrowthExperiments, 10Quibble, 10Growth-Team (Current Sprint), and 3 others: api-testing tests break for GrowthExperiments - https://phabricator.wikimedia.org/T320935 (10EUdoh-WMF) I see it, thanks a lot. The tests are not failing with the earlier identified error anymore.