[00:26:26] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10Performance-Team, 10serviceops-radar: Performance review of MediaWiki on Kubernetes for Test2Wiki - https://phabricator.wikimedia.org/T308763 (10Krinkle) The mw-on-k8s project has troughout involved close collaboration between SRE/ServiceOps and Performanc... [02:30:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10ci-test-error: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893 (10tstarling) I saw four of these today, related to the ResourceLoader namespace ch... [02:55:01] 10Project-Admins: Create project tag for Json5Config - https://phabricator.wikimedia.org/T308199 (10Jasonkhanlar) Thank you for assigning the task to me. I saw the task status change since May 12th, and since I initially considered working from scratch, at the time I did not have any code ready to publish that i... [03:23:06] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Certificate gerrit.wikimedia.org expires in 3 day(s) (Sat 28 May 2022 08:33:22 PM GMT +0000). https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [03:27:30] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 27 Jul 2022 08:27:52 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [04:59:02] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Certificate gerrit.wikimedia.org expires in 3 day(s) (Sat 28 May 2022 08:33:22 PM GMT +0000). https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [05:00:17] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 27 Jul 2022 08:27:52 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [07:19:47] (03CR) 10Hashar: "Thanks James!" [tools/code-utils] - 10https://gerrit.wikimedia.org/r/798654 (https://phabricator.wikimedia.org/T309099) (owner: 10Hashar) [07:25:12] (03PS1) 10Hashar: README.md: hint at composer test [tools/code-utils] - 10https://gerrit.wikimedia.org/r/799260 (https://phabricator.wikimedia.org/T309099) [07:25:38] maintenance-disconnect-full-disks build 389043 integration-agent-docker-1036 (/: 30%, /srv: 96%, /var/lib/docker: 50%): OFFLINE due to disk space [07:29:09] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.39.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T308067 (10Ladsgroup) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TimedMediaHandler/+/523987/ * **Summary**:... [07:30:32] maintenance-disconnect-full-disks build 389044 integration-agent-docker-1036 (/: 30%, /srv: 50%, /var/lib/docker: 48%): RECOVERY disk space OK [08:57:13] (03PS1) 10Hashar: Pass shellcheck on shell scripts [tools/code-utils] - 10https://gerrit.wikimedia.org/r/799274 [09:17:58] Project beta-scap-sync-world build #52555: 04FAILURE in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/52555/ [09:18:04] (03PS1) 10Hashar: Zuul: [mediawiki/tools/code-utils] add shellcheck [integration/config] - 10https://gerrit.wikimedia.org/r/799278 (https://phabricator.wikimedia.org/T309099) [09:24:25] 10Project-Admins: Create project tag for <#DSEA-Cluster> - https://phabricator.wikimedia.org/T309095 (10Aklapper) Hi, is there any further information about who is in charge of this? I assume this is a WMF thingy? Is there a wiki page with more info (or a timeframe)? Thanks! [09:27:04] Project mediawiki-core-doxygen-docker build #34396: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/34396/ [09:27:54] (03PS2) 10Hashar: Pass shellcheck on shell scripts [tools/code-utils] - 10https://gerrit.wikimedia.org/r/799274 (https://phabricator.wikimedia.org/T309099) [09:30:24] Yippee, build fixed! [09:30:24] Project beta-scap-sync-world build #52556: 09FIXED in 4 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/52556/ [09:46:27] (03CR) 10Hashar: "Daniel, may you double check the changes I have made to git-hooks/pre-commit-phpcs ? If it works I will deploy the related CI config chan" [tools/code-utils] - 10https://gerrit.wikimedia.org/r/799274 (https://phabricator.wikimedia.org/T309099) (owner: 10Hashar) [10:02:54] hashar: another refactoring workshop slot tonight if you're interested: https://www.mediawiki.org/wiki/Code_Health_Group/projects/DevEd/Workshops#Refactoring_Workshop [10:10:58] (03PS1) 10Robert Vogel: Add `OOJSPlus` as dependency to `EnhancedUpload` [integration/config] - 10https://gerrit.wikimedia.org/r/799284 [10:19:37] Yippee, build fixed! [10:19:38] Project mediawiki-core-doxygen-docker build #34397: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/34397/ [10:36:00] (03CR) 10Matěj Suchánek: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [10:36:58] (03PS3) 10Matěj Suchánek: Do not enforce Postgres for AbuseFilter yet [integration/config] - 10https://gerrit.wikimedia.org/r/798820 [10:37:28] (03CR) 10Matěj Suchánek: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/790660 (owner: 10Jforrester) [10:40:04] (03CR) 10CI reject: [V: 04-1] Do not enforce Postgres for AbuseFilter yet [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [10:40:06] (03PS4) 10Matěj Suchánek: Do not enforce Postgres for AbuseFilter yet [integration/config] - 10https://gerrit.wikimedia.org/r/798820 [10:43:32] (03CR) 10Matěj Suchánek: Zuul: Enforce Postgres and SQLite support for all MW tarball code (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/797319 (owner: 10Jforrester) [10:55:37] maintenance-disconnect-full-disks build 389085 integration-agent-docker-1036 (/: 30%, /srv: 100%, /var/lib/docker: 49%): OFFLINE due to disk space [11:00:34] maintenance-disconnect-full-disks build 389086 integration-agent-docker-1036 (/: 30%, /srv: 57%, /var/lib/docker: 48%): RECOVERY disk space OK [11:11:04] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T305219 (10abi_) [11:55:26] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: - https://phabricator.wikimedia.org/T309194 (10MSantos) [11:58:01] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: - https://phabricator.wikimedia.org/T309194 (10MSantos) [11:58:10] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: - https://phabricator.wikimedia.org/T309194 (10MSantos) [12:04:35] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: content-transform - https://phabricator.wikimedia.org/T309194 (10Peachey88) [12:21:17] (03CR) 10Jforrester: "If we're doing this, we're going to have to drop AbuseFilter from the tarball. Support for PG is a hard requirement for code being in the " [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [12:49:27] (03CR) 10Matěj Suchánek: Do not enforce Postgres for AbuseFilter yet (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [12:51:08] 10Project-Admins: Archive Kaltura Player Phab project and decline all its open tasks - https://phabricator.wikimedia.org/T306971 (10Jdforrester-WMF) [12:56:06] 10Project-Admins: Archive Kaltura Player Phab project and decline all its open tasks - https://phabricator.wikimedia.org/T306971 (10Jdforrester-WMF) [13:01:23] 10Project-Admins: Archive Kaltura Player Phab project and decline all its open tasks - https://phabricator.wikimedia.org/T306971 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [13:17:58] (03CR) 10Jforrester: "`" [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [14:53:47] 10Gerrit, 10SRE: Icinga Check SSL might have a time based race condition - https://phabricator.wikimedia.org/T308908 (10dancy) Another flap happened last night. @RhinosF1 has suggested restarting apache since it still has a worker running from last month. The restart hasn't happened yet though. ` dancy@ge... [15:14:07] (03CR) 10Thcipriani: [C: 03+2] "Thanks for documenting this!" [tools/release] - 10https://gerrit.wikimedia.org/r/788286 (owner: 10Awight) [15:15:18] (03Merged) 10jenkins-bot: Document how to install the Phabricator API token [tools/release] - 10https://gerrit.wikimedia.org/r/788286 (owner: 10Awight) [15:17:23] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T305219 (10matmarex) [15:38:01] 10Gerrit, 10SRE: Icinga Check SSL might have a time based race condition - https://phabricator.wikimedia.org/T308908 (10hashar) From `apache-status`: ` Current Time: Wednesday, 25-May-2022 15:16:44 UTC Restart Time: Friday, 22-Apr-2022 19:59:53 UTC ` ` +-----------------------------------------------... [15:46:47] (03CR) 10Hashar: [C: 03+2] Add `OOJSPlus` as dependency to `EnhancedUpload` [integration/config] - 10https://gerrit.wikimedia.org/r/799284 (owner: 10Robert Vogel) [15:46:52] !log Restarted apache2 on gerrit1001 [15:46:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:47:21] 10Gerrit, 10SRE: Icinga Check SSL might have a time based race condition - https://phabricator.wikimedia.org/T308908 (10dancy) I restarted apache2 on gerrit1001. ` +-----------------------------------------------------------------------+ | | | |Connections |Threads |Async connections... [15:48:41] (03Merged) 10jenkins-bot: Add `OOJSPlus` as dependency to `EnhancedUpload` [integration/config] - 10https://gerrit.wikimedia.org/r/799284 (owner: 10Robert Vogel) [15:51:31] (03CR) 10Hashar: "I guess we should comment out the postgresql from the in-mediawiki-tarball entirely? I imagine some other repos are broken as well." [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [15:54:35] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/799284 (owner: 10Robert Vogel) [15:57:47] 10Phabricator: Unable to preview video in Phabricator task comments and descriptions - https://phabricator.wikimedia.org/T309222 (10Krinkle) [15:57:51] 10Phabricator, 10Release-Engineering-Team: Unable to preview video in Phabricator task comments and descriptions - https://phabricator.wikimedia.org/T309222 (10Krinkle) [16:29:44] James_F: looks like we need to drop postgresql from AbuseFilter due to Echo [16:29:49] I don't think that ever got enforced [16:30:41] https://gerrit.wikimedia.org/r/c/integration/config/+/797319 is probably good. The alternative is to drop postgresql from the `in-mediawiki-tarball` [16:36:46] 10Project-Admins: Create project tag for <#DSE-Cluster> - https://phabricator.wikimedia.org/T309095 (10EChetty) [16:57:58] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech, and 4 others: Move Termbox SSR for Beta Wikidata into deployment-prep project - https://phabricator.wikimedia.org/T304328 (10ItamarWMDE) **Status Update**: # Logic in shell scripts has been updated to allow fetching of images by '... [17:01:50] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech, and 4 others: Move Termbox SSR for Beta Wikidata into deployment-prep project - https://phabricator.wikimedia.org/T304328 (10Majavah) This new instance is failing to run Puppet: ` taavi@deployment-termbox-ssr:~$ sudo run-puppet-age... [17:02:59] 10Project-Admins: Create project tag for <#DSE-Cluster> - https://phabricator.wikimedia.org/T309095 (10BTullis) Hi @Aklapper - thanks for the explanation. I'll try to fill in some of the details as best I can. This project was originally doing by the codename of the //Train Wing// cluster and it was proposed by... [17:13:09] 10Project-Admins: Create project tag for <#DSE-K8S> - https://phabricator.wikimedia.org/T309095 (10BTullis) [18:19:01] (03CR) 10Jforrester: Do not enforce Postgres for AbuseFilter yet (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [18:20:03] hashar: The problem with AF is that it's got a soft dependency on Echo which isn't in the tarball but is cloned for testing, and Echo completely breaks the world for PG. I think instead we should just temporarily drop Echo testing with AF. [18:38:47] !log (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] [[phab:T309230|T309230]] [18:38:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:38:50] T309230: Reset password for Mahir256 on wikifunctions.beta.wmflabs.org - https://phabricator.wikimedia.org/T309230 [18:59:33] James_F: then I guess it is not a new issue. We probably want to drop postgresql from AbuseFilter as proposed in the change [18:59:53] and probably we will want to craft a set of jobs dedicated to the mediawiki tarball [19:00:07] which will clone all the tarballed repos and be triggered from all those repos [19:23:46] (03PS5) 10Hashar: Remove Postgres/Sqlite from AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [19:23:59] (03CR) 10Hashar: "Removed by https://gerrit.wikimedia.org/r/c/integration/config/+/798820" [integration/config] - 10https://gerrit.wikimedia.org/r/790660 (owner: 10Jforrester) [19:24:17] James_F: I am dropping the sqlite and postgresql jobs from AbuseFilter for now [19:28:42] (03CR) 10Hashar: "I have repurposed the change to also drop Sqlite. It might well later end up having the same issue due to some dependency breaking." [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [19:31:12] (03CR) 10C. Scott Ananian: [C: 03+1] Look for mw:moduleStyles meta tag in Parsoid output as well (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/794774 (owner: 10Subramanya Sastry) [19:31:34] (03CR) 10C. Scott Ananian: [C: 03+2] Fix arwiki Cite CSS [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/795721 (owner: 10Subramanya Sastry) [19:42:58] (03CR) 10Hashar: [C: 03+2] Remove Postgres/Sqlite from AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [19:45:21] (03Merged) 10jenkins-bot: Remove Postgres/Sqlite from AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/798820 (owner: 10Matěj Suchánek) [19:57:34] hashar: I don't agree. When I'm out of meetings I'll respond. [20:05:22] James_F: yeah sorry. Then it is a bit unfair to start enforcing postgresql to AbuseFilter when we never ran the job there apparently [20:06:08] anyway +1 on follow up changes ;] [20:06:19] it is holiday here tomorrow [20:23:46] (03PS2) 10Subramanya Sastry: Look for mw:moduleStyles meta tag in Parsoid output as well [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/794774 [20:23:48] (03PS2) 10Subramanya Sastry: Fix arwiki Cite CSS [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/795721 [20:24:33] (03CR) 10Subramanya Sastry: Look for mw:moduleStyles meta tag in Parsoid output as well (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/794774 (owner: 10Subramanya Sastry) [20:35:43] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: scap backport change_url: Deploy to mwdebug - https://phabricator.wikimedia.org/T308476 (10jeena) [20:36:04] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10jeena) [20:36:06] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: scap backport change_url: Deploy to mwdebug - https://phabricator.wikimedia.org/T308476 (10jeena) 05Open→03In progress a:03jeena [22:04:45] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10brennen) [22:04:50] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10brennen) [22:05:43] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10serviceops, 10Patch-For-Review, 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10brennen) To save splitting discussion here,... [22:05:48] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: scap backport change_url: Deploy to mwdebug - https://phabricator.wikimedia.org/T308476 (10jeena) [22:06:06] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10serviceops, 10Patch-For-Review, 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10brennen) 05Stalled→03Resolved a:03bren... [22:08:58] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10brennen) See {T307537} for relevant discussion here. I think consensus in RelEng is th... [22:29:23] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector, 10Accessibility, and 4 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Skins - https://phabricator.wikimedia.org/T301184 (10nray) a:05nray→03bwang [22:30:30] maintenance-disconnect-full-disks build 389224 integration-agent-docker-1036 (/: 30%, /srv: 96%, /var/lib/docker: 52%): OFFLINE due to disk space [22:31:00] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector, 10Accessibility, and 4 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Skins - https://phabricator.wikimedia.org/T301184 (10nray) 05Open→03Resolved Great job @bwang ! [22:35:30] maintenance-disconnect-full-disks build 389225 integration-agent-docker-1036 (/: 30%, /srv: 71%, /var/lib/docker: 52%): RECOVERY disk space OK [23:46:57] 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10Dzahn) The number of repositories on deploy1002 and deploy2002 under /srv/deployment is the same (as intended by the rsync, it uses --delete). The t... [23:51:26] 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10Dzahn) Top ten oldest repos by modifiation time, oldest first: ` Oct 9 2013 elasticsearch Dec 9 2013 scholarships Apr 18 2014 ocg May 30 201... [23:56:34] 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10Dzahn) list of repositories mentioned in `hieradata/role/common/deployment_server/kubernetes.yaml` (same have a repository but also a "scap_reposit... [23:58:42] 10Deployments, 10Release-Engineering-Team (Doing), 10Parsoid, 10SRE, 10bacula: Accidental removal of some files under /srv/deployment on deploy1002 - https://phabricator.wikimedia.org/T307349 (10Dzahn) @hashar also see T309162#7958786