[00:00:33] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:33] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [01:02:32] 10Continuous-Integration-Config, 10good first task: Remove hardcoded message paths in banana-checker configuration - https://phabricator.wikimedia.org/T346178 (10Krinkle) Are there more than 10 repos in all of Wikimedia Gerrit that don't simply use `i18n/` as the directory for all localisation files? I don't h... [06:41:10] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [libs/JsonCodec] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/959461 [06:41:12] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [libs/JsonCodec] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/959461 (owner: 10QChris) [06:41:21] (03PS1) 10QChris: Import done. Revoke import grants [libs/JsonCodec] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/959472 [06:41:23] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [libs/JsonCodec] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/959472 (owner: 10QChris) [07:44:16] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Create a staging apt repository for CI-based builds of Debian packages - https://phabricator.wikimedia.org/T347004 (10MoritzMuehlenhoff) [08:14:23] 10GitLab: GitLab Private Repository Request for: Wikimedia Stewards - https://phabricator.wikimedia.org/T346937 (10Aklapper) [08:14:30] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, and 3 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Aklapper) [08:58:24] o/ i'm having doubts about rolling train to group 2 due to https://phabricator.wikimedia.org/T347009 (possible bad handling of older cache) and https://phabricator.wikimedia.org/T347007 (possible logspam, at least). [08:58:41] it's highly possible i'm overly paranoid, but i don't know how to handle this [08:59:18] (pinging brennen and jnuche as train conductors, i guess) [09:02:41] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10ihurbain) [09:04:10] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10ihurbain) Just added two blockers for time to have another opinion on them before deploying them to group2.... [09:05:31] ihurbain: hi, the two examples in those bugs are wmf.26 and the train is currently rolling out wmf.27, so maybe it's fine [09:05:40] but could you still flag them here just in case? https://phabricator.wikimedia.org/T345888 [09:05:56] they're flagged [09:05:57] ah, I see you just did :) [09:07:20] mmh. true that it's on .26, I hadn't thought that far. [09:07:43] okay. breathing and having a look. [09:08:39] taking a look too in case I notice something useful [09:09:31] but yeah, no need to stress, train is not being rolled out yet :) [09:09:42] :) [09:10:57] 10Phabricator (Upstream), 10Upstream: Implement "your turn to comment" feature for Phabricator tasks after a username @mention - https://phabricator.wikimedia.org/T346034 (10Aklapper) p:05Triage→03Low [09:12:22] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10Aklapper) p:05Triage→03Medium [09:12:28] 10Phabricator: Tasks created via Conduit API have restricted edit policy (due to general "Default Edit Policy" for Maniphest) - https://phabricator.wikimedia.org/T305998 (10Aklapper) p:05Triage→03Low [09:13:19] 10Phabricator, 10Commons: Review Global Herald H109 - Add #Commons project to #MediaWiki-File-management and #MediaWiki-extensions-GWToolset - https://phabricator.wikimedia.org/T250026 (10Aklapper) @Peachey88: Any better idea how to get feedback from folks? :) [09:13:32] 10Phabricator, 10User-DannyS712: Combine H32 and H194 - https://phabricator.wikimedia.org/T236647 (10Aklapper) 05Open→03Declined [09:16:41] okay. https://phabricator.wikimedia.org/T347007 is on dewiki so it's group2 so it's definitely not related to train. removing that one. [09:17:22] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10ihurbain) [09:18:23] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10ihurbain) Removing https://phabricator.wikimedia.org/T347007 from blockers since it's on dewiki, it's not t... [09:25:20] T347009 looks like there's was a spike essentially at the moment of group1 deploy on group1 wikis, there was like 840 of these between 18:00 and 18:15, and none out of that interval [09:25:21] T347009: PHP Notice: Undefined index: OutputHooks - https://phabricator.wikimedia.org/T347009 [09:25:23] !log gerrit: mark apps/firefox and apps/firefox/wikipedia read-only and archived # T132973 [09:25:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:25:29] T132973: Deprecate the Firefox OS app - https://phabricator.wikimedia.org/T132973 [09:27:03] it feels highly plausible that it's related to https://gerrit.wikimedia.org/r/c/mediawiki/core/+/942499 [09:34:01] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10ihurbain) [09:35:16] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10ihurbain) T347009 is closed as a deployment side effect issue. The same issue and the associated warnings a... [09:36:04] OKAY. anxieties solved, good to go. thanks jnuche for the investigation & support :) [09:36:57] no worries, you were actually faster than me looking into them :) [09:51:45] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Pywikibot, 10Upstream: tox v4 with skipsdist=true does not recognize use_develop=true - https://phabricator.wikimedia.org/T346238 (10hashar) With tox 4, `skipsdist = True` cause it to always skip the package installation (`... [10:00:44] 10GitLab (Integrations), 10collaboration-services: GitLab: Unable to import a repository from gitlab.com - https://phabricator.wikimedia.org/T333048 (10Jelto) 05Open→03Resolved I'll close this task, please re-open if you still have problems importing from GitLab.com [10:05:04] 10Phabricator (Upstream), 10Patch-For-Review, 10Upstream: Hovercards: (RuntimeException) Undefined index: objectPHID - https://phabricator.wikimedia.org/T346876 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/16 Add temporary debug output for T34... [10:05:53] 10Phabricator, 10Commons: Review Global Herald H109 - Add #Commons project to #MediaWiki-File-management and #MediaWiki-extensions-GWToolset - https://phabricator.wikimedia.org/T250026 (10Peachey88) Lets just do it, and then see if anyone complains. Every time I look at the history on the commons project tag... [11:13:01] 10Phabricator (Upstream), 10Upstream: Query for global Feed Transaction Logs: Exception: ConpherenceTransactionQuery overheated - https://phabricator.wikimedia.org/T344232 (10Aklapper) [11:50:10] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Create a staging apt repository for CI-based builds of Debian packages - https://phabricator.wikimedia.org/T347004 (10eoghan) a:03eoghan [12:11:28] 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10phuedx) The wikimedia/metrics-platform Packagist project is now being published to from GitLab. Thanks... [12:16:09] Gerrit question: How to get https://gerrit.wikimedia.org/r/c/953967 deployed? See https://phabricator.wikimedia.org/T345226#9186406 [12:19:19] (03PS1) 10Sophivorus: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/integration/config [integration/config] - 10https://gerrit.wikimedia.org/r/959728 [12:19:21] (03PS1) 10Sophivorus: Zuul: [mediawiki/extensions/GoogleTagManager] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/959729 [12:20:45] (03Abandoned) 10Sophivorus: Zuul: [mediawiki/extensions/GoogleTagManager] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/959729 (owner: 10Sophivorus) [12:27:24] (03PS1) 10Sophivorus: Zuul: [mediawiki/extensions/GoogleTagManager] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/959731 [12:29:00] 10Phabricator, 10Commons: Review Global Herald H109 - Add #Commons project to #MediaWiki-File-management and #MediaWiki-extensions-GWToolset - https://phabricator.wikimedia.org/T250026 (10Aklapper) 05Open→03Resolved Heh, H109 got already disabled in April 2021 and nobody realized (?) or complained so far,... [12:41:53] 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10phuedx) [12:42:41] 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10phuedx) @thcipriani: Is there anything to be done with the mirror on GitHub? [12:43:40] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Create a staging apt repository for CI-based builds of Debian packages - https://phabricator.wikimedia.org/T347004 (10eoghan) [12:55:43] 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10Reedy) >>! In T344733#9186854, @phuedx wrote: > @thcipriani: Is there anything to be done with the mir... [12:55:56] (03Abandoned) 10Jforrester: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/integration/config [integration/config] - 10https://gerrit.wikimedia.org/r/959728 (owner: 10Sophivorus) [12:56:12] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/GoogleTagManager] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/959731 (owner: 10Sophivorus) [12:57:15] 10GitLab (CI & Job Runners), 10collaboration-services: Add additional exporter for GitLab Runner metrics - https://phabricator.wikimedia.org/T347038 (10Jelto) [12:57:25] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/GoogleTagManager] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/959731 (owner: 10Sophivorus) [12:57:55] !log Zuul: [mediawiki/extensions/GoogleTagManager] Add basic CI [12:57:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:11:31] 10GitLab (CI & Job Runners), 10collaboration-services: Add alerts for GitLab Trusted Runner over/underutilization - https://phabricator.wikimedia.org/T347039 (10Jelto) [13:14:24] 10Release-Engineering-Team (Deployment Training Requests), 10Toolforge: Update requests to 2.31.0 on toolforge - https://phabricator.wikimedia.org/T347040 (10Xqt) [14:50:03] 10Release-Engineering-Team: logspam-watch doesn’t handle normalized exceptions well - https://phabricator.wikimedia.org/T347064 (10dancy) [14:54:20] 10Release-Engineering-Team: logspam-watch doesn’t handle normalized exceptions well - https://phabricator.wikimedia.org/T347064 (10Lucas_Werkmeister_WMDE) But the `DBTransactionSizeError` is properly normalized in `/srv/mw-log/exception.log`: ` [Exception Wikimedia\Rdbms\DBTransactionSizeError] (/srv/mediawiki/p... [14:54:21] (03CR) 10Hashar: [C: 03+2] doc: remove Sphinx setuptools integration [integration/quibble] - 10https://gerrit.wikimedia.org/r/959055 (owner: 10Hashar) [14:54:38] (03CR) 10Hashar: [C: 03+2] doc: update Sphinx configuration file [integration/quibble] - 10https://gerrit.wikimedia.org/r/959056 (owner: 10Hashar) [14:54:46] (03CR) 10Hashar: [C: 03+2] release: Quibble 1.5.6 [integration/quibble] - 10https://gerrit.wikimedia.org/r/958952 (https://phabricator.wikimedia.org/T342088) (owner: 10Hashar) [14:55:35] (03CR) 10Hashar: [C: 03+2] release: Start 1.5.7 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/959057 (owner: 10Hashar) [15:08:01] 10Release-Engineering-Team: logspam-watch doesn’t handle normalized exceptions well - https://phabricator.wikimedia.org/T347064 (10dancy) I see what you mean. Interesting. [15:11:47] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10dancy) @xcollazo: Checking in. Are your CI issues resolved with the use of the memory-optimized runner? [15:14:57] (03Merged) 10jenkins-bot: doc: remove Sphinx setuptools integration [integration/quibble] - 10https://gerrit.wikimedia.org/r/959055 (owner: 10Hashar) [15:15:07] (03Merged) 10jenkins-bot: doc: update Sphinx configuration file [integration/quibble] - 10https://gerrit.wikimedia.org/r/959056 (owner: 10Hashar) [15:15:13] (03Merged) 10jenkins-bot: release: Quibble 1.5.6 [integration/quibble] - 10https://gerrit.wikimedia.org/r/958952 (https://phabricator.wikimedia.org/T342088) (owner: 10Hashar) [15:17:46] (03Merged) 10jenkins-bot: release: Start 1.5.7 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/959057 (owner: 10Hashar) [15:19:49] 10Release-Engineering-Team: logspam-watch doesn’t handle normalized exceptions well - https://phabricator.wikimedia.org/T347064 (10Lucas_Werkmeister_WMDE) (At this point I agree it’s not a problem in logspam-watch, but I feel like I don’t know enough yet to retitle the task. I tried looking for more normalized e... [15:22:31] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10brennen) [15:42:46] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T345889 (10thcipriani) [15:53:46] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10xcollazo) >>! In T345000#9187755, @dancy wrote: > @xcollazo: Checking in. Are your CI issues resolved with t... [15:53:54] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for dr0ptp4kt - https://phabricator.wikimedia.org/T347089 (10dr0ptp4kt) [15:54:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10dancy) 05Open→03Resolved [15:58:04] 10MediaWiki-Releasing, 10MW-1.41-release: Announce 1.41 "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T346922 (10thcipriani) 05Open→03Resolved Sent! [15:58:06] 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10thcipriani) [15:58:32] ^^ kindrobot brennen i was wondering if it would be okay if i joined solely to watch for today afternoon and next week thursday afternoon on the backport / deploy training? i'd want to do some further reading and prep ahead of any actual session where i might do stuff hands-on in two weeks on 8-October (don't want to rush any access things and need to wind down some other stuff while i'm amidst project transition, also like to read) [16:02:28] !log Tag Quibble 1.5.6 @ d16af164a7239c58a750b98cb697601eea77327b # T342088 [16:02:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:02:39] T342088: Run Quibble Mariadb server with innodb_print_all_deadlocks - https://phabricator.wikimedia.org/T342088 [16:04:46] Sure, I'd be happy to be there an answer questions. Do you need an invite? [16:08:13] dr0ptp4kt: spectators always welcome. :) [16:08:33] (well, spectators with a signed NDA, at least.) [16:09:21] kindrobot brennen i copied the invitation off the calendar. we'll be in the car today, additionally, so i'll tether, may the 5g powers cooperate [16:09:56] may they for sure cooperate more with you than they do with me [16:12:15] (03PS1) 10Hashar: dockerfiles: update Quibble to 1.5.6 [integration/config] - 10https://gerrit.wikimedia.org/r/959804 (https://phabricator.wikimedia.org/T342088) [16:16:08] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for dr0ptp4kt - https://phabricator.wikimedia.org/T347089 (10dr0ptp4kt) [16:25:37] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for dr0ptp4kt - https://phabricator.wikimedia.org/T347089 (10thcipriani) p:05Triage→03Medium 👋 @dr0ptp4kt I added you to the meeting invite for a bit (since you're interested in watching a few before pushing buttons). We c... [16:27:43] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for dbrant - https://phabricator.wikimedia.org/T344157 (10thcipriani) 05Open→03Resolved a:03thcipriani ## {icon check-circle-o} Training completed! Hey @Dbrant thanks for attending deployment training! It was great chat... [17:58:22] 10GitLab: GitLab Private Repository Request for: Wikimedia Stewards - https://phabricator.wikimedia.org/T346937 (10brennen) [17:58:54] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Stewards-and-global-tools: Create new GitLab project group: stewards - https://phabricator.wikimedia.org/T346924 (10brennen) 05Open→03Resolved a:03brennen `/repos` is pretty much the standard namespace for all things not under the more specifi... [18:00:28] 10GitLab, 10Release-Engineering-Team, 10User-brennen: GitLab Private Repository Request for: Wikimedia Stewards - https://phabricator.wikimedia.org/T346937 (10brennen) 05Open→03Resolved a:03brennen Created: https://gitlab.wikimedia.org/repos/stewards/stewards-scripts @Urbanecm you should have access a... [18:03:43] 10GitLab, 10Release-Engineering-Team, 10User-brennen: GitLab Private Repository Request for: Wikimedia Stewards - https://phabricator.wikimedia.org/T346937 (10Urbanecm) Thanks! This seems to work. [18:03:48] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Stewards-and-global-tools: Create new GitLab project group: stewards - https://phabricator.wikimedia.org/T346924 (10Urbanecm) Thanks! This seems to work. [18:04:51] 10GitLab, 10Release-Engineering-Team, 10User-brennen: GitLab Private Repository Request for: Wikimedia Stewards - https://phabricator.wikimedia.org/T346937 (10Urbanecm) Thanks! This seems to work. [19:18:48] 10WikimediaDebug, 10Observability-Logging, 10Performance-Team, 10Patch-For-Review: Ingest php syslog from Excimer UI (webperf host) into Logstash - https://phabricator.wikimedia.org/T339137 (10colewhite) 05Open→03Resolved I see logs in logstash! \o/ [20:03:52] 10Phabricator, 10Commons: Review Global Herald H109 - Add #Commons project to #MediaWiki-File-management and #MediaWiki-extensions-GWToolset - https://phabricator.wikimedia.org/T250026 (10Peachey88) It probably got replaced with the bot which means it will be a patchset to action