[00:25:24] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:59:56] 10Tool-coverage, 10MediaWiki-Core-Tests, 07Test-Coverage: Figure out how to accurately backfill MediaWiki core test code coverage data - https://phabricator.wikimedia.org/T182750#12366648 (10Legoktm) 05Open→03Resolved a:03Legoktm ` MariaDB [s52302__coverage_p]> select * from coverage where cov_note... [04:25:24] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:49:18] 10Cloud-VPS, 06tools-infrastructure-team: profile::labs::cindermount::srv times out while formatting a 90 GB volume - https://phabricator.wikimedia.org/T439324 (10Physikerwelt) 03NEW [05:49:49] 10Cloud-VPS, 06tools-infrastructure-team: profile::labs::cindermount::srv times out while formatting a 90 GB volume - https://phabricator.wikimedia.org/T439324#12366705 (10Physikerwelt) [05:59:15] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12366712 (10Physikerwelt) @fgiunchedi I discussed the GDPR and privacy issues with @EgonWillighagen and others, and we reached a consensus that it's better to focus on exter... [06:01:40] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: profile::labs::cindermount::srv times out while formatting a 90 GB volume - https://phabricator.wikimedia.org/T439324#12366713 (10Physikerwelt) The patch was originally created to generate a 750GB disk for qlever. [06:05:24] 10Cloud-VPS, 06tools-infrastructure-team: Delete wikiqlever wmflabs project (5238620953c040e7a9effbe47d4e0932) - https://phabricator.wikimedia.org/T439325#12366722 (10Physikerwelt) [06:08:15] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12366727 (10Physikerwelt) 05Open→03Resolved a:03Physikerwelt [06:17:07] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: profile::labs::cindermount::srv times out while formatting a 90 GB volume - https://phabricator.wikimedia.org/T439324#12366730 (10Physikerwelt) [08:25:24] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:20:42] 10PAWS, 06tools-platform-team: Unwanted empty lines automatically created by PAWS when creating pages in el-wiktionary - https://phabricator.wikimedia.org/T439330 (10FocalPoint) 03NEW [10:12:58] 10Tool-clip2commons: Automatically add uploaded video to Wikidata (P10) - https://phabricator.wikimedia.org/T434087#12366827 (10Dev) Implemented and merged. An optional "Wikidata item" field has been added to the upload form; after a successful upload the file is added to the chosen item as a P10 (video) stateme... [10:13:11] 10Tool-clip2commons: Automatically add uploaded video to Wikidata (P10) - https://phabricator.wikimedia.org/T434087#12366828 (10Dev) 05Open→03Resolved [10:59:34] FIRING: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [11:00:05] (03merge) 10countcount: Update pnpm to v12.6.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 (owner: 10renovatebot) [11:00:17] (03merge) 10countcount: Update pnpm to v12.6.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 (owner: 10renovatebot) [11:00:26] (03merge) 10countcount: Update pnpm to v12.6.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 (owner: 10renovatebot) [11:01:05] (03merge) 10countcount: Update pnpm to v12.6.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 (owner: 10renovatebot) [11:01:27] (03merge) 10countcount: Update pnpm to v12.6.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 (owner: 10renovatebot) [11:03:32] 10Tool-clip2commons: iOS WebKit form resets under memory pressure - https://phabricator.wikimedia.org/T434086#12366845 (10Dev) Thanks for turning this around so quickly @Kavaljeet_Singh — the second revision is a real improvement. I pulled `70b7d0b` down and checked it against current `main`: it rebases cleanly,... [11:14:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [11:14:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [11:15:00] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [11:16:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [11:17:47] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [12:25:25] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:42:39] 10Cloud-VPS, 06tools-infrastructure-team: WCMS backup_cinder_volumes unit fails and interrupts if a volume is not available anymore - https://phabricator.wikimedia.org/T439333 (10Volans) 03NEW [13:44:59] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [13:46:11] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [13:58:36] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [14:00:19] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [14:00:41] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12366937 (10Redmin) [14:41:11] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [15:17:55] 10PAWS, 06tools-platform-team: Unwanted empty lines automatically created by PAWS when creating pages in el-wiktionary - https://phabricator.wikimedia.org/T439330#12366996 (10FocalPoint) 05Open→03Resolved a:03FocalPoint and now ... it works properly ... again ... I am marking it Resolved, but the re... [15:55:25] RESOLVED: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:19:36] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:20:19] 10Tool-delintbot, 13Patch-For-Review: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12367019 (10Redmin) [16:30:27] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:32:14] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:39:07] 10Tool-delintbot: Document how to operate the bot - https://phabricator.wikimedia.org/T416070#12367046 (10Redmin) [16:50:10] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [17:04:25] (03merge) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [17:28:19] 10Tool-delintbot: Implement visual regression testing - https://phabricator.wikimedia.org/T415882#12367055 (10Redmin) [18:10:59] (03open) 10waldir: Various styling fixes [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/25 [18:27:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:28:03] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:32:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:33:03] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:48:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:53:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:15:27] (03update) 10waldir: Various styling fixes [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/25 [20:22:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [20:32:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors