[00:09:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance syslog-server-audit01 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [00:14:49] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` failing, blocking new builds - https://phabricator.wikimedia.org/T424191#11853546 (10DamianZaremba) This seems to break when there is not a `web` process. Grabbing the layer config from an externally built container; ` buildpack... [00:19:10] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` failing, blocking new builds - https://phabricator.wikimedia.org/T424191#11853548 (10DamianZaremba) Workaround is https://github.com/cluebotng/bot/commit/468143e5eb21d0b3b620623294e0f9a454932abb which prevents the default `web` e... [00:19:32] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance syslog-server-audit01 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [00:19:40] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` does not support default Procfile entries - https://phabricator.wikimedia.org/T424191#11853549 (10DamianZaremba) [00:19:48] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24865518502 (https://github.com/cluebotng/component-configs/commits/6e953f35fdca38226cde9ea7280f948f34242881) [00:19:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [04:33:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:33:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [04:38:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:38:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [04:43:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:28:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:33:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:42:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [06:44:22] 06cloud-services-team, 10Cloud-VPS: Set external url for prometheus/thanos metricsinfra web interface - https://phabricator.wikimedia.org/T424312 (10fgiunchedi) 03NEW [06:47:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:46:37] 10Data-Services, 06tools-infrastructure-team, 10Datasets-General-or-Unknown, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Dumps access log analytics should support multiple active hosts - https://phabricator.wikimedia.org/T422042#11854218 (10Gehel) [08:45:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:50:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:16:25] 10Cloud Services Proposals, 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#11854476 (10Gehel) [09:16:35] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11854478 (10Gehel) [09:16:49] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11854480 (10Gehel) [09:36:29] 06cloud-services-team, 10Data-Services, 10Toolforge, 07Documentation: Restructure and improve content for: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database - https://phabricator.wikimedia.org/T232404#11854603 (10komla) >>! In T232404#10558893, @fnegri wrote: > Thanks @taavi this task has been in... [09:57:03] 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#11854651 (10Count_Count) Is it planned to store webservice logs in Loki as well? I don't see a task for it. [09:59:34] 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, and 2 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11854700 (10Gehel) [10:14:07] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411#11854758 (10KBach) 05Open→03In progress [10:41:06] 10Tool-humaniki-2: [Birth-years view] Add a form to filter chart by date range - https://phabricator.wikimedia.org/T422339#11854850 (10Danya) 05Open→03In progress [10:42:18] (03open) 10danyya: Draft: Year range form [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/16 [11:34:16] 10Tools, 06All-and-every-Wikisource: ws-page-game.toolforge.org does not receive page scan thumbnails (HTTP 429) - https://phabricator.wikimedia.org/T419239#11854993 (10MatthewVernon) Looking at this now, I am seeing images as expected, and they look (if my inspector isn't misleading me) to be standard size (5... [12:47:57] 06Toolforge-standards-committee, 10Tools, 10Wikidata, 07Privacy: Tool "wd-vectordb" loads assets from Google gstatic.com - https://phabricator.wikimedia.org/T421889#11855231 (10philippe.saade.WMDE) 05In progress→03Resolved a:03philippe.saade.WMDE [12:54:46] (03update) 10danyya: Draft: Year range form [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/16 [13:04:40] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11855305 (10fgiunchedi) [14:23:53] 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#11855731 (10fnegri) > Is it planned to store webservice logs in Loki as well? I don't see a task for it. The plan is that w... [14:31:50] 06cloud-services-team, 10Cloud-VPS, 10Lingua-Libre: Configure vanity domain for lingualibre - https://phabricator.wikimedia.org/T419525#11855789 (10Michael_Barbereau_WMFr) Hello @dcaro, Our dev instance of Lingua Libre is now up on CloudVPS: https://lingualibre.wmcloud.org/ I have just configured the CNAME... [14:34:37] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 13Patch-For-Review: Linter: Split rules into more logical groupings. - https://phabricator.wikimedia.org/T415914#11855803 (10AGhirelli-WMF) [14:42:28] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#11855849 (10Mooeypoo) [14:45:33] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#11855858 (10Mooeypoo) [14:48:23] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#11855871 (10Mooeypoo) [15:06:55] 10Tool-delintbot, 13Patch-For-Review: Fix cases of tags not being closed correctly - https://phabricator.wikimedia.org/T417483#11855941 (10Redmin) Hello @Kavaljeet_Singh, any update? [15:22:28] 10Toolforge, 06tools-platform-team: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362 (10fnegri) 03NEW [15:33:26] 10Toolforge, 06tools-platform-team: Define update process for Toolforge builder/runner images - https://phabricator.wikimedia.org/T424362#11856056 (10fnegri) [15:37:23] (03update) 10danyya: Draft: Year range form [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/16 [15:37:44] (03open) 10fnegri: README: Clarify process for updating Heroku images [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/88 (https://phabricator.wikimedia.org/T424362) [15:44:32] (03update) 10fnegri: README: Clarify process for updating Heroku images [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/88 (https://phabricator.wikimedia.org/T424362) [15:49:02] (03update) 10fnegri: README: Clarify process for updating Heroku images [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/88 (https://phabricator.wikimedia.org/T424362) [15:50:21] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T423789#11856111 (10fnegri) [15:50:44] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T423790#11856114 (10fnegri) [15:58:03] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T423789#11856130 (10fnegri) [15:58:04] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T423790#11856129 (10fnegri) [15:58:43] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T423789#11856135 (10fnegri) [15:59:00] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T423790#11856138 (10fnegri) [17:33:28] 10Tool-wikimedia-attribution: Document referrer recommendations for Participation CTA - https://phabricator.wikimedia.org/T423567#11856480 (10Sarai-WMF) a:03Sarai-WMF [18:09:18] (03update) 10danyya: Draft: Year range form [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/16 [18:10:02] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance syslog-server-audit01 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:10:30] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#11856643 (10Mooeypoo) [18:12:37] (03update) 10danyya: Year range form [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/16 [18:12:43] (03merge) 10danyya: Year range form [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/16 [18:14:31] 10Tool-humaniki-2: [Birth-years view] Add a form to filter chart by date range - https://phabricator.wikimedia.org/T422339#11856671 (10Danya) 05In progress→03Resolved [18:14:59] (03open) 10danyya: Draft: Front end tests [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/17 [18:20:48] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Test Platform, 07OKR-Work: Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#11856677 (10thcipriani) Tagging in #test_platform folks, they may have suggestions on the best way to integrate this with our current system while keepi... [18:51:48] (03update) 10danyya: Draft: Front end tests [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/17 [18:57:20] (03update) 10danyya: Front end tests [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/17 [18:57:27] (03merge) 10danyya: Front end tests [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/17 [19:03:05] (03open) 10andrew: codfw1dev pawsdev: allow 1 floating ip [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/308 [19:03:26] (03merge) 10andrew: codfw1dev pawsdev: allow 1 floating ip [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/308 [19:07:20] (03open) 10danyya: Draft: Optimize ci [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/18 [19:08:33] (03update) 10danyya: Draft: Optimize ci [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/18 [19:15:49] (03update) 10danyya: Draft: Optimize ci [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/18 [19:18:44] (03update) 10danyya: Draft: Optimize ci [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/18 [19:22:18] (03update) 10danyya: Draft: Optimize ci [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/18 [19:24:05] (03update) 10danyya: Draft: Optimize CI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/18 [19:24:17] (03update) 10danyya: Optimize CI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/18 [19:24:36] (03merge) 10danyya: Optimize CI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/18 [19:27:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:36:26] 10Tool-humaniki-2: Reorganize layout of languages page for better readability - https://phabricator.wikimedia.org/T423786#11856926 (10Danya) 05Open→03In progress [19:37:16] (03open) 10danyya: Draft: Reorganize languages view [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/19 [19:50:58] (03update) 10danyya: Draft: Reorganize languages view [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/19 [20:00:19] (03update) 10danyya: Draft: Reorganize languages view [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/19 [20:05:13] (03update) 10danyya: Draft: Reorganize languages view [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/19 [20:05:53] (03update) 10danyya: Reorganize languages view [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/19 [20:06:17] 10Tool-humaniki-2: Reorganize layout of languages page for better readability - https://phabricator.wikimedia.org/T423786#11856997 (10Danya) 05In progress→03Resolved [20:06:30] (03merge) 10danyya: Reorganize languages view [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/19 [20:48:46] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0) [20:48:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:23:48] (03PS1) 10Lokal Profil: simplify post-merge job [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1276870 (https://phabricator.wikimedia.org/T316627) [21:26:07] (03PS2) 10Lokal Profil: simplify post-merge job [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1276870 (https://phabricator.wikimedia.org/T316627) [21:32:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [21:37:37] (03PS2) 10Lokal Profil: Tighten monuments_config schema and validation tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275532 [21:38:22] (03PS3) 10Lokal Profil: Bump pywikibot to >= 11.0.0 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270570 (https://phabricator.wikimedia.org/T409003) [21:59:08] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24913659688 (https://github.com/cluebotng/component-configs/commits/c5ad64125c64f5db76d9a8141163d380f1c2d4f0) [21:59:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [22:05:12] 10Tool-Pageviews: Add option show moving average line over chart - https://phabricator.wikimedia.org/T147515#11857359 (10Prototyperspective) This is very needed. The Pageviews tool in many – probably most – cases isn't really quite useful because the display of the data is flawed whenever there is/are some spike... [22:13:55] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163) [22:13:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [22:18:09] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278570 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d) [22:18:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [22:19:14] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278528 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d) [22:19:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL