[00:31:57] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12165531 (10Andrew) The latest:  - I just now found a Trixie host without an IP. I don't know for certain that it broke post-patch but I fear the worst. We should keep an...
[00:38:46] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: many false positive alertname=InstanceDown - https://phabricator.wikimedia.org/T433173#12165545 (10Andrew) A few spot checks:  accounts-appserver9.account-creation-assistance.eqiad1.wikimedia.cloud: Use the central puppetserver (puppetmaster.cloudinfra.wmflabs.org) and...
[03:08:16] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Send follow up messages to  Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T433334#12165734 (10komla)
[03:37:22] <wikibugs>	 (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59
[03:55:28] <wikibugs>	 (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59
[03:55:35] <wikibugs>	 (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59
[03:58:47] <wikibugs>	 (03merge) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in memory, rebuilt every 10 minutes by a background task [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59
[04:05:24] <wikibugs>	 (03open) 10countcount: Log row count and elapsed time on each probe cache refresh [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/66
[04:08:53] <wikibugs>	 (03merge) 10countcount: Log row count and elapsed time on each probe cache refresh [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/66
[04:16:22] <wikibugs>	 (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61
[05:16:17] <wikibugs>	 (03update) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27
[05:43:49] <wikibugs>	 (03update) 10countcount: List global block exceptions [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64
[05:47:05] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 05MW-1.47-notes (1.47.0-wmf.13; 2026-07-28), 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the Media... - https://phabricator.wikimedia.org/T428153#12165879
[05:47:07] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12165881 (10KBach)
[06:11:02] <wikibugs>	 (03update) 10countcount: List global block exceptions [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64
[06:11:20] <wikibugs>	 (03update) 10countcount: Serve global block exceptions from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64
[06:15:44] <wikibugs>	 (03update) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27
[06:51:25] <wikibugs>	 (03update) 10countcount: Serve global block exceptions from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64
[07:24:56] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166026 (10KBach)
[07:43:18] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166067 (10KBach)
[07:47:36] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [infra] Allow some opt-in scraping solution for tools - https://phabricator.wikimedia.org/T433395#12166073 (10dcaro) >>! In T433395#12164572, @taavi wrote: > In the end it'll need to be a HAProxy map file on the proxy hosts, with an entry per d...
[07:47:59] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166074 (10KBach) 05Open→03Resolved This hypothesis is complete so I'm resolving this task. Any open sub-tasks are either...
[07:48:44] <wikibugs>	 10Toolforge, 06tools-platform-team: [components-api] Allow setting opt-in flags for special scraping control measures - https://phabricator.wikimedia.org/T433397#12166081 (10dcaro)
[07:51:34] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166083 (10KBach)
[07:55:03] <wikibugs>	 10Tool-wikicontest, 07Privacy: WikiEval Tool / WikiContest violates Toolforge Privacy Policy - https://phabricator.wikimedia.org/T427744#12166085 (10Aklapper) 05Open→03Resolved Thank you, this seems to have been fixed in the meantime (please resolve tickets once they are fixed).
[07:58:14] <wikibugs>	 (03update) 10dcaro: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage)
[07:59:02] <wikibugs>	 (03update) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (owner: 10tlepage)
[07:59:48] <wikibugs>	 (03update) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage)
[07:59:56] <wikibugs>	 (03update) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage)
[08:13:14] <wikibugs>	 (03update) 10countcount: Serve global block exceptions from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64
[08:13:29] <wikibugs>	 (03update) 10countcount: Report locally disabled global blocks from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64
[08:55:23] <wikibugs>	 (03close) 10countcount: Replace the OAuth login with an optional SOCKS5 proxy to Toolforge [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/25
[09:14:24] <wikibugs>	 (03update) 10countcount: Report locally disabled global blocks from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64
[09:17:42] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 pageviews START - Cookbook wmcs.vps.create_project for project pageviews in eqiad1 (T433305)
[09:17:44] <stashbot>	 dcaro@cloudcumin1001: Unknown project "pageviews"
[09:17:44] <stashbot>	 dcaro@cloudcumin1001: Did you mean to say "tools.pageviews" instead?
[09:17:44] <stashbot>	 T433305: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305
[09:17:46] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 pageviews END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project pageviews in eqiad1 (T433305)
[09:17:46] <stashbot>	 dcaro@cloudcumin1001: Unknown project "pageviews"
[09:17:47] <stashbot>	 dcaro@cloudcumin1001: Did you mean to say "tools.pageviews" instead?
[09:20:00] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12166397 (10dcaro) Hi @MusikAnimal, @Ottomata, the name `pageviews` is already taken by the web proxy `pageviews.wmcloud.org`, we would need either changing the webproxy, or t...
[09:26:42] <wikibugs>	 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166415 (10dcaro) a:03dcaro
[09:28:19] <wikibugs>	 10Toolforge (Quota-requests), 06tools-platform-team: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166416 (10dcaro)
[09:28:57] <wikibugs>	 10Toolforge (Quota-requests), 06tools-platform-team: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166418 (10dcaro)
[09:29:00] <wikibugs>	 10Toolforge (Quota-requests), 06tools-platform-team: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166419 (10dcaro) 05Open→03In progress
[09:34:35] <wikibugs>	 (03open) 10dcaro: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350
[09:35:52] <wikibugs>	 (03approved) 10tlepage: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350 (owner: 10dcaro)
[09:40:06] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T433184)
[09:40:11] <stashbot>	 T433184: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184
[09:40:19] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T433184)
[09:44:24] <wikibugs>	 (03update) 10dcaro: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350
[09:45:23] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T433184)
[09:45:28] <stashbot>	 T433184: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184
[09:45:36] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T433184)
[10:00:02] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325#12166549 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done -- toolforge now propagates `/mnt/nfs` mounts...
[10:00:09] <wikibugs>	 (03approved) 10dcaro: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350
[10:00:23] <wikibugs>	 (03merge) 10dcaro: statanalizer: increase quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1350
[10:00:34] <wikibugs>	 10Toolforge (Quota-requests), 06tools-platform-team: Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12166554 (10dcaro) 05In progress→03Resolved
[10:20:13] <wikibugs>	 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 03[MWI] FY2025-26 Q4, 07OKR-Work: [Hypothesis] 5.2.5b: Productionalize API spec linting - https://phabricator.wikimedia.org/T422476#12166609 (10KBach)
[10:23:38] <wikibugs>	 (03merge) 10countcount: Report locally disabled global blocks from an in-memory snapshot [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64
[11:17:04] <wikibugs>	 06tools-platform-team: [components-api] ugrade pre-commit (and ruff) - https://phabricator.wikimedia.org/T433492 (10dcaro) 03NEW
[11:17:05] <wikibugs>	 06tools-platform-team: [components-api] ugrade pre-commit (and ruff) - https://phabricator.wikimedia.org/T433492#12166763 (10dcaro) 05Open→03In progress
[11:17:16] <wikibugs>	 06tools-platform-team: [components-api] ugrade pre-commit (and ruff) - https://phabricator.wikimedia.org/T433492#12166764 (10dcaro) p:05Triage→03Medium
[11:17:54] <wikibugs>	 (03open) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177
[11:19:15] <wikibugs>	 (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61
[11:23:07] <wikibugs>	 (03open) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67
[11:23:14] <wikibugs>	 (03update) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67
[11:26:25] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10CFSSL-PKI, 06Infrastructure-Foundations: Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#12166774 (10elukey) 05Open→03Resolved a:03elukey Old VMs deleted!
[11:33:08] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Upgrade IDP server in cloudinfra project off of Bullseye - https://phabricator.wikimedia.org/T402006#12166790 (10SLyngshede-WMF) We do still add the client support, but apparently we lost server support, at least that's my current theory....
[11:49:29] <wikibugs>	 (03update) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67
[11:49:36] <wikibugs>	 (03update) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67
[12:08:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[12:25:21] <wikibugs>	 (03merge) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 (owner: 10renovatebot)
[12:28:50] <wikibugs>	 (03merge) 10countcount: Parse block durations with timelib for PHP strtotime parity [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/67
[12:29:45] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge, 06tools-infrastructure-team: [infra] Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#12166931 (10aputhin)
[12:31:05] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12166932 (10aputhin) p:05Triage→03High
[12:31:21] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: Set up a Toolforge “alerting service” that emails tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12166934 (10aputhin) p:05Triage→03High
[12:31:43] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12166940 (10aputhin) p:05Triage→03High
[12:33:08] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12166943 (10aputhin) p:05Triage→03High a:05dcaro→03None
[12:35:02] <wikibugs>	 10Toolforge, 06tools-platform-team: [components-api] build quota failures leave state unclear - https://phabricator.wikimedia.org/T422951#12166960 (10aputhin) Moving back to backlog until we get more info (i.e. answer to question above)
[12:36:09] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12166962 (10aputhin) p:05Triage→03High
[12:36:44] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12166964 (10aputhin) p:05Triage→03Medium
[12:38:48] <wikibugs>	 (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177
[12:38:51] <wikibugs>	 (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177
[12:40:47] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12166983 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet
[12:49:09] <wikibugs>	 (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177
[12:49:34] <wikibugs>	 (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177
[12:55:48] <wikibugs>	 (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177
[13:08:10] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12167095 (10Marostegui)
[13:08:23] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12167096 (10Marostegui) clouddb1033 pooled in s5 and s8
[13:18:52] <wikibugs>	 (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61
[13:19:32] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24
[13:19:47] <wikibugs>	 (03update) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27
[13:37:15] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12167173 (10aputhin) p:05Triage→03High a:05dcaro→03None
[13:37:36] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12167178 (10aputhin) p:05Triage→03High a:05dcaro→03None
[13:38:20] <wikibugs>	 10Toolforge, 06tools-platform-team: [Toolforge Sustainability Framework] Create an inventory of Toolforge actions - https://phabricator.wikimedia.org/T420559#12167183 (10aputhin) @komla what is missing here to resolve this issue?
[13:38:25] <wikibugs>	 10Toolforge, 06tools-platform-team: [Toolforge Sustainability Framework] Create an inventory of Toolforge actions - https://phabricator.wikimedia.org/T420559#12167184 (10aputhin) p:05Triage→03Medium
[13:40:46] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12167196 (10aputhin) p:05Triage→03High a:05dcaro→03None
[14:09:51] <wikibugs>	 10Toolforge, 06tools-platform-team: [components-api] Allow setting opt-in flags for special scraping control measures - https://phabricator.wikimedia.org/T433397#12167385 (10aputhin)
[14:13:41] <wikibugs>	 10Toolforge, 06tools-platform-team: [components-api] Allow setting opt-in flags for special scraping control measures - https://phabricator.wikimedia.org/T433397#12167397 (10aputhin) p:05Triage→03Medium
[14:14:59] <wikibugs>	 (03open) 10countcount: Switch API_BASE to the multiuserinfo production deployment [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/13
[14:19:19] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: Expose state and webservice_defaults on /images endpoint - https://phabricator.wikimedia.org/T431679#12167413 (10aputhin) p:05Triage→03High
[14:24:10] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: image-config: add webservice-defaults to standard images - https://phabricator.wikimedia.org/T431403#12167446 (10aputhin) 05In progress→03Resolved
[14:24:40] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: resolve some env variable differences between jobs-api and webservice-cli - https://phabricator.wikimedia.org/T431274#12167454 (10aputhin) 05Open→03Resolved
[14:24:54] <wikibugs>	 (03PS3) 10Krinkle: Clarify placeholder: regexp not wrapped in /…/ [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04)
[14:25:30] <wikibugs>	 (03PS4) 10Krinkle: frontend: Clarify placeholder that regexp must not be wrapped in slashes [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04)
[14:27:58] <wikibugs>	 (03CR) 10Krinkle: [C:03+1] "It's a bit noisy as good interfaces should not require this, but I don't have a better idea in the interim. Perhaps once we add plaintext " [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04)
[14:29:38] <wikibugs>	 (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253)
[14:46:35] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] add job publish support - https://phabricator.wikimedia.org/T433355#12167566 (10aputhin) p:05Triage→03High
[14:48:34] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12167578 (10aputhin) p:05Triage→03High
[14:51:11] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team: [webservice-cli] use jobs-api to start the webservice - https://phabricator.wikimedia.org/T431421#12167610 (10Raymond_Ndibe) while getting the webservice-cli to use the jobs-api directly, what we should actually do is to get the webservice-cli to use the job...
[14:53:05] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api, jobs-cli: investigate shell / exec into jobs-api containers - https://phabricator.wikimedia.org/T431276#12167622 (10taavi) Dupe of {T311917}?
[14:53:11] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: Expose state and webservice_defaults on /images endpoint - https://phabricator.wikimedia.org/T431679#12167625 (10dcaro) @Raymond_Ndibe If we go with not using jobs-api directly, this is not needed right? (I think that was my c...
[15:05:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[15:09:11] <wikibugs>	 (03open) 10dcaro: toolforge_deploy: allow following the logs after deploy [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/333
[15:10:13] <wikibugs>	 (03update) 10dcaro: toolforge_deploy: allow following the logs after deploy [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/333
[15:10:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[15:13:04] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 10Wikilink-Tool, 07Essential-Work, 06Moderator-Tools-Team (Kanban): wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12167721 (10MusikAnimal) 05In progress→03Resolved If anybody ho...
[15:13:23] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12167727 (10MusikAnimal) 05In progress→03Resolved If anybody hollers, we'll move it ba...
[15:34:52] <wikibugs>	 (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177
[15:39:32] <wikibugs>	 (03open) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/3 (https://phabricator.wikimedia.org/T428230)
[15:45:05] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12167886 (10cmooney) Reading through the task it strikes me that this might be an issue with DHCP lease renewal.  A set of operations which could lead to this scenario is...
[15:46:59] <wikibugs>	 (03open) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/4 (https://phabricator.wikimedia.org/T428230)
[15:47:03] <wikibugs>	 (03close) 10raymond-ndibe: compile: use PWD instead of hardcoded /app for locale paths [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/3 (https://phabricator.wikimedia.org/T428230)
[15:50:39] <wikibugs>	 (03open) 10raymond-ndibe: testing the locale fix [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/92
[15:50:47] <wikibugs>	 (03update) 10raymond-ndibe: testing the locale fix [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/92
[15:51:52] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 07Epic: Migrate Moderator Tools Cloud VPS tools from Debian Bullseye - https://phabricator.wikimedia.org/T429442#12167926 (10Samwalton9-WMF) 05Open→03Resolved a:03Samwalton9-WMF
[15:51:53] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12167922 (10fnegri) 05In progress→03Resolved
[15:51:59] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Moderator-Tools-Team, 07Epic: Migrate Moderator Tools Cloud VPS tools from Debian Bullseye - https://phabricator.wikimedia.org/T429442#12167928 (10Samwalton9-WMF) a:05Samwalton9-WMF→03None
[16:08:50] <wikibugs>	 (03open) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863)
[16:08:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[16:12:46] <wikibugs>	 (03merge) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6
[16:12:53] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Send follow up messages to  Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T433334#12168058 (10aputhin) 05Open→03In progress
[16:19:41] <wikibugs>	 10Toolforge, 06tools-platform-team: [Toolforge Sustainability Framework] Create an inventory of Toolforge actions - https://phabricator.wikimedia.org/T420559#12168101 (10komla) I'm sure the Toolforge actions list has grown since the last inventory, but since this is tied to the Sustainability work, we can clos...
[16:19:54] <wikibugs>	 10Toolforge, 06tools-platform-team: [Toolforge Sustainability Framework] Create an inventory of Toolforge actions - https://phabricator.wikimedia.org/T420559#12168102 (10komla) 05Open→03Resolved
[16:27:54] <wikibugs>	 (03open) 10taavi: envvars-admission: Update open/elasticsearch related vars [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1351 (https://phabricator.wikimedia.org/T401818)
[16:39:07] <wikibugs>	 (03approved) 10dcaro: envvars-admission: Update open/elasticsearch related vars [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1351 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi)
[16:42:31] <wikibugs>	 (03update) 10fnegri: Draft: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863)
[16:42:43] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12168311 (10cmooney) We were asked about potential network congestion or QoS causing problems here.  There are some very v...
[16:43:24] <wikibugs>	 10Toolforge, 06tools-platform-team: [logs-api] Using --latest N gives the first N lines since the last hour, not the last - https://phabricator.wikimedia.org/T419863#12168322 (10dcaro) Might get fixed by {T428253}
[16:43:46] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253#12168326 (10dcaro) Might fix {T419863}
[16:44:11] <wikibugs>	 (03update) 10dcaro: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe)
[16:46:51] <wikibugs>	 (03update) 10dcaro: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) (owner: 10raymond-ndibe)
[16:47:44] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168403 (10Ottomata) Oh!  Can we just use the existent project then? Probably just needs us added as admins.
[16:52:32] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168448 (10dcaro) >>! In T433305#12168403, @Ottomata wrote: > Oh!  Can we just use the existent project then? Probably just needs us added as admins.  Maybe, the project is c...
[16:53:06] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168453 (10bd808) >>! In T433305#12168403, @Ottomata wrote: > Oh!  Can we just use the existent project then? Probably just needs us added as admins.  The existent project is...
[17:05:49] <wikibugs>	 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12168560 (10CCiufo-WMF) Noting some additional things that came out of a discussion with @dcaro that we still need to think through / answer:  # **We still need to clarify w...
[17:08:08] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168577 (10Ottomata) >  name pageviews is already taken by the web proxy pageviews.wmcloud.org, we would need either Oh, sorry, maybe I misunderstood, but I assumed this mean...
[17:14:24] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168598 (10dcaro) >>! In T433305#12168577, @Ottomata wrote: >>  name pageviews is already taken by the web proxy pageviews.wmcloud.org, we would need either > Oh, sorry, mayb...
[17:30:04] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12168653 (10BLiviero-WMF) Hi!   I got notice from DCOps that clouddb1030-1033 appear to be in "InSetup" Puppet mode.   Just checking to see i...
[17:30:33] <wikibugs>	 10Tool-wikinewsie: Wikidata qualifier statements on Wikinewsie - https://phabricator.wikimedia.org/T433527 (10Pharos) 03NEW
[17:33:28] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168684 (10Ottomata) Ah, got it.   @MusikAnimal in that case, got another name suggestion? Or, should we use the `analytics` project after all?  Probably not given "better pe...
[17:33:51] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168685 (10MusikAnimal) > +1 LGTM, in the future would be good IMHO to explore exactly what went wrong with Toolforge, hosting code there would definitely require less mainte...
[17:38:49] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168693 (10Ottomata) Are there any other relevant tools you might want to have a non commtech VPS project to migrate too?  If so, we could think of some more generic name. Ot...
[17:40:18] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12168694 (10BLiviero-WMF) I should add that the InSetup message I received (regarding these hosts) was July 15 and so these systems may have...
[17:47:09] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168703 (10MusikAnimal) Not on my end. The other VPS instances on the [[ https://openstack-browser.toolforge.org/project/commtech | commtech project ]] are planned to be remo...
[17:54:47] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12168714 (10Marostegui) >>! In T409557#12168694, @BLiviero-WMF wrote: > I should add that the InSetup message I received (regarding these hos...
[18:01:26] <wikibugs>	 10Tool-wikinewsie: Recent sources for Wikinewsie - https://phabricator.wikimedia.org/T433528 (10Pharos) 03NEW
[18:21:26] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168792 (10bd808) >>! In T433305#12168693, @Ottomata wrote: > Are there any other relevant tools you might want to have a non commtech VPS project to migrate too?  If so, we...
[18:27:09] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12168832 (10VRiley-WMF) 05Open→03In progress Moving this now.
[18:29:52] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168839 (10bd808) >>! In T433305#12168685, @MusikAnimal wrote: > Every tool I've ever maintained eventually hits a breaking point on Toolforge if receives a lot of traffic. S...
[18:50:23] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168889 (10Ottomata) Since DE will be owning this...if we could use toolforge I think we would prefer that over a VPS that we have to occasionally replace for debian upgrades...
[18:57:12] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12168920 (10VRiley-WMF) Moved unit into   C8 U26 cable ID: 5346 Port 26  Updating netbox now.
[18:57:26] <wikibugs>	 10Tool-wikimedia-attribution, 06App Growth, 06MediaWiki-API-Platform-Team, 06WikipediaApp, 07OKR-Work: Make mobile app store badges more visible on Wikipedia.org portal page - https://phabricator.wikimedia.org/T432758#12168922 (10HCoplin-WMF)
[19:08:59] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168954 (10MusikAnimal) Given the sysadmin docs on Wikitech, setting up a new VM from scratch should be fairly straightforward. In my experience, I spend a lot more time fixi...
[19:18:42] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12168995 (10Ottomata) >  I am of course keen to respect Data Engineering's desires, as well as Cloud Services' recommendations.  FWIW, I am not strongly opinionated.   > I spe...
[19:29:45] <wikibugs>	 (03update) 10raymond-ndibe: testing the locale fix [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/92
[19:40:49] <wikibugs>	 (03update) 10raymond-ndibe: testing the locale fix [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/92
[19:40:50] <wmcs-alerts>	 FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[19:45:50] <wmcs-alerts>	 RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[19:47:12] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12169083 (10VRiley-WMF) Updating firmware on cloudvirt1048
[20:08:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[20:12:42] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12169163 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1...
[20:26:32] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12169221 (10VRiley-WMF) @fgiunchedi I was able to get through a good majority of this process, but I'm getting a hangup...
[20:44:08] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12169303 (10bd808) >>! In T433305#12168954, @MusikAnimal wrote: > Would it be possible that some day, we impose the API limitations on WMCS as well, and we'd need a floating I...
[20:51:14] <wikibugs>	 (03PS1) 10Perryprog: Add a link to Bitu for account management [labs/striker] - 10https://gerrit.wikimedia.org/r/1319174
[21:00:54] <wikibugs>	 (03CR) 10Perryprog: "I went to the depths of hel... er, ldap and back to test this change fully. Did I need to set-up every single Striker-related service to t" [labs/striker] - 10https://gerrit.wikimedia.org/r/1319174 (owner: 10Perryprog)
[21:13:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[21:18:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[21:26:04] <wmcs-alerts>	 FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud is about to expire in 25d 23h 58m 16s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[21:31:04] <wmcs-alerts>	 FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud is about to expire in 25d 23h 58m 21s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[21:32:50] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12169496 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1048....