[00:12:23] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) [03:30:33] 10Tool-wikinewsie: Wikinewsie suggested headline from AI recap - https://phabricator.wikimedia.org/T436419#12331842 (10Pharos) I was also thinking of where the "Generate AI recap" button should go, as it's recently been moved from the top of the article card to the bottom. That's a reasonable accommodation for o... [05:09:27] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-rdb01 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436480#12331904 (10Ladsgroup) Hi, I think the simplest solution would be to create a new VM with redis, set the password accordingly and then just flip... [05:42:01] 06cloud-services-team, 10Data-Services, 06tools-platform-team: Upgrade clouddb* hosts to Debian Trixie - https://phabricator.wikimedia.org/T428047#12331923 (10Marostegui) @fnegri can we resume this? Obviously we have hosts that do not need this: clouddb1013-clouddb1020 https://phabricator.wikimedia.org/T434048 [06:53:18] 10Cloud-VPS, 06tools-infrastructure-team: Establish an automated cinder volume restore procedure and testing - https://phabricator.wikimedia.org/T438051#12332021 (10fgiunchedi) [06:53:36] 10Tool-projo: Projo: Prevent repeated topic clicks from dispatching duplicate requests - https://phabricator.wikimedia.org/T438289 (10Audiodude) 03NEW [07:18:52] (03approved) 10phedenskog: Update timing cache data before server deployment [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/16 (https://phabricator.wikimedia.org/T437796) (owner: 10arthurtaylor) [07:19:49] (03merge) 10arthurtaylor: Update timing cache data before server deployment [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/16 (https://phabricator.wikimedia.org/T437796) [07:54:13] 10Cloud-VPS, 06tools-infrastructure-team: Establish an automated cinder volume restore procedure and testing - https://phabricator.wikimedia.org/T438051#12332144 (10Volans) As I imagine that the restore process by backy2 can be a lengthy process depending on the size of the volume, I suggest to decouple it fr... [08:03:53] (03CR) 10Raymond Ndibe: [C:03+2] webservice-runner: default to 8000 only if PORT and TOOL_WEB_PORT has no value [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) (owner: 10Raymond Ndibe) [08:04:36] (03Merged) 10jenkins-bot: webservice-runner: default to 8000 only if PORT and TOOL_WEB_PORT has no value [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) (owner: 10Raymond Ndibe) [08:34:52] (03approved) 10tlepage: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) (owner: 10dcaro) [08:47:46] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:51:08] 06cloud-services-team, 10Data-Services, 06tools-platform-team: Upgrade clouddb* hosts to Debian Trixie - https://phabricator.wikimedia.org/T428047#12332377 (10fnegri) [08:52:00] 06cloud-services-team, 10Data-Services, 06tools-platform-team: Upgrade clouddb* hosts to Debian Trixie - https://phabricator.wikimedia.org/T428047#12332386 (10fnegri) @Marostegui sure, I added the list of the missing ones to the task description. [08:52:26] 06cloud-services-team, 10Data-Services, 06tools-platform-team: Upgrade clouddb* hosts to Debian Trixie - https://phabricator.wikimedia.org/T428047#12332388 (10fnegri) a:03fnegri [08:52:46] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:03:06] 10Tool-inteGraality: Persistence backend for integraality - https://phabricator.wikimedia.org/T436062#12332433 (10JeanFred) 05Open→03Resolved [09:05:17] 10Tool-inteGraality: Track integraality run history - https://phabricator.wikimedia.org/T438301 (10JeanFred) 03NEW [09:06:04] 10Tool-inteGraality: Track integraality run history - https://phabricator.wikimedia.org/T438301#12332449 (10JeanFred) [09:06:05] 10Tool-inteGraality: Dashboard list/status page - https://phabricator.wikimedia.org/T435900#12332450 (10JeanFred) [09:12:03] (03update) 10taavi: eqiad1: Remove references to legacy VLAN metricsinfra VIPs [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 [09:12:03] (03update) 10taavi: project: Add shortcut for metricsinfra scraping secgroup rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:12:05] (03update) 10taavi: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) [09:12:22] (03open) 10taavi: eqiad1: Remove references to legacy VLAN metricsinfra VIPs [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 [09:12:23] (03open) 10taavi: project: Add shortcut for metricsinfra scraping secgroup rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:12:24] (03open) 10taavi: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) [09:12:36] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:12:56] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:14:37] (03update) 10taavi: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) [09:14:38] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 [09:14:38] (03update) 10taavi: project: Add shortcut for metricsinfra scraping secgroup rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:15:02] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 [09:15:05] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:15:25] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:16:12] (03update) 10taavi: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) [09:16:12] (03update) 10taavi: project: Add shortcut for metricsinfra scraping secgroup rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:16:12] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:16:33] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:17:40] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:17:42] (03update) 10taavi: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) [09:17:43] (03update) 10taavi: project: Add shortcut for metricsinfra scraping secgroup rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:18:04] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:18:57] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 [09:19:22] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 [09:20:16] (03update) 10taavi: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) [09:20:20] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 [09:20:44] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 [09:26:07] 10Tool-inteGraality: Add functional tests running SPARQL queries against live endpoints - https://phabricator.wikimedia.org/T438304 (10JeanFred) 03NEW [09:26:37] 10Tool-inteGraality: Add functional tests running SPARQL queries against live endpoints - https://phabricator.wikimedia.org/T438304#12332539 (10JeanFred) Done in {ea800c526585b9a12ab9a36ee7d42761085e2f15} [09:27:26] 10Tool-inteGraality: Add functional tests running SPARQL queries against live endpoints - https://phabricator.wikimedia.org/T438304#12332543 (10JeanFred) [09:37:24] 10Tool-inteGraality, 07Epic: Dashboard list/status page - https://phabricator.wikimedia.org/T435900#12332576 (10JeanFred) [09:38:40] 10Tool-inteGraality: Track integraality run history - https://phabricator.wikimedia.org/T438301#12332580 (10JeanFred) [09:47:04] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12332616 (10dcaro) When introducing deployment event too (multiple sources), there's two ways of handling if the user sends no... [09:51:47] (03approved) 10filippo: eqiad1: Remove references to legacy VLAN metricsinfra VIPs [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 (owner: 10taavi) [09:52:26] (03approved) 10filippo: project: Add shortcut for metricsinfra scraping secgroup rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 (owner: 10taavi) [09:52:39] (03approved) 10filippo: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) (owner: 10taavi) [09:53:03] (03merge) 10taavi: eqiad1: Remove references to legacy VLAN metricsinfra VIPs [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 [09:53:11] (03update) 10taavi: project: Add shortcut for metricsinfra scraping secgroup rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:53:17] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:53:38] (03update) 10dcaro: Draft: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [09:53:54] (03update) 10dcaro: Draft: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [09:55:02] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [09:55:09] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:55:30] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:55:56] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:55:57] (03merge) 10taavi: project: Add shortcut for metricsinfra scraping secgroup rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/355 [09:56:05] (03update) 10taavi: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) [09:56:53] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [09:57:22] (03merge) 10taavi: eqiad1: cloudinfra: Allow metricsinfra to scrape etcd metrics [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/356 (https://phabricator.wikimedia.org/T410721) [09:57:59] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:59:02] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:29:14] (03close) 10dcaro: celery: upgrade to a version without the timeout bug [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/4 [10:39:18] (03update) 10dcaro: Draft: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [10:52:15] 10PAWS, 06tools-platform-team: PAWS cannot be activated for my account FocalPointBot - https://phabricator.wikimedia.org/T437766#12332800 (10Aklapper) a:05FocalPoint→03None [10:58:07] 10Tool-inteGraality, 07Epic: Health and discoverability - https://phabricator.wikimedia.org/T435900#12332815 (10JeanFred) [10:58:45] 10Tool-inteGraality: Registry of dashboards - https://phabricator.wikimedia.org/T438313 (10JeanFred) 03NEW [11:00:41] 10Tool-inteGraality: Health and discoverability - https://phabricator.wikimedia.org/T438313#12332837 (10JeanFred) [11:00:58] 10Tool-inteGraality: Registry of dashboards - https://phabricator.wikimedia.org/T435900#12332844 (10JeanFred) [11:01:14] 10Tool-inteGraality, 07Epic: Health and discoverability - https://phabricator.wikimedia.org/T438313#12332847 (10JeanFred) [11:01:53] 10Tool-inteGraality: Track integraality run history - https://phabricator.wikimedia.org/T438301#12332848 (10JeanFred) [11:01:54] 10Tool-inteGraality, 07Epic: Health and discoverability - https://phabricator.wikimedia.org/T438313#12332849 (10JeanFred) [11:01:55] 10Tool-inteGraality: Registry of dashboards - https://phabricator.wikimedia.org/T435900#12332850 (10JeanFred) [11:01:57] 10Tool-inteGraality: Persistence backend for integraality - https://phabricator.wikimedia.org/T436062#12332851 (10JeanFred) [11:01:58] 10Tool-inteGraality, 07Epic: Health and discoverability - https://phabricator.wikimedia.org/T438313#12332852 (10JeanFred) [11:02:03] 10Tool-inteGraality: Registry of dashboards - https://phabricator.wikimedia.org/T435900#12332855 (10JeanFred) [11:02:05] 10Tool-inteGraality, 07Epic: Health and discoverability - https://phabricator.wikimedia.org/T438313#12332856 (10JeanFred) [11:05:05] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [11:15:18] 10Tool-inteGraality: Syntax-check SPARQL queries before sending them to the endpoint - https://phabricator.wikimedia.org/T437894#12332951 (10JeanFred) [11:17:00] 10Tool-inteGraality: Ability to re-parse integraality results from Wikitext tables - https://phabricator.wikimedia.org/T438317 (10JeanFred) 03NEW [11:28:17] 10Tool-Global-user-contributions, 10Toolforge, 06tools-platform-team: Linking toolforge:guc with URL arguments leads to 404 as ? is turned into %3F - https://phabricator.wikimedia.org/T438320 (10nclm) 03NEW [11:28:39] 10Tool-Global-user-contributions, 10Toolforge, 06tools-platform-team: Linking toolforge:guc with URL arguments leads to 404 as ? is turned into %3F - https://phabricator.wikimedia.org/T438320#12333022 (10nclm) [11:30:50] 10Tool-Global-user-contributions, 10Toolforge, 06tools-platform-team: Linking toolforge:guc with URL arguments leads to 404 as ? is turned into %3F - https://phabricator.wikimedia.org/T438320#12333029 (10nclm) [11:33:57] (03open) 10phedenskog: Disable the gunicorn control socket [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/17 [11:34:08] (03update) 10phedenskog: Disable the gunicorn control socket [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/17 [11:46:23] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12333108 (10TLepage-WMF) The latter approach requires logs-api to be aware of the dependency between a source and its relevant... [12:00:22] (03open) 10rodrigoargenton: Draft: V0 41 sqlite clean [toolforge-repos/fotowikibr-review] - 10https://gitlab.wikimedia.org/toolforge-repos/fotowikibr-review/-/merge_requests/1 [12:06:50] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/48 [12:07:21] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1342646 (owner: 10L10n-bot) [12:12:13] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12333235 (10taavi) a:03taavi For future me reference: `lang=shell-session taavi@cloudcontrol1007 ~ $ sudo wmcs-webproxy -p commtech list domain... [12:13:47] !log taavi@cloudcumin1001 pageviews START - Cookbook wmcs.vps.create_project for project pageviews in eqiad1 [12:13:49] taavi@cloudcumin1001: Unknown project "pageviews" [12:13:49] taavi@cloudcumin1001: Did you mean to say "tools.pageviews" instead? [12:14:22] !log taavi@cloudcumin1001 pageviews END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project pageviews in eqiad1 [12:14:22] taavi@cloudcumin1001: Unknown project "pageviews" [12:14:22] taavi@cloudcumin1001: Did you mean to say "tools.pageviews" instead? [12:14:35] !log taavi@cloudcumin1001 pageviews START - Cookbook wmcs.vps.create_project for project pageviews in eqiad1 [12:14:35] taavi@cloudcumin1001: Unknown project "pageviews" [12:14:35] taavi@cloudcumin1001: Did you mean to say "tools.pageviews" instead? [12:15:17] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project pageviews [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/357 [12:15:54] (03merge) 10taavi: projects: added project pageviews [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/357 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [12:16:53] !log taavi@cloudcumin1001 pageviews END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project pageviews in eqiad1 [12:16:53] taavi@cloudcumin1001: Unknown project "pageviews" [12:16:53] taavi@cloudcumin1001: Did you mean to say "tools.pageviews" instead? [12:20:32] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12333301 (10taavi) 05Open→03Resolved Project `pageviews` created, and moved the web proxy definition to that project (even if it still pointing to the old `commtech` i... [12:27:42] 10Cloud-VPS, 06tools-infrastructure-team: Upgrade project-proxy proxies to Trixie - https://phabricator.wikimedia.org/T438333 (10taavi) 03NEW [12:40:00] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) [12:44:47] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' [12:44:55] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'proxy' [12:46:18] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' [12:47:05] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12333450 (10fnegri) Per the discussion in {T433690}, storing the opt-in preference in the tool configuration data structure could look like `global.alerts.enabled:... [12:48:18] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) [12:51:03] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-7 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [12:52:02] !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'proxy' [12:53:56] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' [12:54:02] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.refresh_puppet_certs on proxy-7.project-proxy.eqiad1.wikimedia.cloud [12:54:03] FIRING: [2x] TargetDown: Job main-haproxy is unreachable in project project-proxy instance proxy-7 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:56:37] !log taavi@cloudcumin1001 project-proxy END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on proxy-7.project-proxy.eqiad1.wikimedia.cloud [13:00:10] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12333503 (10dcaro) Let me put them in a tool config format: ` source_url: https://path.to.my/config.yaml alerts: enabled: true componets: mycomponent1: .... [13:01:03] FIRING: [2x] ProjectProxyMainProxyInstanceDown: Proxy on proxy-7 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [13:01:58] !log taavi@cloudcumin1001 project-proxy END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'proxy' [13:02:12] 10Cloud-VPS, 06tools-infrastructure-team: Upgrade project-proxy proxies to Trixie - https://phabricator.wikimedia.org/T438333#12333514 (10taavi) a:03taavi Wrote docs for this process: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Web_proxy#Add_a_new_proxy_node [13:04:03] FIRING: [4x] TargetDown: Job main-haproxy is unreachable in project project-proxy instance proxy-7 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:04:55] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [13:06:36] 10Cloud-VPS, 06tools-infrastructure-team: cloudinfra-db backups - https://phabricator.wikimedia.org/T301640#12333535 (10taavi) a:03taavi I am tackling this now since it makes upgrading the proxy hosts (T438333) somewhat simpler. [13:09:40] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-backup' [13:09:45] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'cloudinfra-backup' [13:10:29] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.openstack.quota_increase by 10 server-groups [13:10:37] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 10 server-groups [13:11:18] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-backup' [13:13:06] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [13:18:22] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-backup' [13:19:39] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'cloudinfra-backup' [13:19:39] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:22:09] (03PS1) 10Majavah: Add cloudinfra backup pass [labs/private] - 10https://gerrit.wikimedia.org/r/1342676 (https://phabricator.wikimedia.org/T301640) [13:22:37] (03CR) 10Majavah: [V:03+2 C:03+2] Add cloudinfra backup pass [labs/private] - 10https://gerrit.wikimedia.org/r/1342676 (https://phabricator.wikimedia.org/T301640) (owner: 10Majavah) [13:26:49] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'cloudinfra-backup' [13:26:51] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:26:52] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12333679 (10fnegri) Discussing it in a meeting, we're discussing about another option: `lang=yaml source_url: https://path.to.my/config.yaml defaults: alerts:... [13:27:03] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12333680 (10dcaro) a:03dcaro [13:28:03] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:33:03] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:42:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance project-proxy-puppetserver-1 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:44:13] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.openstack.quota_increase by 4 volumes [13:44:15] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:44:21] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 4 volumes [13:44:22] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:57:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance project-proxy-puppetserver-1 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:58:30] 10Tool-ghanasupremecases1, 10Toolforge, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12333880 (10taavi) 05Open→03Invalid Closing due to lack of response. For proper support venues plea... [14:04:01] 10Cloud-VPS (Debian Bullseye Deprecation), 06Collaboration-Services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12333932 (10komla) @Arnoldokoth how's the progress on this? [14:09:01] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wikidocumentaries away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435789#12333958 (10komla) @Zache any luck with progress on this? [14:09:32] 06tools-platform-team: toollabs-images: bullseye image rebuilds fail. decision needed - https://phabricator.wikimedia.org/T438356 (10Raymond_Ndibe) 03NEW [14:09:41] 06tools-platform-team: toollabs-images: bullseye image rebuilds fail. decision needed - https://phabricator.wikimedia.org/T438356#12333972 (10Raymond_Ndibe) a:03Raymond_Ndibe [14:10:48] 10wikitech.wikimedia.org, 10Parsoid-Read-Views, 06Content-Platform-Team (Work In Progress): Parsoid Read Views: Deploy to wikitech - https://phabricator.wikimedia.org/T437916#12333977 (10OSleger-WMF) [14:10:50] 06tools-platform-team: toollabs-images: bullseye image rebuilds fail. decision needed - https://phabricator.wikimedia.org/T438356#12333978 (10Raymond_Ndibe) We should also release newer version of these images so we can comfortably deprecate these ones, but again, not today obviously. [14:13:08] 06tools-platform-team: toollabs-images: bullseye image rebuilds fail. decision needed - https://phabricator.wikimedia.org/T438356#12333992 (10Raymond_Ndibe) @dcaro curious to hear if you prefer a particular approach. I prefer `Preferably, for each of the affected images, we may be able to do a patch like this... [14:36:49] 10Toolforge, 06tools-platform-team: toollabs-images: bullseye image rebuilds fail. decision needed - https://phabricator.wikimedia.org/T438356#12334190 (10Raymond_Ndibe) [14:39:34] 10Cloud-VPS (Project-requests): Request creation of Trove Database for Broad Interwiki Enhanced Retrieval VPS project - https://phabricator.wikimedia.org/T438363 (10WikiBayer) 03NEW [14:39:39] 06tools-platform-team: [paws] Do a 1h check with @fnegri on bad behavior detection - https://phabricator.wikimedia.org/T438364 (10dcaro) 03NEW [14:39:50] 10PAWS, 06tools-platform-team: [paws] Do a 1h check with @fnegri on bad behavior detection - https://phabricator.wikimedia.org/T438364#12334232 (10dcaro) [14:39:57] 10PAWS, 06tools-platform-team: [paws] Do a 1h check with @fnegri on bad behavior detection - https://phabricator.wikimedia.org/T438364#12334235 (10dcaro) p:05Triage→03Medium [14:40:33] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12334238 (10dcaro) 05Open→03In progress [14:40:49] (03open) 10dcaro: config: add defaults.alerts.enable setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:45:21] (03update) 10dcaro: config: add defaults.alerts.enable setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:45:22] (03update) 10dcaro: config: add defaults.alerts.enable setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [14:45:38] 10Data-Services, 06tools-infrastructure-team: clouddb1023 memory alert - https://phabricator.wikimedia.org/T438200#12334287 (10Marostegui) This is not an uncommon thing for clouddb* hosts given their workload and the occasionally heavy queries they can receive. I was a bit worried it could have come from the... [14:53:46] (03update) 10dcaro: config: add defaults.alerts.enable setting [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/191 (https://phabricator.wikimedia.org/T432864) [15:07:33] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: More granular 'wmcs' clusters for better observability - https://phabricator.wikimedia.org/T437272#12334417 (10Andrew) this looks right to me! [15:11:25] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: More granular 'wmcs' clusters for better observability - https://phabricator.wikimedia.org/T437272#12334469 (10taavi) > wmcs::openstack::*::services > wmcs::openstack::*::net These are actually the DNS and networking data plane and have no control... [15:20:42] FIRING: AlertLintProblem: Linting problems found for CloudVPSMagnumDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:24:27] (03approved) 10dcaro: jobs-cli: warn on duplicates and report loaded count on load [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/181 (https://phabricator.wikimedia.org/T438202) (owner: 10raymond-ndibe) [16:08:22] 10PAWS, 06tools-platform-team: [paws] Do a 1h check with @fnegri on bad behavior detection - https://phabricator.wikimedia.org/T438364#12334819 (10dcaro) @fnegri do you want to do this during the monday pairing slot? [16:23:24] FIRING: [3x] SystemdUnitFailed: magnum-api.service on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:24:33] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Traffic: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12334903 (10BCornwall) 05Open→03Resolved deployment-cache-(text|upload)08 have been shut down... [16:26:49] 10Tool-projo: Projo: Add .swiki export option to the Improve page - https://phabricator.wikimedia.org/T438377 (10Audiodude) 03NEW [16:27:12] RESOLVED: AlertLintProblem: Linting problems found for CloudVPSMagnumDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:28:12] FIRING: AlertLintProblem: Linting problems found for CloudVPSMagnumDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:28:57] 10PAWS, 06tools-platform-team: [paws] Do a 1h check with @fnegri on bad behavior detection - https://phabricator.wikimedia.org/T438364#12334921 (10fnegri) @dcaro yes I think that works! [16:33:12] RESOLVED: AlertLintProblem: Linting problems found for CloudVPSMagnumDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:38:46] 10Tool-wiki-gender-stats: Add a search bar and sorting options in the aggregated regions dialog - https://phabricator.wikimedia.org/T432534#12334970 (10Danya) [16:39:56] 10Tool-wiki-gender-stats: Add a search bar and sorting options in the aggregated regions dialog - https://phabricator.wikimedia.org/T432534#12334990 (10Danya) [16:41:14] 10Tool-wiki-gender-stats: Add a search bar and sorting options in the aggregated regions dialog - https://phabricator.wikimedia.org/T432534#12334993 (10Danya) [16:41:25] 10Tool-wiki-gender-stats: Add a search bar and sorting options in the aggregated regions dialog - https://phabricator.wikimedia.org/T432534#12334994 (10Danya) [16:43:24] RESOLVED: [3x] SystemdUnitFailed: magnum-api.service on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:33:49] (03approved) 10dcaro: support --webservice option [repos/cloud/toolforge/jobs-cli] (load_dedup_and_count) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [17:43:51] 10Toolforge, 06tools-platform-team: [jobs-cli] Potential tech-debt improvements - https://phabricator.wikimedia.org/T438391 (10dcaro) 03NEW [17:54:08] 10Toolforge, 06tools-platform-team: toollabs-images: bullseye image rebuilds fail. decision needed - https://phabricator.wikimedia.org/T438356#12335325 (10dcaro) >>! In T438356#12333991, @Raymond_Ndibe wrote: > @dcaro curious to hear if you prefer a particular approach. > > I prefer `Preferably, for each of... [18:01:20] (03PS1) 10Majavah: Remove bullseye based images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1342772 (https://phabricator.wikimedia.org/T400258) [18:01:31] (03open) 10taavi: Mark remaining bullseye images as deprecated [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/24 (https://phabricator.wikimedia.org/T400258) [18:06:19] FIRING: CloudVPSMagnumCapiClusterDown: Magnum cluster api worker node is not ready - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Magnum - https://grafana.wikimedia.org/d/anbc675/wmcs-openstack-magnum-status - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSMagnumCapiClusterDown [18:09:03] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:11:19] RESOLVED: CloudVPSMagnumCapiClusterDown: Magnum cluster api worker node is not ready - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Magnum - https://grafana.wikimedia.org/d/anbc675/wmcs-openstack-magnum-status - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSMagnumCapiClusterDown [18:14:03] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-70 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:48:57] 10Tool-wikinewsie: Wikinewsie language switcher only shows English - https://phabricator.wikimedia.org/T438399 (10Pharos) 03NEW [18:50:23] 10Tool-wikinewsie: Wikinewsie language switcher only shows English - https://phabricator.wikimedia.org/T438399#12335523 (10Pharos) [19:17:13] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Implement magnum monitoring - https://phabricator.wikimedia.org/T429557#12335627 (10Andrew) 05Open→03Resolved [19:33:10] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/48 (owner: 10l10n-bot) [19:33:15] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/48 (owner: 10l10n-bot) [20:04:32] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [20:04:37] (03update) 10tlepage: refactor: Parse Loki result streams into logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [21:18:41] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [23:44:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:49:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors