[00:15:04] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10431215 (10JJMC89) [00:23:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.808% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:31:15] 10Cloud-VPS (Quota-requests): Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020 (10Huji) 03NEW [03:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:23:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.519% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:55:06] PROBLEM - Disk space on cloudbackup1004 is CRITICAL: DISK CRITICAL - free space: /srv 565374MiB (3% inode=99%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1004&var-datasource=eqiad+prometheus/ops [06:13:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.978% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:15:05] RECOVERY - Disk space on cloudbackup1004 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1004&var-datasource=eqiad+prometheus/ops [06:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:29:55] 10Cloud-VPS (Quota-requests): Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020#10431584 (10Slst2020) Hi @Huji Have you tried using streaming XML parsing for processing the wiki dumps? Most programming languages offer ways to process XML files sequentially without loading them entirely... [08:45:10] 06cloud-services-team, 10Toolforge: toolforge webservice logs is broken - https://phabricator.wikimedia.org/T382685#10431621 (10Slst2020) Hi @Don-vip ! A bit late to this because of holidays and such – is this still happening? [09:47:38] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10431763 (10dcaro) I see no logs on the 14th (https://sal.toolforge.org/tools?d=2024-12-14), the whole team was on an offsite during that week, so there were no big inte... [09:52:37] 06cloud-services-team, 10Toolforge: [jobs-emailer] duplicate failure emails - https://phabricator.wikimedia.org/T382866#10431777 (10dcaro) Interesting, there's a "big" change pending to get deployed improving performance, but it's not there yet, so afaik, there have not been any big changes in the code. The c... [09:52:44] 06cloud-services-team, 10Toolforge: [jobs-emailer] duplicate failure emails - https://phabricator.wikimedia.org/T382866#10431778 (10dcaro) p:05Triage→03Medium [09:58:00] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 07Kubernetes: Allow Toolforge roots to use the cookbook to reboot k8s worker nodes (without wmcs-root) - https://phabricator.wikimedia.org/T382977#10431818 (10dcaro) Moving the cookbooks to use the openstack APIs is a long-term goal (since we opened the APIs... [10:57:18] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10432020 (10dcaro) Hi! Late to the task :), just coming back from a long PTO, I had started playing with https://grafana.wikimedia.o... [11:04:00] 06cloud-services-team, 10Toolforge: [infra,k8s,o11y] Introduce worker checks - https://phabricator.wikimedia.org/T380985#10432025 (10dcaro) >>! In T380985#10364874, @aborrero wrote: > I believe this is duplicate of {T380892} It's a bit wider, as the idea is to also test NFS and other things, not just calico/n... [11:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:19:40] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/27 [12:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:41:57] 06cloud-services-team, 10Toolforge: toolforge webservice logs is broken - https://phabricator.wikimedia.org/T382685#10432394 (10Don-vip) Hi @Slst2020, no problem (and happy new year!) I don't face the issue anymore. Feel free to close this ticket :) [13:45:59] 06cloud-services-team, 10Toolforge: toolforge webservice logs is broken - https://phabricator.wikimedia.org/T382685#10432412 (10Slst2020) 05Open→03Resolved a:03Slst2020 Happy new year! ok, closing. Feel free to reopen if you run into this again. [13:50:23] (03update) 10raymond-ndibe: [toolforge-deploy] add more test cases to job loads [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/646 (https://phabricator.wikimedia.org/T364204) [14:37:12] (03update) 10raymond-ndibe: [toolforge-deploy] refetch default to main branch [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/645 (https://phabricator.wikimedia.org/T358225) [14:51:33] 06cloud-services-team, 10Toolforge, 07Documentation: Update maintain-harbor documentation after move from a tool to a component - https://phabricator.wikimedia.org/T381650#10432732 (10Raymond_Ndibe) a:03Raymond_Ndibe [15:17:55] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/37 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:18:00] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/37 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:18:11] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/36 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:19:53] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/36 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:19:56] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/36 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:20:27] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/218 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:20:35] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/218 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:21:19] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/73 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:21:23] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/73 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:21:31] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/72 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:23:51] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/72 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:25:07] (03open) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor image retention tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/648 [16:01:50] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10432942 (10JJMC89) Yes, its still happening. [16:11:53] 10Cloud-Services, 06Moderator-Tools-Team: The Wikipedia Library emails aren't being received by @wikimedia.org email inboxes - https://phabricator.wikimedia.org/T382314#10433002 (10Samwalton9-WMF) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wi... [16:12:09] 06cloud-services-team, 10Cloud-VPS, 06Moderator-Tools-Team: The Wikipedia Library emails aren't being received by @wikimedia.org email inboxes - https://phabricator.wikimedia.org/T382314#10433007 (10Samwalton9-WMF) [16:12:26] 06cloud-services-team, 10Cloud-VPS, 10Library-Card-Platform, 06Moderator-Tools-Team: The Wikipedia Library emails aren't being received by @wikimedia.org email inboxes - https://phabricator.wikimedia.org/T382314#10433010 (10Samwalton9-WMF) [16:19:43] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [16:22:43] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [16:23:17] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [16:28:50] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [16:34:05] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for ou... - https://phabricator.wikimedia.org/T374830#10433117 [16:35:27] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [16:37:05] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [16:46:39] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [16:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:54:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [16:57:46] 06cloud-services-team, 10Cloud-VPS, 10Library-Card-Platform, 06Moderator-Tools-Team: The Wikipedia Library emails aren't being received by @wikimedia.org email inboxes - https://phabricator.wikimedia.org/T382314#10433215 (10jsn.sherman) Suggest that we do a proof of concept on staging to send emails via wi... [16:57:59] (03PS1) 10Raymond Ndibe: [wmcs-cookbooks] verify admin and tools tests all passing before reporting passed [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1108454 [16:58:31] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10433222 (10dcaro) p:05Triage→03Medium [16:59:05] 10Tool-python-toolforge: Support reading Wiki Replica/ToolsDB credentials from envvars - https://phabricator.wikimedia.org/T339940#10433225 (10Sascha) Curious, what would it take to make this happen? [17:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:04:51] 06cloud-services-team, 10Toolforge: toolforge webservice logs is broken - https://phabricator.wikimedia.org/T382685#10433254 (10Wargo) But the log viewer should be prepared for this. [17:05:37] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor image retention tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/648 [17:10:25] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/98 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:10:28] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/98 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:10:31] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/97 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:10:42] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/97 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:10:52] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:10:55] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:11:06] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:11:10] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:11:18] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/79 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:11:22] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/79 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:12:11] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/97 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:12:19] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/69 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:12:23] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/69 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:12:45] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:24:40] 10Tool-python-toolforge: Support reading Wiki Replica/ToolsDB credentials from envvars - https://phabricator.wikimedia.org/T339940#10433368 (10bd808) >>! In T339940#10433225, @Sascha wrote: > Curious, what would it take to make this happen? The current connection logic is in https://gitlab.wikimedia.org/toolfor... [17:27:19] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10433406 (10JJMC89) [17:32:54] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10433433 (10Jhancock.wm) @Andrew checking back on this one. anything i can help with? [17:41:55] 06cloud-services-team, 10Toolforge: [jobs-emailer] duplicate failure emails - https://phabricator.wikimedia.org/T382866#10433464 (10JJMC89) [17:53:22] (03update) 10raymond-ndibe: [jobs-api] convert all quotas to appropriate units [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361120) [17:55:16] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10433513 (10dcaro) > running these tasks should not require any work from Data-Persistence I guess this restriction was... [17:55:53] (03update) 10raymond-ndibe: [jobs-api] convert all quotas to appropriate units [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361120) [18:05:44] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10433555 (10Marostegui) >>! In T382607#10433513, @dcaro wrote: >> running these tasks should not require any work from D... [18:22:39] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10433585 (10dcaro) > Those cookbooks are for the views, which is something we've never touched, designed or maintained.... [19:25:03] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10433836 (10bd808) 05Resolved→03Open Not sure why @andrew closed this as resolved. `lang=shell-session bd808@deployment-m... [19:28:51] 10Toolforge (Toolforge iteration 16): Persist maintain-harbor logs - https://phabricator.wikimedia.org/T383081 (10Raymond_Ndibe) 03NEW [19:38:07] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10433901 (10bd808) `counterexample $ sudo -i puppet agent -tv ... Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Option... [19:38:52] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10433905 (10bd808) p:05Triage→03High [19:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:56:06] 10Toolforge (Toolforge iteration 16): Persist maintain-harbor logs - https://phabricator.wikimedia.org/T383081#10433933 (10Raymond_Ndibe) update https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Harbor/maintain-harbor when this is done [20:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:02:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 07Documentation: Update maintain-harbor documentation after move from a tool to a component - https://phabricator.wikimedia.org/T381650#10433945 (10Raymond_Ndibe) [20:02:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 07Documentation: Update maintain-harbor documentation after move from a tool to a component - https://phabricator.wikimedia.org/T381650#10433949 (10Raymond_Ndibe) 05Open→03Resolved [20:03:12] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-cli,jobs-api] quota shows different units for limit and usage - https://phabricator.wikimedia.org/T361120#10433951 (10Raymond_Ndibe) 05In progress→03Resolved [20:03:27] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-cli,jobs-api] quota shows different units for limit and usage - https://phabricator.wikimedia.org/T361120#10433954 (10Raymond_Ndibe) 05Resolved→03In progress [20:19:15] (03open) 10raymond-ndibe: [maintain-harbor] get_example_config() return content of .env file [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/41 [20:23:31] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/27 (owner: 10l10n-bot) [20:23:36] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/27 (owner: 10l10n-bot) [20:36:44] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform, 13Patch-For-Review: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#10434115 (10VirginiaPoundstone) [20:54:46] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10434225 (10bd808) [21:21:44] 10Tool-Pageviews, 10Tool-wikistatistics2-0, 06Data-Engineering, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#10434582 (10VirginiaPoundstone) [21:30:39] 06cloud-services-team, 10Data-Services, 06Data-Engineering-Icebox, 13Patch-For-Review: Log_param is redacted in wiki replica when only comment and/or user should be - https://phabricator.wikimedia.org/T301943#10434761 (10VirginiaPoundstone) [21:32:59] 06cloud-services-team, 10Data-Services, 06Data-Engineering-Icebox, 06Privacy Engineering: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944#10434806 (10VirginiaPoundstone) [21:37:25] 10Tool-Pageviews, 10Tool-wikistatistics2-0, 06Data-Engineering, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#10435091 (10VirginiaPoundstone) [21:41:13] 10Tool-Pageviews, 06Data-Engineering, 10Pageviews-API: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#10435194 (10VirginiaPoundstone) [21:44:02] 10Tool-Pageviews, 10Tool-wikistatistics2-0, 06Data-Engineering, 10Data-Engineering-Wikistats, and 3 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#10435276 (10VirginiaPoundstone) [21:44:36] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10435300 (10bd808) [21:45:32] 10Tool-Pageviews, 10Tool-wikistatistics2-0, 06Data-Engineering, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#10435341 (10VirginiaPoundstone) [21:45:55] 10Tool-Pageviews, 10Tool-wikistatistics2-0, 06Data-Engineering, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#10435347 (10Ottomata) [21:50:09] 10Tool-Pageviews, 06Data-Engineering, 10Pageviews-API: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#10435452 (10Ottomata) [22:55:32] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/40 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:55:36] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/40 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:55:56] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/57 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:58:53] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/56 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:58:57] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/56 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:59:10] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/134 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:59:17] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/134 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:59:18] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: api-gateway: bump to 0.0.59-20250106225612-c6f0834c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/650 [22:59:20] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/134 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:59:31] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/133 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:59:40] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/14 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:59:47] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/14 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [22:59:58] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:00:02] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:00:10] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:00:29] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:00:39] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:00:40] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:00:51] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/48 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:00:58] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/48 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:01:08] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/48 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:01:17] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: api-gateway: bump to 0.0.59-20250106225612-c6f0834c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/650 [23:01:47] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-emailer: bump to 0.0.47-20250106225958-1a5bfe51 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/651 [23:01:49] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/133 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:01:52] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/133 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:01:53] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/133 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:07] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:09] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/37 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:13] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/40 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:17] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/218 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:22] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/36 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:28] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.343-20250106225928-1a371b63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/652 [23:02:29] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/57 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:35] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/56 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:39] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/48 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:43] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/73 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:02:50] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/72 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:03:14] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.74-20250106230052-21087bd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/653 [23:04:26] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.343-20250106225928-1a371b63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/652 [23:05:00] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-emailer: bump to 0.0.47-20250106225958-1a5bfe51 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/651 [23:05:18] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [23:05:48] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.74-20250106230052-21087bd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/653 [23:12:33] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [23:13:46] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [23:21:45] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [23:22:50] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.343-20250106225928-1a371b63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/652 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:22:52] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.343-20250106225928-1a371b63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/652 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:22:56] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.343-20250106225928-1a371b63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/652 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:23:28] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [23:30:00] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [23:31:05] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [23:31:09] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [23:36:55] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [23:38:19] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [23:38:44] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [23:39:14] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.64-20241220145144-0b848d21 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/647 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:39:17] (03approved) 10raymond-ndibe: envvars-api: bump to 0.0.64-20241220145144-0b848d21 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/647 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:39:28] (03merge) 10raymond-ndibe: envvars-api: bump to 0.0.64-20241220145144-0b848d21 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/647 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:41:23] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [23:45:23] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [23:46:02] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [23:46:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor [23:48:37] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [23:48:56] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [23:49:09] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [23:53:40] 10Cloud-VPS (Quota-requests): Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020#10435808 (10Huji) I think that may address part of my issue and I will look into it. However, articles are indeed the majority of pages (and the largest volume) from that dump, so when it comes to the subseq... [23:55:25] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [23:55:48] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [23:56:03] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [23:56:53] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [23:57:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [23:57:27] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.59-20250106225612-c6f0834c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/650 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:32] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.59-20250106225612-c6f0834c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/650 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:57:52] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.59-20250106225612-c6f0834c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/650 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:27] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.47-20250106225958-1a5bfe51 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/651 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:31] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.47-20250106225958-1a5bfe51 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/651 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [23:58:47] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.47-20250106225958-1a5bfe51 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/651 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)