[00:30:41] 10Cloud-VPS (Project-requests): Request creation of WikiPeopleStats VPS project - https://phabricator.wikimedia.org/T378042 (10Danielyepezgarces) 03NEW [01:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:58:37] 10Tool-ldap: "Service group" is incomplete - https://phabricator.wikimedia.org/T378049 (10Bugreporter) 03NEW [04:49:24] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10257609 (10Harej) I have two servers with 384 GB of RAM each, plus a workstation with 256 GB of RAM that can be used for testing... [05:07:39] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10257614 (10Seppl2013) @Harej thank you for pointing out your infrastructure availabilty which is sort of already "priced" in in... [05:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:39:04] 06cloud-services-team, 10Toolforge, 07User-notice: Copyvios tool: investigate/block suspicious web traffic - https://phabricator.wikimedia.org/T285450#10257667 (10Pppery) Can this be closed as resolved now? [05:49:04] 06cloud-services-team, 10Toolforge, 07User-notice: Copyvios tool: investigate/block suspicious web traffic - https://phabricator.wikimedia.org/T285450#10257678 (10Chlod) 05Open→03Resolved a:03Chlod Marking as resolved since the underlying issue (bot scraping of the tool) has been dealt with. As for... [06:59:53] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for nrwiki - https://phabricator.wikimedia.org/T375101#10257759 (10ABran-WMF) All done, ready for the views creation. [07:01:19] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for rskwiki - https://phabricator.wikimedia.org/T375016#10257765 (10ABran-WMF) All done, ready for the views creation., also for tddwiki [07:01:40] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for bclwikisource - https://phabricator.wikimedia.org/T377087#10257764 (10ABran-WMF) All done, ready for the views creation. [07:02:55] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for ibawiki - https://phabricator.wikimedia.org/T376571#10257758 (10ABran-WMF) All done, ready for the views creation. [07:04:12] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for nrwiki - https://phabricator.wikimedia.org/T375101#10257768 (10ABran-WMF) a:05ABran-WMF→03None [07:33:46] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for rskwiki - https://phabricator.wikimedia.org/T375016#10257795 (10ABran-WMF) a:05ABran-WMF→03None [07:33:49] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for annwiki - https://phabricator.wikimedia.org/T377118#10257800 (10ABran-WMF) All done, ready for the views creation. [07:51:27] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10257853 (10aborrero) >>! In T377467#10256868, @Dzahn wrote: > > Would it be possible to (centrally / scripted) cre... [08:05:19] 10Cloud-VPS (Project-requests): Request creation of WikiPeopleStats VPS project - https://phabricator.wikimedia.org/T378042#10257876 (10aborrero) p:05Triage→03Medium [08:07:44] 10Cloud-VPS (Project-requests): Request creation of WikiPeopleStats VPS project - https://phabricator.wikimedia.org/T378042#10257874 (10aborrero) a:03Raymond_Ndibe LGTM. +1 [09:10:03] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10258003 (10cmooney) >>! In T377467#10256868, @Dzahn wrote: > Would it be possible to (centrally / scripted) create... [09:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:23:48] (03PS1) 10EoghanGaffney: apt-staging: Add empty secret for staging_secring.gpg [labs/private] - 10https://gerrit.wikimedia.org/r/1082719 [09:31:10] (03CR) 10EoghanGaffney: [V:03+2 C:03+2] apt-staging: Add empty secret for staging_secring.gpg [labs/private] - 10https://gerrit.wikimedia.org/r/1082719 (owner: 10EoghanGaffney) [10:47:41] 10VPS-project-Wikistats: Add YsArchives to Mediawikis - https://phabricator.wikimedia.org/T378074 (10Zorua_Fox) 03NEW [11:02:16] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10258244 (10aborrero) 05Open→03In progress p:05Low→03Medium In codfw1dev, I will do the following to test the upstream extension: * in cloudservices no... [11:35:43] 10VPS-project-Wikistats: Add YsArchives to Mediawikis - https://phabricator.wikimedia.org/T378074#10258301 (10Dzahn) a:03Dzahn [11:39:35] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10258312 (10aborrero) in cloudcontrol nodes: `name=/etc/designate/designate.conf #[service:sink] # List of notification handlers to enable, configuration of t... [12:15:50] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10258420 (10aborrero) also, in cloudcontrol nodes: `name=/etc/neutron/neutron.conf,lang=ini [DEFAULT] external_dns_driver = designate # ... more stuff [desi... [12:17:55] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10258428 (10aborrero) after the file changes above, I see this: `lang=shell-session aborrero@cloudcontrol2006-dev:~$ sudo wmcs-openstack extension list --netw... [12:26:06] (03open) 10aborrero: codfw1dev: use neutron extension to publish DNS records for ports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/101 (https://phabricator.wikimedia.org/T377740) [12:27:16] (03update) 10aborrero: codfw1dev: use neutron extension to publish DNS records for ports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/101 (https://phabricator.wikimedia.org/T377740) [12:28:39] (03update) 10aborrero: codfw1dev: use neutron extension to publish DNS records for ports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/101 (https://phabricator.wikimedia.org/T377740) [12:39:41] (03update) 10aborrero: codfw1dev: use neutron extension to publish DNS records for ports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/101 (https://phabricator.wikimedia.org/T377740) [12:48:06] (03approved) 10fnegri: codfw1dev: use neutron extension to publish DNS records for ports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/101 (https://phabricator.wikimedia.org/T377740) (owner: 10aborrero) [12:50:56] (03merge) 10aborrero: codfw1dev: use neutron extension to publish DNS records for ports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/101 (https://phabricator.wikimedia.org/T377740) [12:51:08] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:51:38] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:23:31] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10258671 (10bking) >>! In T377655#10250598, @aborrero wrote: >>>! In T377655#10250552, @bking wrote: >> @aborrero sorry for the c... [13:51:14] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10258763 (10hashar) 05O... [13:53:42] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [14:09:20] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10258837 (10Jhancock.wm) is this ticket still needed? [14:11:38] 10Tools: zoomviewer delete Github repo - https://phabricator.wikimedia.org/T378088#10258842 (10Aklapper) @Kristbaum: Could you elaborate please how it is outdated? [14:15:22] 10Tools: zoomviewer delete Github repo - https://phabricator.wikimedia.org/T378088#10258863 (10Reedy) Removed the link to GitHub, gitlab is linked on the right [14:15:28] 10Tools: zoomviewer delete Github repo - https://phabricator.wikimedia.org/T378088#10258864 (10Kristbaum) This is this old Github Page, that is prominently linked on the toolpage, but is outdated for over a year: https://github.com/toollabs/zoomviewer This is the new Wikimedia Gitlab page, which for example has... [14:16:20] 10Tools: zoomviewer delete Github repo - https://phabricator.wikimedia.org/T378088#10258867 (10Kristbaum) 05Open→03Resolved [14:16:30] 10Tools: zoomviewer delete Github repo - https://phabricator.wikimedia.org/T378088#10258868 (10Reedy) >>! In T378088#10258863, @Reedy wrote: > Removed the link to GitHub, gitlab is linked on the right https://toolsadmin.wikimedia.org/tools/id/zoomviewer/info/id/2141/history?version_id2=4239&version_id1=3374 [14:24:21] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [14:24:52] (03update) 10raymond-ndibe: [lima-kilo] minor project refactor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [14:25:51] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [14:26:54] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [14:35:35] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [14:50:32] 06cloud-services-team, 10Data-Services, 06Data-Engineering-Icebox: Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users - https://phabricator.wikimedia.org/T204950#10259085 (10taavi) 05Open→03Declined That seems unnecessarily complicated when I just wan... [14:53:22] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10259114 (10aborrero) on the first test, it seems neutron expects a literal zone `.eqiad1.wikimedia.cloud` to exist in order to create the A/AAAA... [15:03:12] (03update) 10aborrero: codfw1dev: add dedicated VM zones for admin-monitoring and cloudinfra-codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/102 (https://phabricator.wikimedia.org/T377740) [15:03:17] (03open) 10aborrero: codfw1dev: add dedicated VM zones for admin-monitoring and cloudinfra-codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/102 (https://phabricator.wikimedia.org/T377740) [15:08:30] (03approved) 10aborrero: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 (owner: 10raymond-ndibe) [15:15:47] 10Tool-video-answer-tool, 06Future-Audiences: Create video for arbitrary topic - https://phabricator.wikimedia.org/T376660#10259242 (10Maryana) 05Open→03Resolved [15:23:32] (03merge) 10aborrero: codfw1dev: add dedicated VM zones for admin-monitoring and cloudinfra-codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/102 (https://phabricator.wikimedia.org/T377740) [15:23:40] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:24:08] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:31:01] (03open) 10rook: Update .gitlab-ci.yml file [repos/cloud/paws] - 10https://gitlab.wikimedia.org/repos/cloud/paws/-/merge_requests/1 [15:32:42] (03open) 10aborrero: codfw1dev: admin-monitoring: delete DNS zone admin-monitoring.codfw1dev.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/103 [15:33:25] (03merge) 10aborrero: codfw1dev: admin-monitoring: delete DNS zone admin-monitoring.codfw1dev.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/103 [15:36:46] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10259410 (10Seppl2013) Do you know a tool that will track CPU and Mem load of a server over time (on an Ubuntu server)? Currently... [15:41:21] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10259485 (10aborrero) got a traceback: ` 2024-10-24 15:34:54.073 1987217 ERROR neutron.plugins.ml2.extensions.dns_integration [req-f7516c6a-f20c-48a4-b570-1... [15:59:44] 10Tools: [Zoomviewer] Update iipmooviewer 1.1 to 2.0 - https://phabricator.wikimedia.org/T378099#10259549 (10Aklapper) [17:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:59:04] (03update) 10rook: Update .gitlab-ci.yml file [repos/cloud/paws] - 10https://gitlab.wikimedia.org/repos/cloud/paws/-/merge_requests/1 [18:01:19] (03update) 10rook: Update .gitlab-ci.yml file [repos/cloud/paws] - 10https://gitlab.wikimedia.org/repos/cloud/paws/-/merge_requests/1 [19:00:36] (03update) 10rook: Update .gitlab-ci.yml file [repos/cloud/paws] - 10https://gitlab.wikimedia.org/repos/cloud/paws/-/merge_requests/1 [21:05:49] 10VPS-project-Wikistats: Add YsArchives to Mediawikis - https://phabricator.wikimedia.org/T378074#10260787 (10Dzahn) 05Open→03Resolved Added! What I'm actually adding is the URL to api.php, so ` MariaDB [wikistats]> insert into mediawikis (statsurl,method) values ("https://youshou.wiki/api.php",8); `... [21:21:30] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:47:37] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire