[01:15:57] 10Cloud-VPS (Project-requests): Request creation of my first testing on linux VPS project - https://phabricator.wikimedia.org/T383197#10443270 (10bd808) @Gowthamkodali27real: You might try using some service like https://getvm.io/ as a place to explore learning Linux. I have not used it myself, but it was a... [01:44:09] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: consider removing labs-ip-aliaser - https://phabricator.wikimedia.org/T374129#10443304 (10Andrew) Here is an example of not being able to reach a public IP without the name being aliased. enc-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud a... [02:42:05] (03update) 10raymond-ndibe: [jobs-api] convert all quotas to appropriate units [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361120) [03:59:14] FIRING: KernelError: Server cloudcontrol1011 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DKernelError [03:59:14] FIRING: KernelWarning: Server cloudcontrol1011 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DKernelWarning [03:59:29] (03PS2) 10Raymond Ndibe: [wmcs-cookbooks] verify admin and tools tests all passing before reporting passed [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1108454 [04:00:07] (03CR) 10Raymond Ndibe: [wmcs-cookbooks] verify admin and tools tests all passing before reporting passed (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1108454 (owner: 10Raymond Ndibe) [04:03:05] (03update) 10raymond-ndibe: [jobs-api] replicas default to 1 in NewJob model [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T364204) [04:04:54] (03CR) 10Raymond Ndibe: [C:03+2] [wmcs-cookbooks] verify admin and tools tests all passing before reporting passed [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1108454 (owner: 10Raymond Ndibe) [04:08:29] (03Merged) 10jenkins-bot: [wmcs-cookbooks] verify admin and tools tests all passing before reporting passed [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1108454 (owner: 10Raymond Ndibe) [07:59:14] FIRING: KernelError: Server cloudcontrol1011 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DKernelError [07:59:14] FIRING: KernelWarning: Server cloudcontrol1011 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DKernelWarning [08:17:19] 10tool-wscontest: The score command throws deprecated warning - https://phabricator.wikimedia.org/T348270#10443649 (10Samwilson) 05Open→03Resolved a:03Samwilson [08:17:29] 10tool-wscontest: Upgrade to Symfony 7 - https://phabricator.wikimedia.org/T375335#10443651 (10Samwilson) 05Open→03Resolved a:03Samwilson PR (now merged): https://github.com/wikisource/wscontest/pull/69 [08:23:46] 10PAWS, 10Pywikibot, 10Pywikibot-login.py, 07Pywikibot-Wikidata, 07TestMe: Querying wikidata with pywikibot fails for items with images when user is not registered for commons - https://phabricator.wikimedia.org/T168222#10443706 (10Xqt) [08:41:04] 10tool-wscontest: Add health-check-script for scores command runner - https://phabricator.wikimedia.org/T383304 (10Samwilson) 03NEW [08:41:59] 10tool-wscontest: WS Contest has stopped updating its score - https://phabricator.wikimedia.org/T382336#10443730 (10Samwilson) 05Open→03Resolved a:03Samwilson Yes, things are working again now (sorry for my really slow response!). I'm going to add a system of automatically restarting the scoring job if... [09:40:55] 10Tool-lexeme-forms, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Dec), 07Unplanned-Sprint-Work: translatewiki export for Wikidata Lexeme Forms tries to remove sh-latn translations - https://phabricator.wikimedia.org/T379188#10443830 (10Nikerabbit) [09:45:07] 10wikitech.wikimedia.org, 05SUL3: User "Newmcpe", having SUL, can't autocreate account on Wikitech (now SUL wiki) - https://phabricator.wikimedia.org/T383308 (10MBH) 03NEW [10:19:09] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789) [10:19:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:19:15] T309789: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789 [11:12:00] 06cloud-services-team, 10Toolforge, 07Epic: [WIP] Toolforge UI: Investigate integration of Striker functionality - https://phabricator.wikimedia.org/T383146#10444000 (10dcaro) >>! In T383146#10442764, @bd808 wrote: > Are the various "running in k8s" statements in the task description specifically about deplo... [11:30:03] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications: Notification from Wikitech can't be seen in Wikipedia - https://phabricator.wikimedia.org/T383313 (10MBH) 03NEW [11:34:08] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications: Notification from Wikitech can't be seen in Wikipedia - https://phabricator.wikimedia.org/T383313#10444064 (10Bugreporter) Do you see anything in browser console? [11:36:57] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10444076 (10MBH) See T383308 [11:39:31] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications: Notification from Wikitech can't be seen in Wikipedia - https://phabricator.wikimedia.org/T383313#10444080 (10MBH) @Bugreporter I'm using many userscripts, so {F58152826} [11:46:52] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) (T309789) [11:46:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:46:59] T309789: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789 [11:47:08] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications: Notification from Wikitech can't be seen in Wikipedia - https://phabricator.wikimedia.org/T383313#10444089 (10Bugreporter) What you see if you added ?safemode=1&debug=true to URL? [11:50:00] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications: Notification from Wikitech can't be seen in Wikipedia - https://phabricator.wikimedia.org/T383313#10444103 (10MBH) After clicking on bell icon, I redirected to https://ru.wikipedia.org/wiki/Special:Notifications and have the same issue {F58152878} [11:59:14] FIRING: KernelError: Server cloudcontrol1011 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DKernelError [11:59:14] FIRING: KernelWarning: Server cloudcontrol1011 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DKernelWarning [12:19:57] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/28 [12:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:33:43] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10444177 (10cmooney) >>! In T372457#10437119, @dcaro wrote: > I'm the worst xd, `_in_octects` means in traffic, in octets xd, not ju... [12:43:38] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#10444210 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dcaro@cumin... [12:44:30] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/28 (owner: 10l10n-bot) [12:44:32] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/28 (owner: 10l10n-bot) [13:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:17:38] vivian-rook opened https://github.com/toolforge/paws/pull/476 [14:24:37] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#10444599 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dcaro@cumin1002... [14:32:13] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#10444612 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dcaro@cumin... [14:38:02] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#10444619 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dcaro@cumin1002... [14:38:10] 10PAWS: github action update - https://phabricator.wikimedia.org/T383334 (10rook) 03NEW [14:38:39] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:38:47] 10PAWS: github action update - https://phabricator.wikimedia.org/T383334#10444629 (10rook) [14:43:39] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:51:39] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:56:39] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:19:26] vivian-rook closed https://github.com/toolforge/paws/pull/476 [15:43:32] 10Cloud-VPS (Project-requests): Request creation of DefectDojo VPS project - https://phabricator.wikimedia.org/T383344 (10Jly) 03NEW [16:00:13] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of DefectDojo VPS project - https://phabricator.wikimedia.org/T383344#10445044 (10bd808) +1 [16:04:21] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of DefectDojo VPS project - https://phabricator.wikimedia.org/T383344#10445057 (10taavi) Who is "we" here? [16:14:03] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of DefectDojo VPS project - https://phabricator.wikimedia.org/T383344#10445099 (10Slst2020) >>! In T383344#10445057, @taavi wrote: > Who is "we" here? I would guess the Product Security team. [16:24:00] 10wikitech.wikimedia.org, 05SUL3: User "Newmcpe", having SUL, can't autocreate account on Wikitech (now SUL wiki) - https://phabricator.wikimedia.org/T383308#10445143 (10Reedy) 05Open→03Resolved a:03Reedy ` reedy@deploy2002:~$ mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=... [16:24:09] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [openstack object storage] deleted files still occupying space - https://phabricator.wikimedia.org/T376673#10445146 (10dcaro) [16:24:11] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade to v16 - https://phabricator.wikimedia.org/T306820#10445147 (10dcaro) [16:24:25] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#10445150 (10dcaro) [16:24:26] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade to v16 - https://phabricator.wikimedia.org/T306820#10445149 (10dcaro) [16:44:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [16:44:47] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10445237 (10JJMC89) [16:56:30] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10445298 (10dcaro) > I had a stab at making that graph myself (and a few others) if you want to see how it compares. Fwiw you want... [16:56:50] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [17:05:28] FIRING: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:10:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:25:48] 06cloud-services-team, 10Toolforge: [jobs-emailer] duplicate failure emails - https://phabricator.wikimedia.org/T382866#10445408 (10JJMC89) [17:34:07] (03open) 10andrew: New big flavor for wikitextexp [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/146 (https://phabricator.wikimedia.org/T383252) [17:37:00] (03merge) 10andrew: New big flavor for wikitextexp [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/146 (https://phabricator.wikimedia.org/T383252) [17:40:02] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of DefectDojo VPS project - https://phabricator.wikimedia.org/T383344#10445446 (10Jly) >>! In T383344#10445057, @taavi wrote: > Who is "we" here? Yes correct, the Product Security team [17:40:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [17:41:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [17:41:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [17:41:59] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [17:42:53] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team-WIP, 07Essential-Work, 10Parsoid-Read-Views (Phase 1 - DiscussionTools support), 13Patch-For-Review: Bump up quota for wikitextexp to let us spin up a more powerful test server - https://phabricator.wikimedia.org/T383252#10445463 (10Andrew) [17:43:42] (03open) 10andrew: type fix [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/147 (https://phabricator.wikimedia.org/T383252) [17:45:17] (03close) 10andrew: type fix [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/147 (https://phabricator.wikimedia.org/T383252) [17:46:13] (03open) 10andrew: Type fix: wikitextexp, not wikitexexp [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/148 (https://phabricator.wikimedia.org/T383252) [17:46:23] (03merge) 10andrew: Type fix: wikitextexp, not wikitexexp [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/148 (https://phabricator.wikimedia.org/T383252) [17:47:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [17:48:14] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [17:50:47] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned: [openstack] 2025-01-08 nova-api-metadata.service down on cloudcontrol1005 - https://phabricator.wikimedia.org/T383203#10445479 (10Andrew) At least some nova-api-metadata logs are now... [18:05:59] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10445536 (10cmooney) >>! In T372457#10445298, @dcaro wrote: > I'd like to retain the info of which rack to which rack it goes, as th... [18:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:30:22] 10Cloud-VPS (Quota-requests), 06Release-Engineering-Team: New flavor for integration project for larger worker testing - https://phabricator.wikimedia.org/T383357 (10brennen) 03NEW [18:30:25] vivian-rook opened https://github.com/toolforge/paws/pull/477 [18:30:40] 10Cloud-VPS (Quota-requests), 06Release-Engineering-Team: New flavor for integration project for larger worker testing - https://phabricator.wikimedia.org/T383357#10445623 (10brennen) [18:31:54] 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: New flavor for integration project for larger worker testing - https://phabricator.wikimedia.org/T383357#10445629 (10bd808) [18:32:03] 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: New flavor for integration project for larger worker testing - https://phabricator.wikimedia.org/T383357#10445631 (10brennen) (Based format of this request on the previous {T370127}.) [18:33:22] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: New flavor for integration project for larger worker testing - https://phabricator.wikimedia.org/T383357#10445633 (10bd808) +1 [18:35:29] vivian-rook closed https://github.com/toolforge/paws/pull/477 [18:37:26] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of DefectDojo VPS project - https://phabricator.wikimedia.org/T383344#10445643 (10Reedy) I wonder if we can just reuse the `security-tools` project, maybe turning this into a quota increase for that project (though it's got some spare to b... [18:37:47] vivian-rook opened https://github.com/vivian-rook/paws/pull/4 [18:45:36] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications: Notification from Wikitech can't be seen in Wikipedia - https://phabricator.wikimedia.org/T383313#10445653 (10Reedy) →14Duplicate dup:03T376305 [18:45:42] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10445655 (10Reedy) [18:49:37] (03open) 10bd808: Add g4.cores16.ram48.disk20.ephemeral90.4xiops for integration [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/149 (https://phabricator.wikimedia.org/T383357) [18:52:44] RESOLVED: KernelError: Server cloudcontrol1011 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DKernelError [18:52:44] RESOLVED: KernelWarning: Server cloudcontrol1011 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DKernelWarning [18:58:46] vivian-rook opened https://github.com/toolforge/paws/pull/478 [18:59:25] vivian-rook opened https://github.com/vivian-rook/paws/pull/5 [19:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:05:00] vivian-rook opened https://github.com/toolforge/paws/pull/479 [19:05:15] 10PAWS: github action update - https://phabricator.wikimedia.org/T383334#10445721 (10rook) https://github.com/toolforge/paws/pull/479 [19:06:15] (03approved) 10dcaro: Add g4.cores16.ram48.disk20.ephemeral90.4xiops for integration [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/149 (https://phabricator.wikimedia.org/T383357) (owner: 10bd808) [19:08:40] vivian-rook opened https://github.com/toolforge/paws/pull/480 [19:38:38] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10445801 (10cmooney) @Andrew I've updated the switch config for this host to also trunk the //cloud-pirvate-b1-codfw// vlan, so should be ok on that front n... [19:50:35] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10445805 (10Jhancock.wm) it is cabled up and connected to port 43 on the cloud switch [19:57:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-16 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:36:59] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of DefectDojo VPS project - https://phabricator.wikimedia.org/T383344#10445896 (10Andrew) @Reedy, projects are cheap and it's usually easier to have different (openstack) projects for different (actual) projects. I do note that 'security-... [20:38:39] !log andrew@cloudcumin1001 defectdojo START - Cookbook wmcs.vps.create_project for project defectdojo in eqiad1 [20:38:41] andrew@cloudcumin1001: Unknown project "defectdojo" [20:38:58] (03open) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project defectdojo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/150 [20:41:28] (03merge) 10andrew: projects: added project defectdojo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/150 (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [20:42:29] !log andrew@cloudcumin1001 defectdojo END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project defectdojo in eqiad1 [20:42:29] andrew@cloudcumin1001: Unknown project "defectdojo" [20:44:35] !log andrew@cloudcumin1001 defectdojo START - Cookbook wmcs.vps.add_user_to_project for user 'jly' in role 'reader' [20:44:36] andrew@cloudcumin1001: Unknown project "defectdojo" [20:44:38] !log andrew@cloudcumin1001 defectdojo END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'jly' in role 'reader' [20:44:39] andrew@cloudcumin1001: Unknown project "defectdojo" [20:45:45] !log andrew@cloudcumin1001 defectdojo START - Cookbook wmcs.vps.add_user_to_project for user 'jly' in role 'member' [20:45:45] andrew@cloudcumin1001: Unknown project "defectdojo" [20:45:50] !log andrew@cloudcumin1001 defectdojo END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'jly' in role 'member' [20:45:50] andrew@cloudcumin1001: Unknown project "defectdojo" [20:46:40] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of DefectDojo VPS project - https://phabricator.wikimedia.org/T383344#10445935 (10Andrew) 05Open→03Resolved a:03Andrew I have created this project. @Jly you can add other members to the project as needed; I believe that the defau... [20:47:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-16 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:52:57] (03update) 10andrew: Add g4.cores16.ram48.disk20.ephemeral90.4xiops for integration [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/149 (https://phabricator.wikimedia.org/T383357) (owner: 10bd808) [20:54:29] (03merge) 10andrew: Add g4.cores16.ram48.disk20.ephemeral90.4xiops for integration [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/149 (https://phabricator.wikimedia.org/T383357) (owner: 10bd808) [20:54:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [20:55:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [20:56:26] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: New flavor for integration project for larger worker testing - https://phabricator.wikimedia.org/T383357#10445996 (10Andrew) 05Open→03Resolved a:03Andrew ` root@cl... [21:04:42] 10Cloud-VPS (Quota-requests): Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020#10446011 (10rook) +1 [21:07:45] !log andrew@cloudcumin1001 fa-wp START - Cookbook wmcs.openstack.quota_increase (T383020) [21:07:49] T383020: Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020 [21:07:53] !log andrew@cloudcumin1001 fa-wp END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T383020) [21:08:55] 10Cloud-VPS (Quota-requests): Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020#10446020 (10Andrew) 05Open→03Resolved a:03Andrew I agree that this work would best be done in chunks but it's also totally fine for you to use this extra RAM for a while. Please re-open if I made a... [21:09:27] !log andrew@cloudcumin1001 fa-wp START - Cookbook wmcs.openstack.quota_increase (T383020) [21:09:35] !log andrew@cloudcumin1001 fa-wp END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T383020) [21:32:25] 06cloud-services-team, 10Horizon: Horizon: obsessive redirects during logins - https://phabricator.wikimedia.org/T383370 (10Andrew) 03NEW [22:07:45] 10Tool-Pageviews, 10Tool-wikistatistics2-0, 06Data-Engineering, 06Data-Engineering-Icebox, and 3 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#10446184 (10Ottomata) [22:11:23] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#10446236 (10Ottomata) [22:17:34] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138#10446291 (10PixDeVl) >>! In T381138#10369300, @JJMC89 wrote: > FYI, there is nothing useful remaining in the tool's directory - any files to run the tool were removed prior to disabling. Other than the URL, t... [22:22:55] 10Tool-toolwatch, 06Toolforge-standards-committee, 07Privacy: toolwatch loads third party resources - https://phabricator.wikimedia.org/T378901#10446310 (10PixDeVl) >>! In T378901#10289683, @Himacharanbatchu wrote: > Thanks @Taavi for the nice observations you made, I'll fix this :) Was this completed and t... [22:36:30] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138#10446363 (10bd808) >>! In T381138#10446291, @PixDeVl wrote: > Hm, it is possible/worth it/has precedent to set a domain redirect to wherever the new version of the tool is hosted to minimize confusion? Yes.... [22:36:59] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138#10446364 (10bd808) 05Open→03Invalid Tool was archived by its prior maintainer. [22:42:36] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138#10446385 (10PixDeVl) >>! In T381138#10446364, @bd808 wrote: > Tool was archived by its prior maintainer. Forgive me for being unfamiliar with Toolforge currently- does archive entail completely hiding as... [22:48:06] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138#10446388 (10bd808) >>! In T381138#10446385, @PixDeVl wrote: >>>! In T381138#10446364, @bd808 wrote: >> Tool was archived by its prior maintainer. > > Forgive me for being unfamiliar with Toolforge curren... [22:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:05:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:11:16] 06Toolforge-standards-committee: Adoption request for ftools - https://phabricator.wikimedia.org/T381138#10446428 (10JJPMaster) I have now become the maintainer of the tool. It can be found at https://ftools.toolforge.org/.