[00:04:14] 10Grid-Engine-to-K8s-Migration, 10Event Metrics, 10Community-Tech (CommTech-Kanban): Migrate grantmetrics from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319779 (10TheresNoTime) Moving to Kanban for attention — one job still running: https://grid-deprecation.toolforge.o... [00:08:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:32:29] 10Grid-Engine-to-K8s-Migration, 10Community-Tech (CommTech-Kanban): Migrate commtech-commons from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319642 (10MusikAnimal) [00:32:41] 10Grid-Engine-to-K8s-Migration, 10Community-Tech (CommTech-Kanban): Migrate commtech-commons from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319642 (10MusikAnimal) 05Stalled→03Open [00:34:49] 10Grid-Engine-to-K8s-Migration, 10Commons Deletion Notification bot, 10Community-Tech (CommTech-Kanban): Migrate commtech-commons from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319642 (10MusikAnimal) a:03MusikAnimal [02:06:56] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [tools,harbor] Cleanup old production images - https://phabricator.wikimedia.org/T348538 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/20 [maintain-harbor]: cleanup old produc... [02:10:09] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [tools,harbor] Cleanup old production images - https://phabricator.wikimedia.org/T348538 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/20 [maintain-harbor]: cleanup old produc... [02:10:56] 10Grid-Engine-to-K8s-Migration: Migrate commons-delinquent from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319640 (10mdaniels5757) This should be all set now, but leaving this open for verification. [02:23:06] 10Toolforge (Software install/update), 10User-bd808: mysqldump is not present in Kubernetes container images - https://phabricator.wikimedia.org/T254636 (10MusikAnimal) Hi! I tried using this but ran into some problems (T319779#9278315): ` tools.grantmetrics@tools-sgebastion-10:~$ toolforge-jobs run backup --... [02:25:49] 10Grid-Engine-to-K8s-Migration: Migrate musikbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319914 (10MusikAnimal) >>! In T319914#9350868, @komla wrote: > @MusikAnimal have you taken another look at this T254636 was resolved? I have over at T319779#9278315 (it's the s... [02:38:49] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [tools,harbor] Cleanup old production images - https://phabricator.wikimedia.org/T348538 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/21 [maintain-harbor]: cleanup old produc... [02:39:40] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [tools,harbor] Cleanup old production images - https://phabricator.wikimedia.org/T348538 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/21 [maintain-harbor]: cleanup old produc... [02:44:44] 10Toolforge (Toolforge iteration 02): [tools,harbor] Cleanup old production images - https://phabricator.wikimedia.org/T348538 (10Raymond_Ndibe) [02:44:54] 10Toolforge (Toolforge iteration 02): [tools,harbor] Cleanup old production images - https://phabricator.wikimedia.org/T348538 (10Raymond_Ndibe) 05In progress→03Resolved [02:58:18] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10tstarling) [02:58:55] 10Toolforge (Software install/update): Please install hugin-tools and pillow again - https://phabricator.wikimedia.org/T347446 (10tstarling) 05Open→03Declined Thanks @bd808. I'll close this as declined. This task is requesting hugin in the base image, and that's not going to happen. Making a buildpack can be... [03:13:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:18:18] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10tstarling) Tool synopsis ([[https://github.com/toollabs/panoviewer|source]]): * The tool has a static HTML/JS frontend which embeds Panellum, an upstream project wh... [05:17:10] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10KartikMistry) @Slst2020 Any updates on this? Let me know if more information is required. [05:22:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:27:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:28:13] 10Tool-extjsonuploader: extjsonuploader should differentiate between master and last release - https://phabricator.wikimedia.org/T325487 (10Samwilson) I'd like to add `vcs-version` to the extjsonuploader data, so we can at least see exactly which commit is being talked about. That's the same key that the siteinf... [06:04:08] 10Tool-extjsonuploader: extjsonuploader should differentiate between master and last release - https://phabricator.wikimedia.org/T325487 (10Tgr) The git commit probably won't be that useful for extensions which use Translatewiki. There is certainly no harm in exposing it, though. [06:11:39] 10Tool-extjsonuploader, 10GitLab: Support gitlab.wikimedia.org in extjsonuploader - https://phabricator.wikimedia.org/T352831 (10Tgr) [06:13:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:26:06] 10Toolforge Jobs framework: `toolforge jobs` command is slow - https://phabricator.wikimedia.org/T352832 (10Legoktm) [06:59:42] 10Tool-extjsonuploader: extjsonuploader should differentiate between master and last release - https://phabricator.wikimedia.org/T325487 (10Samwilson) Good point. You mean, because the most recent commit will quite often be one from TranslateWiki and not a meaningful code update? I guess it's a similar case with... [08:10:59] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10Slst2020) >>! In T352136#9385549, @KartikMistry wrote: > @Slst2020 Any updates on this? Let me know if more information is required. Oh no – I... [08:30:22] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10KartikMistry) >>! In T352136#9385682, @Slst2020 wrote: >>>! In T352136#9385549, @KartikMistry wrote: >> @Slst2020 Any updates on this? Let me k... [08:33:16] 10Grid-Engine-to-K8s-Migration: Migrate joinedventure from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319832 (10Steinsplitter) 05Open→03Resolved a:05Rillke→03Steinsplitter [08:43:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:48:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:48:33] !log sstefanova@cloudcumin1001 language START - Cookbook wmcs.openstack.quota_increase [08:48:36] !log sstefanova@cloudcumin1001 language END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) [08:59:41] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10Slst2020) [09:05:31] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10dcaro) LGFM +1 Remember to remove the old instance once you don't need it :) [09:07:16] !log sstefanova@cloudcumin1001 language START - Cookbook wmcs.openstack.quota_increase [09:07:20] !log sstefanova@cloudcumin1001 language END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) [09:13:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:22:45] (03PS1) 10Muehlenhoff: Revert "Remove ganeti RAPI dummy certs" [labs/private] - 10https://gerrit.wikimedia.org/r/980816 [09:25:20] (03CR) 10Muehlenhoff: [V: 03+2 C: 03+2] Revert "Remove ganeti RAPI dummy certs" [labs/private] - 10https://gerrit.wikimedia.org/r/980816 (owner: 10Muehlenhoff) [09:33:28] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10Slst2020) 05Open→03Resolved a:03Slst2020 Done: ` sstefanova@cloudcontrol1005:~$ sudo wmcs-openstack quota show language | grep ram | ram... [09:36:58] 10Cloud-VPS: [wmcs-cookbook] increase_quota cookbook fails - https://phabricator.wikimedia.org/T352840 (10Slst2020) [09:38:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:39:41] 10Cloud-VPS: [wmcs-cookbook] increase_quota cookbook fails - https://phabricator.wikimedia.org/T352840 (10Slst2020) [09:53:59] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [tbs, builds-api] change local environment to use admin account - https://phabricator.wikimedia.org/T352770 (10CodeReviewBot) sstefanova opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/143 dev: use admin Harb... [09:54:51] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [tbs, builds-api] change local environment to use admin account - https://phabricator.wikimedia.org/T352770 (10Slst2020) 05Open→03In progress a:03Slst2020 [10:22:19] 10Toolforge (Toolforge iteration 02): [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10dcaro) a:03dcaro [10:22:25] 10Toolforge (Software install/update): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10dcaro) [10:22:47] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10dcaro) [10:23:17] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10dcaro) 05Open→03In progress [10:31:27] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [tbs, builds-api] change local environment to use admin account - https://phabricator.wikimedia.org/T352770 (10CodeReviewBot) sstefanova merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/143 dev: use admin Harb... [10:35:01] (03PS1) 10Muehlenhoff: Revert "Revert "Remove ganeti RAPI dummy certs"" [labs/private] - 10https://gerrit.wikimedia.org/r/980825 [10:37:38] (03CR) 10Muehlenhoff: [V: 03+2 C: 03+2] Revert "Revert "Remove ganeti RAPI dummy certs"" [labs/private] - 10https://gerrit.wikimedia.org/r/980825 (owner: 10Muehlenhoff) [10:41:00] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [tbs, builds-api] change local environment to use admin account - https://phabricator.wikimedia.org/T352770 (10Slst2020) 05In progress→03Resolved [10:46:20] 10Cloud-VPS (Quota-requests): Please delete meet and chat VPS projects - https://phabricator.wikimedia.org/T352727 (10Slst2020) a:03Slst2020 [10:46:22] 10Toolforge (Toolforge iteration 02): [tbs] cleanup robot account related code - https://phabricator.wikimedia.org/T352763 (10Slst2020) Or should we try to replicate what we do in prod, i.e. keep using the robot account for the builds-builder (the tekton pipeline), and create separate users for the other compone... [11:10:33] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10dcaro) Looking into changing the builder image depending on the code language found some things... [11:28:33] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2), 10Goal, 10Patch-For-Review: Support 'unmanaged' projects in cloud-vps - https://phabricator.wikimedia.org/T326818 (10dcaro) >>! In T326818#9383803, @Andrew wrote: > My imagined support policy would be: > > * We'll help you with OpenStack issues (e.g.... [11:41:51] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Increase quota to create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10KartikMistry) [11:45:35] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Increase quota to create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10KartikMistry) [12:00:40] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10dcaro) For the option of injecting the buildpack as we do with others, we would need to add it t... [12:02:52] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10dcaro) I think I'll start working on the latter, specially given that we want to re-think the wa... [12:09:29] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10Slst2020) >>! In T352774#9386216, @dcaro wrote: > For the option of injecting the buildpack as w... [12:11:43] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10dcaro) >>! In T352774#9386250, @Slst2020 wrote: >>>! In T352774#9386216, @dcaro wrote: >> For th... [12:24:30] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10dcaro) I think that we can try the following: * Before the detect step ** create a combined orde... [12:35:03] 10Tools, 10WMDE-TechWish-Maintenance, 10WMDE-TechWish-Sprint-2023-11-22, 10WMDE-TechWish-Sprint-2023-12-06: Check technischewuensche tool code and publish in a public repo - https://phabricator.wikimedia.org/T350352 (10thiemowmde) [12:37:45] (ProbeDown) firing: Service tools-k8s-haproxy-4:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:42:45] (ProbeDown) resolved: Service tools-k8s-haproxy-4:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:54:40] 10Toolforge (Toolforge iteration 02): Decide what abstractions we want to expose to Toolforge users in the longer term - https://phabricator.wikimedia.org/T352857 (10Slst2020) [13:11:54] 10PAWS: Upgrade openrefine to 3.7.7 - https://phabricator.wikimedia.org/T352865 (10rook) [13:21:28] 10Striker, 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth, 10TestMe: Wikitech 2FA does not appear to allow recovery with scratch codes - https://phabricator.wikimedia.org/T204682 (10Reedy) [13:54:14] 10Data-Services, 10cloud-services-team, 10Data-Engineering, 10Epic: Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema - https://phabricator.wikimedia.org/T215858 (10Gehel) Removing DPE SRE from this task until it is picked up by #data-engine... [14:28:37] 10PAWS: New upstream release 8.6 for Pywikibot - https://phabricator.wikimedia.org/T352794 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/356 [14:28:43] vivian-rook closed https://github.com/toolforge/paws/pull/356 [14:29:08] 10PAWS: New upstream release 8.6 for Pywikibot - https://phabricator.wikimedia.org/T352794 (10rook) 05Open→03Resolved a:03rook [14:33:38] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417 (10CodeReviewBot) sstefanova opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 jobs: add job for managin... [14:47:49] 10Toolforge (Toolforge iteration 02): Decide what abstractions we want to expose to Toolforge users in the longer term - https://phabricator.wikimedia.org/T352857 (10dcaro) Thanks for starting this! Adding a bit more detail, the top level abstractions are "apps" and "pipelines". Inside an "app" there's one sin... [14:53:16] 10Toolforge Jobs framework: `toolforge jobs` command is slow - https://phabricator.wikimedia.org/T352832 (10taavi) a:03taavi [15:12:09] 10Toolforge (Toolforge iteration 02): Decide what abstractions we want to expose to Toolforge users in the longer term - https://phabricator.wikimedia.org/T352857 (10Slst2020) That's nicely detailed! Something else I'd like as a user is when I run a maintenance task like a db migration, I don't want to be expos... [15:13:14] 10Toolforge Jobs framework: toolforge jobs restart sometimes times out - https://phabricator.wikimedia.org/T352874 (10taavi) [15:13:55] 10Toolforge Jobs framework: `toolforge jobs` command is slow - https://phabricator.wikimedia.org/T352832 (10taavi) 05Open→03Resolved It's much more reasonable now: ` tools.taavi-test-tool@tools-sgebastion-11:~$ time toolforge jobs list real 0m1.042s user 0m0.532s sys 0m0.062s ` Filed {T352874} for the resta... [15:22:14] 10Toolforge (Toolforge iteration 02): Decide what abstractions we want to expose to Toolforge users in the longer term - https://phabricator.wikimedia.org/T352857 (10dcaro) Oh, yes, heroku has that special "migrate" procfile entry, that it runs on every "deployment", that is done every time you push to your git... [15:26:01] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review, 10User-dcaro: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools - https://phabricator.wikimedia.org/T352774 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolf... [15:27:50] 10Toolforge (Toolforge iteration 02): Decide what abstractions we want to expose to Toolforge users in the longer term - https://phabricator.wikimedia.org/T352857 (10Slst2020) Ah yes. I was wondering if that runs as some sort of init container, but no, they say it's a one-off dyno so it's basically just a regula... [15:28:51] 10Toolforge (Toolforge iteration 02): Decide what abstractions we want to expose to Toolforge users in the longer term - https://phabricator.wikimedia.org/T352857 (10dcaro) >>! In T352857#9387027, @Slst2020 wrote: > Ah yes. I was wondering if that runs as some sort of init container, but no, they say it's a one-... [15:36:19] 10Toolforge: Standardize Toolforge CLI user interface looks - https://phabricator.wikimedia.org/T348442 (10Slst2020) The planned consolidation of the CLIs should help with this {T348749} [15:47:57] (03PS1) 10JMeybohm: kubernetes: Remove cergen certs from kubernetes secrets [labs/private] - 10https://gerrit.wikimedia.org/r/980891 (https://phabricator.wikimedia.org/T300033) [16:08:43] 10Toolforge (Toolforge iteration 02): [tbs][builds-api] Refactor `internal/builds.go` - https://phabricator.wikimedia.org/T352762 (10Slst2020) a:03Slst2020 [16:11:44] 10Toolforge (Toolforge iteration 02): [tbs][builds-api] Refactor `internal/builds.go` - https://phabricator.wikimedia.org/T352762 (10Slst2020) 05Open→03In progress [16:21:38] (03PS1) 10Andrew Bogott: Keystone: add fake cred keys [labs/private] - 10https://gerrit.wikimedia.org/r/980900 [16:35:13] (03CR) 10Andrew Bogott: [V: 03+2 C: 03+2] Keystone: add fake cred keys [labs/private] - 10https://gerrit.wikimedia.org/r/980900 (owner: 10Andrew Bogott) [16:43:48] 10Toolforge: php 8.2 crashes when using XMLReader - https://phabricator.wikimedia.org/T352886 (10Wurgl) [17:10:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:15:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:19:46] 10Grid-Engine-to-K8s-Migration: Migrate croptool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319653 (10nskaggs) @Danmichaelo I think https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Installing_Apt_packages should meet your needs now if I'm understanding... [17:19:58] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Maintenance, 10User-dcaro: [tf-infra-tests] Failing to destroy - volumes stuck - https://phabricator.wikimedia.org/T352895 (10dcaro) p:05Triage→03High [17:20:50] 10Grid-Engine-to-K8s-Migration, 10Toolforge (Software install/update), 10WMCZ-General: Make it possible to run pandoc in Toolforge's jobs framework - https://phabricator.wikimedia.org/T345029 (10nskaggs) [17:21:30] 10Grid-Engine-to-K8s-Migration, 10Toolforge (Software install/update), 10Toolforge Jobs framework, 10WMCZ-General: Make it possible to run pandoc in Toolforge's jobs framework - https://phabricator.wikimedia.org/T345029 (10nskaggs) [17:21:34] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Maintenance, 10User-dcaro: [tf-infra-tests] Failing to destroy - volumes stuck - https://phabricator.wikimedia.org/T352895 (10dcaro) 05Open→03In progress [17:25:38] (CephSlowOps) firing: Ceph cluster in eqiad has 3 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [17:25:44] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [17:30:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 90 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [17:36:55] (03CR) 10Catrope: [C: 03+2] releases: Bump Codex to 1.1.1 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/980909 (owner: 10Anne Tomasevich) [17:37:56] (03Merged) 10jenkins-bot: releases: Bump Codex to 1.1.1 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/980909 (owner: 10Anne Tomasevich) [18:44:53] 10Cloud-VPS, 10SRE: enable lists.wikimedia.org or wikimedia.org addresses to receive dmarc reports for *.wmflabs.org - https://phabricator.wikimedia.org/T352902 (10jsn.sherman) [18:55:09] 10Cloud-VPS, 10SRE: enable lists.wikimedia.org or wikimedia.org email addresses to receive dmarc reports for *.wmflabs.org - https://phabricator.wikimedia.org/T352902 (10jsn.sherman) [19:10:28] 10Toolforge (Toolforge iteration 02): [tbs.build.logs] Show a more user-friendly error message when logs are not ready - https://phabricator.wikimedia.org/T341059 (10Raymond_Ndibe) 05In progress→03Resolved [19:13:45] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service (Beta release), 10User-Raymond_Ndibe, 10User-dcaro: Add a way to wait for a Toolforge build to finish - https://phabricator.wikimedia.org/T337043 (10Raymond_Ndibe) 05In progress→03Resolved a:03Raymond_Ndibe [19:23:05] (03PS1) 10JJMC89: remove PAWS from #pywikibot [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/980928 [19:34:35] 10Grid-Engine-to-K8s-Migration: Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320140 (10LucasWerkmeister) 05Stalled→03Open I suppose this is unstalled now, or at least the cloud maintainers think so (given that there’s now a hard deadline for thi... [19:39:29] (03CR) 10Majavah: [C: 03+2] remove PAWS from #pywikibot [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/980928 (owner: 10JJMC89) [19:40:02] (03Merged) 10jenkins-bot: remove PAWS from #pywikibot [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/980928 (owner: 10JJMC89) [19:55:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [19:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [20:22:09] 10Grid-Engine-to-K8s-Migration: Migrate dplbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319701 (10russblau) I would appreciate some confirmation that this tool's jobs will not be deleted on 14 December. Working on migration. [20:23:09] 10Grid-Engine-to-K8s-Migration: Migrate dplbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319701 (10russblau) a:05Dispenser→03russblau [20:56:33] 10Cloud-VPS, 10SRE: enable lists.wikimedia.org or wikimedia.org email addresses to receive dmarc reports for *.wmflabs.org - https://phabricator.wikimedia.org/T352902 (10herron) In cases where outbound mail delivery is important basic inbound mail handling should be configured for the (sub)domain and any from... [22:01:03] 10Grid-Engine-to-K8s-Migration: Migrate dplbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319701 (10nskaggs) @russblau Thanks for reaching out! We won't shutdown the tool on December 14th, as we've been able to establish contact with at least one maintainer (yourself).... [22:06:28] 10Grid-Engine-to-K8s-Migration: Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320140 (10nskaggs) @LucasWerkmeister I would encourage you to try using apt to fulfill missing dependencies: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Servic... [22:09:56] 10Grid-Engine-to-K8s-Migration: Migrate wikihistory from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320157 (10nskaggs) @Wurgl Can you use the system package for mono? If so, try building now with the php buildpack and add mono using apt. See https://wikitech.wikimedia.org/w... [22:16:29] 10Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883 (10nskaggs) @MBH for #3, have you tried using the system installed mono via apt https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Installing_Apt_packages? [22:27:30] 10Cloud-VPS, 10SRE: enable lists.wikimedia.org or wikimedia.org email addresses to receive dmarc reports for *.wmflabs.org - https://phabricator.wikimedia.org/T352902 (10jsn.sherman) For inbound mail delivery: What are our options that avoid exposing an unmaintained mail server to the Internet? Internal mail r... [22:27:46] 10Grid-Engine-to-K8s-Migration: Migrate wikihistory from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320157 (10Wurgl) @nskaggs Okay. I have already read that magic "Aptfile". Question 1: Which of the packages here do I need? Please note: I have almost zero experiance with... [22:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [22:55:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [23:21:30] 10Cloud-VPS, 10DNS, 10SRE, 10Traffic: DNS name resolution failure with www.spacecom.mil from Cloud VPS - https://phabricator.wikimedia.org/T346471 (10Dzahn) Can confirm this is still the case. From a random different cloud VPS instance: ` dig www.spacecom.mil @172.20.255.1` fails. (and 172.20.255.1 is... [23:24:01] 10Cloud-VPS, 10DNS, 10SRE, 10Traffic: DNS name resolution failure with www.spacecom.mil from Cloud VPS - https://phabricator.wikimedia.org/T346471 (10Dzahn) It's not ALL of .mil either. For example "dig cybercoe.army.mil @172.20.255.1" works and also points to an Akamai edge. [23:32:55] 10Cloud-VPS, 10DNS, 10SRE, 10Traffic: DNS name resolution failure with www.spacecom.mil from Cloud VPS - https://phabricator.wikimedia.org/T346471 (10Don-vip) Yes, my tool scans for free media at following .mil domains without problem: - www.afspc.af.mil - www.buckley.spaceforce.mil - www.jtf-spaced... [23:37:42] 10Grid-Engine-to-K8s-Migration: Migrate noclaims from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319927 (10komla) @Multichill we will definitely not sabotage your grid jobs! If there are any issues stopping the migration for a specific tool, kindly let us know under each t... [23:38:45] 10Grid-Engine-to-K8s-Migration: Migrate multichill from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319912 (10komla) @Multichill again, we will definitely not sabotage your grid jobs! If there are any issues stopping the migration for a specific tool, kindly let us know unde... [23:39:41] 10Grid-Engine-to-K8s-Migration: Migrate geograph from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319765 (10komla) @Multichill we will definitely not sabotage your grid jobs! If there are any issues stopping the migration for a specific tool, kindly let us know under each ti... [23:40:52] 10Grid-Engine-to-K8s-Migration: Migrate family from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319739 (10komla) @Multichill we will definitely not sabotage your grid jobs! If there are any issues stopping the migration for a specific tool, kindly let us know under each tick... [23:41:44] 10Grid-Engine-to-K8s-Migration: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787 (10komla) @Multichill we will definitely not sabotage your grid jobs! If there are any issues stopping the migration for a specific tool, kindly let us know under each ti...