[00:04:49] 10Tool-ifttt, 10User-bd808: Setup toolinfo record for IFTTT tool - https://phabricator.wikimedia.org/T342009 (10bd808) 05Open→03Resolved a:03bd808 https://toolsadmin.wikimedia.org/tools/id/ifttt [00:12:03] (TfInfraTestApplyFailed) resolved: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:33:06] 10Toolforge Jobs framework: Add a container for Swift - https://phabricator.wikimedia.org/T354815 (10Fastily) [00:35:33] 10Toolforge Jobs framework: Add a container for Swift - https://phabricator.wikimedia.org/T354815 (10Fastily) [00:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:07:46] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:12:46] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-4:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:21:14] 10Toolforge (Toolforge iteration 03): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10Hawkeye7) I have no access to podman on your server, but I can run locally on my own. ram900@ram900-test2:~/milhistbot-liftwing$ bin/Debug/net7.0/Liftwing usage: Liftwing (ProbeDown) firing: Service tools-k8s-haproxy-4:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:04:03] (PuppetAgentFailure) firing: Puppet agent failure detected on instance tools-k8s-worker-100 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [03:07:53] (ProbeDown) resolved: Service tools-k8s-haproxy-4:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:52:54] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:57:54] (ProbeDown) resolved: Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:20:53] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:21:48] 10Toolforge (Toolforge iteration 03): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10Hawkeye7) Aargh. Error on my part. I will check in a version that reports errors better. [04:25:53] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:30:53] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:34:53] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:39:54] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:06:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:11:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:35:53] (ProbeDown) firing: Service tools-k8s-haproxy-4:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:40:53] (ProbeDown) resolved: Service tools-k8s-haproxy-4:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:04:03] (PuppetAgentFailure) firing: Puppet agent failure detected on instance tools-k8s-worker-100 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [06:30:54] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:35:54] (ProbeDown) resolved: Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:01:54] (ProbeDown) firing: Service tools-k8s-haproxy-4:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:06:53] (ProbeDown) resolved: Service tools-k8s-haproxy-4:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:28:54] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:33:53] (ProbeDown) resolved: Service tools-k8s-haproxy-3:30000 has failed probes (http_this_tool_does_not_exist_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:24:03] (PuppetAgentFailure) resolved: Puppet agent failure detected on instance tools-k8s-worker-100 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:29:03] (InstanceDown) firing: Project tools instance tools-k8s-worker-100 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:34:03] (InstanceDown) resolved: Project tools instance tools-k8s-worker-100 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:01:07] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10cloud-services-team: [harbor] Use `harbor_up` for alerting about harbor components being down - https://phabricator.wikimedia.org/T354736 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_request... [09:18:09] 10Toolforge Build Service, 10cloud-services-team, 10Patch-For-Review: [harbor,trove] Trove DB filled disk and caused toolforge-build to fail as a result - https://phabricator.wikimedia.org/T354714 (10dcaro) [09:18:12] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10cloud-services-team: [harbor] Use `harbor_up` for alerting about harbor components being down - https://phabricator.wikimedia.org/T354736 (10dcaro) 05Open→03Resolved [09:18:40] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: [harbor] Redis using all available memory - https://phabricator.wikimedia.org/T354176 (10dcaro) [09:21:31] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: [harbor] Redis using all available memory - https://phabricator.wikimedia.org/T354176 (10dcaro) Having a quick look at the graph (there was a hiccup on the last outage though), it seems... [09:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:50:19] 10Toolforge (Toolforge iteration 03): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10dcaro) For the Procfile, I think there's a bug and it's being ignored: ` [step-inject-buildpacks] 2024-01-10T23:47:36.073122841Z [[order]] [step-inject-buildpacks] 2024-01-10T23... [09:51:23] 10Toolforge (Toolforge iteration 03): [bulids-builder,dotnet] The Procfile buildpack is not being injected in the dotnet group - https://phabricator.wikimedia.org/T354831 (10dcaro) [09:51:33] 10Toolforge (Toolforge iteration 03): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10dcaro) [09:52:08] 10Toolforge (Toolforge iteration 03): [bulids-builder,dotnet] The Procfile buildpack is not being injected in the dotnet group - https://phabricator.wikimedia.org/T354831 (10dcaro) 05Open→03In progress [09:56:24] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [bulids-builder,dotnet] The Procfile buildpack is not being injected in the dotnet group - https://phabricator.wikimedia.org/T354831 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/28 do... [10:50:03] (03PS1) 10Majavah: Add fake wmcs-rsa DKIM keys for Cloud VPS [labs/private] - 10https://gerrit.wikimedia.org/r/989738 (https://phabricator.wikimedia.org/T354112) [11:44:29] 10Toolforge, 10Patch-For-Review: Setup Pint for tools-prometheus - https://phabricator.wikimedia.org/T354760 (10CodeReviewBot) taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/9 monitoring: Add alerts for Pint lint failures [11:44:55] (03CR) 10FNegri: [C: 03+1] Add fake wmcs-rsa DKIM keys for Cloud VPS [labs/private] - 10https://gerrit.wikimedia.org/r/989738 (https://phabricator.wikimedia.org/T354112) (owner: 10Majavah) [11:45:10] (03CR) 10Majavah: [V: 03+2 C: 03+2] Add fake wmcs-rsa DKIM keys for Cloud VPS [labs/private] - 10https://gerrit.wikimedia.org/r/989738 (https://phabricator.wikimedia.org/T354112) (owner: 10Majavah) [12:21:45] 10Cloud-VPS, 10Toolforge, 10cloud-services-team, 10Patch-For-Review: Ensure Toolforge and Cloud VPS comply with Google's new email sender guidelines - https://phabricator.wikimedia.org/T354112 (10taavi) [12:27:46] 10VPS-project-Codesearch: Add bluespice/mw-config/overrides to CodeSearch - https://phabricator.wikimedia.org/T354852 (10Reedy) [12:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:41:03] (InstanceDown) resolved: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:46:30] 10Cloud-VPS (Quota-requests), 10cloud-services-team: Increase disk quota for math - https://phabricator.wikimedia.org/T354579 (10Physikerwelt) 05Resolved→03Open @Andrew thank you. For the next migration steps, I would need to reallocate resources from math-docker (11f33a4a-abee-4c40-b0c3-8df59658faac), whi... [13:10:07] 10Openstack-Magnum: magnum: kubectl fails to connect after time - https://phabricator.wikimedia.org/T336586 (10rook) Updating the fcos version seems to be working in dev. [13:11:09] 10Openstack-Magnum: magnum: kubectl fails to connect after time - https://phabricator.wikimedia.org/T336586 (10rook) 05Stalled→03In progress [13:23:03] (03CR) 10Ayounsi: [C: 03+1] hiera: add fake swift passwords for netbox_dev user [labs/private] - 10https://gerrit.wikimedia.org/r/989531 (https://phabricator.wikimedia.org/T354766) (owner: 10MVernon) [13:37:41] (CloudVPSDesignateLeaks) firing: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:55:14] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10User-Raymond_Ndibe: [builds-api,logs] Increas pod starting timeout to the same as the request - https://phabricator.wikimedia.org/T354856 (10dcaro) [14:10:54] 10Toolforge, 10Patch-For-Review: Setup Pint for tools-prometheus - https://phabricator.wikimedia.org/T354760 (10CodeReviewBot) taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/9 monitoring: Add alerts for Pint lint failures [14:21:51] 10Cloud-VPS (Quota-requests), 10cloud-services-team: Increase disk quota for math - https://phabricator.wikimedia.org/T354579 (10Physikerwelt) 05Open→03Resolved Thank you. [14:42:11] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: [build-serviece,clojure] Current supported heroku builder does not yet include clojure support - https://phabricator.wikimedia.org/T353575 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/bui... [14:44:46] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: [build-serviece,clojure] Current supported heroku builder does not yet include clojure support - https://phabricator.wikimedia.org/T353575 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gi... [14:45:06] 10Cloud-VPS (Quota-requests), 10cloud-services-team: Increase disk quota for math - https://phabricator.wikimedia.org/T354579 (10Andrew) It looks like you got that VM deleted, finally... is that right? [14:53:41] 10Cloud-VPS, 10cloud-services-team (Hardware), 10SRE, 10ops-eqiad: Cloudvirt1063.eqiad.wmnet overheating - https://phabricator.wikimedia.org/T353408 (10Jclark-ctr) Updated firmware per Dells Request cleared logs resent new tsr report. waiting for response. [14:54:17] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10User-Raymond_Ndibe: [builds-api,logs] Increase pod starting timeout to the same as the request - https://phabricator.wikimedia.org/T354856 (10dcaro) [15:10:14] !log toolsbeta dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:10:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:10:49] !log toolsbeta dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [15:10:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:13:37] !log tools dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:13:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:14:12] !log tools dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [15:14:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:15:41] 10Cloud-VPS (Quota-requests), 10cloud-services-team: Increase disk quota for math - https://phabricator.wikimedia.org/T354579 (10Physikerwelt) I thought it was you. Maybe it just took longer, but it was deleted eventually. [15:18:05] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: [build-serviece,clojure] Current supported heroku builder does not yet include clojure support - https://phabricator.wikimedia.org/T353575 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/too... [15:18:19] (03CR) 10Majavah: [C: 03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/989219 (https://phabricator.wikimedia.org/T346631) (owner: 10FNegri) [15:18:48] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: [build-serviece,clojure] Current supported heroku builder does not yet include clojure support - https://phabricator.wikimedia.org/T353575 (10dcaro) \o/ support restored! Will move to the cloud-native buildpack once they ha... [15:19:16] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: [build-serviece,clojure] Current supported heroku builder does not yet include clojure support - https://phabricator.wikimedia.org/T353575 (10dcaro) 05In progress→03Resolved [15:21:48] 10Toolforge Build Service: builds-cli utils/bump_version.sh fails with '--userns: invalid USER mode.' - https://phabricator.wikimedia.org/T354876 (10taavi) [15:30:32] 10Toolforge Build Service: builds-cli utils/bump_version.sh fails with '--userns: invalid USER mode.' - https://phabricator.wikimedia.org/T354876 (10dcaro) that is actually meant for podman [15:31:34] 10Toolforge Build Service: builds-cli utils/bump_version.sh fails with '--userns: invalid USER mode.' - https://phabricator.wikimedia.org/T354876 (10dcaro) Do you have podman installed? I think that the check does: ` if command -v podman >/dev/null; then ` We can overwrite that with an envar or similar, or chan... [15:34:31] 10Toolforge Build Service: builds-cli utils/bump_version.sh fails with '--userns: invalid USER mode.' - https://phabricator.wikimedia.org/T354876 (10taavi) 05Open→03Invalid Apparently yes? I don't think I use it for anything so I'll uninstalled it and the error went away. Thanks. :/ [15:38:31] 10Toolforge Build Service, 10Patch-For-Review: builds-cli utils/bump_version.sh fails with '--userns: invalid USER mode.' - https://phabricator.wikimedia.org/T354876 (10dcaro) I created https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/50 anyhow, that might make it clearer that it'... [15:38:52] 10Toolforge: Setup Pint for tools-prometheus - https://phabricator.wikimedia.org/T354760 (10taavi) 05Open→03Resolved [15:39:11] 10Toolforge Build Service, 10Patch-For-Review: builds-cli utils/bump_version.sh fails with '--userns: invalid USER mode.' - https://phabricator.wikimedia.org/T354876 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/50 bump_version/build_deb: use pod... [15:41:31] (03CR) 10MVernon: [V: 03+2 C: 03+2] hiera: add fake swift passwords for netbox_dev user [labs/private] - 10https://gerrit.wikimedia.org/r/989531 (https://phabricator.wikimedia.org/T354766) (owner: 10MVernon) [15:50:41] 10Toolforge Build Service, 10Patch-For-Review: builds-cli utils/bump_version.sh fails with '--userns: invalid USER mode.' - https://phabricator.wikimedia.org/T354876 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/50 bump_version/build_deb: use pod... [16:37:04] (03CR) 10FNegri: SAL logging: invert user and project (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/989219 (https://phabricator.wikimedia.org/T346631) (owner: 10FNegri) [16:50:31] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/24 buildpacks:... [16:51:49] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tool... [16:54:01] 10Cloud-VPS (Quota-requests), 10cloud-services-team: Increase disk quota for math - https://phabricator.wikimedia.org/T354579 (10Andrew) It wasn't me but it might've been someone :) Anyway, glad you're all set. [17:07:14] !log toolsbeta dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [17:07:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:07:51] !log toolsbeta dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [17:07:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:12:21] !log tools dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [17:12:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:12:55] !log tools dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [17:12:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:18:46] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/173 builds-bu... [17:21:59] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service, 10Patch-For-Review: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10dcaro) @0xDeadbeef I've just enabled the locale buildpack, can you try it out? (all you have to do is to add the locales you ne... [17:25:03] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [bulids-builder,dotnet] The Procfile buildpack is not being injected in the dotnet group - https://phabricator.wikimedia.org/T354831 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/28 do... [17:26:27] !log toolsbeta dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [17:26:27] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [bulids-builder,dotnet] The Procfile buildpack is not being injected in the dotnet group - https://phabricator.wikimedia.org/T354831 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/tool... [17:26:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:27:01] !log toolsbeta dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [17:27:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:30:15] !log tools dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [17:30:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:37:09] 10Toolforge (Toolforge iteration 03): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10dcaro) [17:37:12] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10dcaro) 05In progress→03Resolved [17:38:07] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [bulids-builder,dotnet] The Procfile buildpack is not being injected in the dotnet group - https://phabricator.wikimedia.org/T354831 (10dcaro) 05In progress→03Resolved [18:34:29] 10Grid-Engine-to-K8s-Migration: Migrate postcardcatfinder from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319973 (10DB111) Migrated. [18:34:49] 10Grid-Engine-to-K8s-Migration: Migrate postcardcatfinder from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319973 (10DB111) 05Open→03Resolved a:03DB111 [18:52:55] 10cloud-services-team (Hardware), 10DC-Ops, 10ops-codfw: Q3:rack/setup/install cloudcontrol2006-dev.codfw.wmnet - https://phabricator.wikimedia.org/T354896 (10RobH) [18:53:16] 10cloud-services-team (Hardware), 10DC-Ops, 10ops-codfw: Q3:rack/setup/install cloudcontrol2006-dev.codfw.wmnet - https://phabricator.wikimedia.org/T354896 (10RobH) [19:26:09] (CloudVPSDesignateLeaks) resolved: (2) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:35:10] 10PAWS: update helm chart and jupyterhub - https://phabricator.wikimedia.org/T354898 (10rook) [20:05:05] RECOVERY - Host cloudvirt1063 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [20:08:49] PROBLEM - Host cloudvirt1063 is DOWN: PING CRITICAL - Packet loss = 100% [20:12:43] RECOVERY - Host cloudvirt1063 is UP: PING OK - Packet loss = 0%, RTA = 1.79 ms [20:19:01] PROBLEM - Host cloudvirt1063 is DOWN: PING CRITICAL - Packet loss = 100% [20:23:25] (NodeDown) firing: Cloudvirt node cloudvirt1063 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1063 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:29:27] 10Toolforge (Toolforge iteration 03): [envvars-cli] move pytest from tox to pre-commit - https://phabricator.wikimedia.org/T351476 (10Raymond_Ndibe) 05In progress→03Declined [20:37:03] (InstanceDown) firing: Project tools instance tools-k8s-worker-84 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:42:23] (ToolforgeKubernetesNodeNotReady) firing: Kubernetes node tools-k8s-worker-84 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [21:32:03] (InstanceDown) resolved: Project tools instance tools-k8s-worker-84 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:32:23] (ToolforgeKubernetesNodeNotReady) resolved: Kubernetes node tools-k8s-worker-84 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [23:46:56] 10Toolforge (Toolforge iteration 03): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10Hawkeye7) It seems to be working now: tools.milhistbot@tools-sgebastion-10:~/bin$ runit 'Hanford Engineer Works' tools.milhistbot@tools-sgebastion-10:~/bin$ toolforge jobs logs... [23:48:18] 10Toolforge (Toolforge iteration 03): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10Hawkeye7) I find it strange that you are looking for the Program.cs file rather than the *.csproj file, which would seem more logical.