[02:36:28] <wmcs-alerts>	 FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: project-proxy-puppetmaster-01.project-proxy.eqiad.wmflabs is about to expire in 27d 23h 58m 54s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[08:12:58] <wikibugs>	 (03merge) 10taavi: eqiad1: Enable IPv6 default security group rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218 (https://phabricator.wikimedia.org/T245495)
[08:13:13] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[08:13:53] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[08:26:31] <wikibugs>	 06cloud-services-team, 10Data-Services: maintain-dbusers: Use cloud-private to talk to NFS servers instead of proxies - https://phabricator.wikimedia.org/T392794 (10taavi) 03NEW
[08:47:59] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10771751 (10fnegri) > We can close this, right?  Yep, and the task is already "Resolved", so nothing left to do!
[09:00:36] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10771775 (10dcaro) > Can you help me think about this by providing a concrete use case...
[09:09:30] <wikibugs>	 06cloud-services-team, 10Toolforge: Retire explicit 'roots' sudo policies - https://phabricator.wikimedia.org/T392797 (10taavi) 03NEW
[09:25:31] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: tofu-infra: re-implement security group rules as a map instead of array - https://phabricator.wikimedia.org/T392799 (10aborrero) 03NEW
[09:29:52] <wikibugs>	 10VPS-project-Codesearch, 10Wikidata, 10Wikidata Query UI, 10wmde-wikidata-tech: Update Wikidata Query GUI URL in Codesearch - https://phabricator.wikimedia.org/T392691#10771867 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03Lucas_Werkmeister_WMDE Seems to be working now \o/
[09:43:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudcontrol1011:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:44:46] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10771887 (10dcaro) Some notes:  * I'd add `components-api` to `builds-api` and `jobs-api` (as it uses them, if not now i...
[09:45:58] <wikibugs>	 10Tool-wosretbot, 06Toolforge-standards-committee: Adoption request for wosretbot - https://phabricator.wikimedia.org/T392781#10771889 (10Tkarcher) > I see two pieces of secret information so far: [...] wosretbot/nonpublic.toml, with user name(s) to exclude from certain notifications? I don’t really understand...
[09:53:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol1011:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:56:46] <wikibugs>	 (03open) 10aborrero: tofu-infra: re-implement security group rules as a map instead of array [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/219 (https://phabricator.wikimedia.org/T392799)
[09:59:43] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10771955 (10dcaro) Note that I'd wait until we have https://gitlab.wikimedia.org/repos...
[09:59:47] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] Upgrade from 10.6.20 to 10.6.21 - https://phabricator.wikimedia.org/T392596#10771956 (10fnegri)
[10:00:09] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] Upgrade from 10.6.20 to 10.6.21 - https://phabricator.wikimedia.org/T392596#10771957 (10fnegri) p:05Triage→03Medium
[10:00:47] <wikibugs>	 (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) (owner: 10raymond-ndibe)
[11:20:16] <wikibugs>	 (03approved) 10taavi: tofu-infra: re-implement security group rules as a map instead of array [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/219 (https://phabricator.wikimedia.org/T392799) (owner: 10aborrero)
[11:20:20] <wikibugs>	 (03merge) 10taavi: tofu-infra: re-implement security group rules as a map instead of array [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/219 (https://phabricator.wikimedia.org/T392799) (owner: 10aborrero)
[11:20:24] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[11:25:10] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[11:25:39] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: re-implement security group rules as a map instead of array - https://phabricator.wikimedia.org/T392799#10772125 (10taavi) 05Open→03Resolved
[11:51:25] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: openstack: make vxlan/ipv6-dualstack network the default for new instances - https://phabricator.wikimedia.org/T374824#10772207 (10taavi) p:05Low→03Medium
[11:51:33] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#10772209 (10taavi) 05Stalled→03Open
[11:58:30] <wikibugs>	 06cloud-services-team, 10Horizon, 10Striker, 06serviceops, 06SRE: Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478#10772235 (10taavi) /cc @Andrew   Main thing to note here is that Horizon needs to be able to talk to cloud-realm services....
[11:58:32] <wikibugs>	 10Tool-wikicordo: In the cordo tool also show thumbnails for files that are candidates for deletion - https://phabricator.wikimedia.org/T392809 (10Prototyperspective) 03NEW
[11:59:46] <wikibugs>	 10Tool-wikicordo: In the cordo tool also show thumbnails for files that are candidates for deletion - https://phabricator.wikimedia.org/T392809#10772249 (10Prototyperspective)
[12:03:28] <wikibugs>	 (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) (owner: 10raymond-ndibe)
[12:08:14] <wikibugs>	 (03update) 10dcaro: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) (owner: 10raymond-ndibe)
[12:10:32] <wikibugs>	 (03open) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717)
[12:23:32] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6: metricsinfra: Support scraping v6-enabled instances - https://phabricator.wikimedia.org/T392570#10772314 (10taavi) This is also an issue on Toolforge. I'm going to use that to prototype a fix and then roll out a proper fix onto metricsinfra.
[12:24:02] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 07IPv6: metricsinfra: Support scraping v6-enabled instances - https://phabricator.wikimedia.org/T392570#10772315 (10taavi)
[12:28:05] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10772339 (10Raymond_Ndibe) > Now, if we wanted to support a job exposing arbitrary TCP...
[12:31:04] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10772346 (10Raymond_Ndibe) > I don't feel there is a clear and significant request fro...
[12:34:09] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10772360 (10Raymond_Ndibe) > One thing that I'd like to make possible is routing speci...
[12:35:57] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10772365 (10Raymond_Ndibe) >>! In T388092#10769733, @bd808 wrote: >> In the future `jo...
[12:39:53] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] Upgrade from 10.6.20 to 10.6.21 - https://phabricator.wikimedia.org/T392596#10772371 (10fnegri) 05Open→03In progress
[12:40:01] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10772372 (10taavi) >>! In T388092#10771775, @dcaro wrote: >> Can you help me think abo...
[12:48:44] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10772393 (10dcaro) > In general no CA will issue double-wildcard certificates (e.g. *....
[12:49:55] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10772394 (10taavi) At that point you might as well have multiple tools..
[12:51:44] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 13Patch-For-Review: wikireplicas: maintain-views should not create _p databases - https://phabricator.wikimedia.org/T392105#10772398 (10fnegri) @FCeratto-WMF @Marostegui I would like to hear your thoughts on this task and on th...
[13:08:14] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#10772495 (10fnegri) >>! In T318479#10772483, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-cloud), href=https:/...
[13:13:31] <wmcs-alerts>	 FIRING: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState
[13:13:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[13:14:31] <wmcs-alerts>	 FIRING: ToolsToolsDBReplicationError: ToolsDB replication is broken on tools-db-5 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError
[13:21:38] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Split ProjectProxyMainProxyDown to only page when main VIP is unreachable - https://phabricator.wikimedia.org/T381107#10772569 (10taavi) a:03taavi
[13:21:48] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717)
[13:28:40] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Split ProjectProxyMainProxyDown to only page when main VIP is unreachable - https://phabricator.wikimedia.org/T381107#10772591 (10taavi) 05Open→03Resolved ` MariaDB [prometheusconfig]> select * from alerts where id in (15, 30)\...
[13:35:44] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#10772621 (10dcaro) >>! In T388092#10772394, @taavi wrote: > At that point you might as...
[13:35:50] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Documentation, 10Sustainability (Incident Followup): ProjectProxyMainProxyDown should have a runbook page - https://phabricator.wikimedia.org/T361873#10772622 (10taavi)
[13:51:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Documentation, 10Sustainability (Incident Followup): ProjectProxyMainProxyDown should have a runbook page - https://phabricator.wikimedia.org/T361873#10772694 (10taavi) 05Open→03Resolved a:03taavi I've bootstrapped these two pages: * https://wikitech.wikimedi...
[13:53:38] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 10Sustainability (Incident Followup): [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components - https://phabricator.wikimedia.org/T380959#10772710 (10taavi) >  jobs-service (review https://wikitech.wikim...
[13:56:07] <wikibugs>	 (03open) 10fnegri: toolsdb: Failover primary [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/220 (https://phabricator.wikimedia.org/T392596)
[13:57:48] <wikibugs>	 (03update) 10fnegri: toolsdb: Failover primary [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/220 (https://phabricator.wikimedia.org/T392596)
[13:58:15] <wikibugs>	 (03approved) 10taavi: toolsdb: Failover primary [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/220 (https://phabricator.wikimedia.org/T392596) (owner: 10fnegri)
[13:59:31] <wmcs-alerts>	 FIRING: ToolsToolsDBReplicationMissing: ToolsDB replication is not running on tools-db-5 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing
[14:01:01] <wmcs-alerts>	 RESOLVED: ToolsToolsDBReplicationError: ToolsDB replication is broken on tools-db-5 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError
[14:02:01] <wmcs-alerts>	 RESOLVED: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState
[14:03:56] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[14:04:31] <wmcs-alerts>	 RESOLVED: ToolsToolsDBReplicationMissing: ToolsDB replication is not running on tools-db-5 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing
[14:08:15] <wikibugs>	 (03close) 10fnegri: toolsdb: Failover primary [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/220 (https://phabricator.wikimedia.org/T392596)
[14:25:54] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 13Patch-For-Review: [toolsdb] Upgrade from 10.6.20 to 10.6.21 - https://phabricator.wikimedia.org/T392596#10772808 (10fnegri) Restarting mariadb on tools-db-5 was very fast (just a few seconds).  On tools-db-4, the shutdown took about 50 minutes. It's n...
[14:30:33] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 07Puppet: Puppet removed "nameserver" line from /etc/resolv.conf - https://phabricator.wikimedia.org/T379927#10772825 (10taavi) Anything left to do here?
[14:45:48] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 07Puppet: Puppet removed "nameserver" line from /etc/resolv.conf - https://phabricator.wikimedia.org/T379927#10772886 (10ssingh) >>! In T379927#10772825, @taavi wrote: > Anything left to do here?  Nothing on the prod DNS hosts side; if you k...
[15:13:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6: Enable IPv6 for the maps proxy - https://phabricator.wikimedia.org/T392826 (10taavi) 03NEW
[15:16:37] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'maps-proxy' (T392826)
[15:16:40] <stashbot>	 T392826: Enable IPv6 for the maps proxy - https://phabricator.wikimedia.org/T392826
[15:16:41] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy END (ERROR) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=97) with prefix 'maps-proxy' (T392826)
[15:16:46] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'maps-proxy' (T392826)
[15:23:03] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'maps-proxy'
[15:24:28] <wmcs-alerts>	 FIRING: InstanceDown: Project project-proxy instance maps-proxy-5 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:25:24] <wikibugs>	 06cloud-services-team, 10Toolforge: [toolsdb] MariaDB sometimes takes very long to shut down - https://phabricator.wikimedia.org/T392828 (10fnegri) 03NEW
[15:27:56] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'maps-proxy' (T392826)
[15:27:59] <stashbot>	 T392826: Enable IPv6 for the maps proxy - https://phabricator.wikimedia.org/T392826
[15:28:56] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] Upgrade from 10.6.20 to 10.6.21 - https://phabricator.wikimedia.org/T392596#10773116 (10fnegri) While the shutdown was in progress, there was a constant write activity on disk (hovering between 10 and 20 Mbps). Some MariaDB threads were sometime...
[15:34:07] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'maps-proxy'
[15:34:23] <wikibugs>	 06cloud-services-team, 10Toolforge: [toolsdb] MariaDB sometimes takes very long to shut down - https://phabricator.wikimedia.org/T392828#10773144 (10fnegri) p:05Triage→03Low In the last occurrence, nothing was logged between those 2 lines:  ` Apr 28 13:07:32 tools-db-4 mysqld[992820]: 2025-04-28 13:07:32 0...
[15:34:28] <wmcs-alerts>	 FIRING: InstanceDown: Project project-proxy instance maps-proxy-6 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:37:04] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): [toolsdb] Use DNS CNAMEs instead of A records - https://phabricator.wikimedia.org/T392831 (10fnegri) 03NEW
[15:37:13] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): [toolsdb] Use DNS CNAMEs instead of A records - https://phabricator.wikimedia.org/T392831#10773190 (10fnegri) p:05Triage→03Low
[15:45:37] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] Upgrade from 10.6.20 to 10.6.21 - https://phabricator.wikimedia.org/T392596#10773222 (10fnegri) 05In progress→03Resolved I created {T392828} to track the issue with slow shutdowns.  I also changed the [upgrade procedure in Wikitech](http...
[15:48:57] <wikibugs>	 (03open) 10taavi: eqiad1: maps-proxy: Import static-proxy rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221 (https://phabricator.wikimedia.org/T392826)
[15:49:05] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
[15:49:23] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221
[16:29:39] <wikibugs>	 (03PS1) 10Majavah: Format with latest black [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139513
[16:29:39] <wikibugs>	 (03PS1) 10Majavah: prometheus: Use DNS names to refer to instances [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139514 (https://phabricator.wikimedia.org/T392570)
[16:29:58] <wikibugs>	 (03open) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11
[16:30:16] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Format with latest black [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139513 (owner: 10Majavah)
[16:30:21] <wikibugs>	 (03CR) 10CI reject: [V:04-1] prometheus: Use DNS names to refer to instances [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139514 (https://phabricator.wikimedia.org/T392570) (owner: 10Majavah)
[16:31:51] <wikibugs>	 (03PS2) 10Majavah: Format with latest black [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139513
[16:31:51] <wikibugs>	 (03PS2) 10Majavah: prometheus: Use DNS names to refer to instances [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139514 (https://phabricator.wikimedia.org/T392570)
[16:32:33] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Format with latest black [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139513 (owner: 10Majavah)
[16:32:34] <wikibugs>	 (03CR) 10CI reject: [V:04-1] prometheus: Use DNS names to refer to instances [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139514 (https://phabricator.wikimedia.org/T392570) (owner: 10Majavah)
[16:35:14] <wikibugs>	 (03PS3) 10Majavah: Fix build [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139513
[16:35:14] <wikibugs>	 (03PS3) 10Majavah: prometheus: Use DNS names to refer to instances [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139514 (https://phabricator.wikimedia.org/T392570)
[16:38:23] <wikibugs>	 (03CR) 10Majavah: [C:03+2] Fix build [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139513 (owner: 10Majavah)
[16:39:39] <wikibugs>	 (03Merged) 10jenkins-bot: Fix build [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139513 (owner: 10Majavah)
[16:45:15] <wikibugs>	 10Tool-wosretbot, 06Toolforge-standards-committee: Adoption request for wosretbot - https://phabricator.wikimedia.org/T392781#10773537 (10LucasWerkmeister) >>! In T392781#10771889, @Tkarcher wrote: > I can tell you the name is misleading, as the file is fully public (I just tested it, I can read it).  I was aw...
[16:57:58] <wikibugs>	 (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11
[16:58:56] <wikibugs>	 10Tool-wosretbot, 06Toolforge-standards-committee: Adoption request for wosretbot - https://phabricator.wikimedia.org/T392781#10773564 (10LucasWerkmeister) I looked for non-world-readable files again with `find . -name .kube -prune -or -name .cache -prune -or ! -perm -o=r`, and apart from the files mentioned a...
[16:59:53] <wikibugs>	 (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11
[17:01:06] <wikibugs>	 10Tool-wosretbot, 06Toolforge-standards-committee: Adoption request for wosretbot - https://phabricator.wikimedia.org/T392781#10773574 (10LucasWerkmeister) Should be done, I think – please check :)
[17:21:24] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10773653 (10LucasWerkmeister) >>! In T389540#10664553, @AntiCompositeNumber wrote: > That provision was introduced in May 2023, and we know that FNDE logged in to Toolforge in Jan 2024 (T320003#947490...
[17:28:21] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10773665 (10LucasWerkmeister) 05Stalled→03Open
[17:31:07] <wikibugs>	 (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11
[17:32:09] <wikibugs>	 10Toolforge (Toolforge iteration 19), 07Epic: Fix toolforge tests and deployment cicd pipelines - https://phabricator.wikimedia.org/T392524#10773677 (10dcaro) We might want to split this task in two, one for the voulteer/non-admin MRs testing, and one for the cli pipeline.   For the volunteer flow, we had a qu...
[17:36:59] <wikibugs>	 (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) (owner: 10raymond-ndibe)
[17:40:16] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717)
[17:40:51] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717)
[17:44:16] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [jobs-api] Introduce deprecation metrics - https://phabricator.wikimedia.org/T390137#10773711 (10dcaro)
[17:46:17] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [jobs-api] Introduce deprecation metrics - https://phabricator.wikimedia.org/T390137#10773727 (10dcaro)
[17:48:00] <wikibugs>	 10superset.wmcloud.org: Upgrade to 4.0.0 - https://phabricator.wikimedia.org/T364022#10773749 (10bd808) a:05rook→03None
[17:48:26] <wikibugs>	 10superset.wmcloud.org: Public viewing of superset - https://phabricator.wikimedia.org/T336522#10773753 (10bd808) a:05rook→03None
[19:39:08] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10774116 (10LucasWerkmeister) Well, there are three sets of credentials that I’ve found:  1. The password of FNBot on Wikimedia wikis. (Not a bot password!) I expect that, as in T392781, we’ll want to...
[19:47:40] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10774148 (10bd808) >>! In T389540#10774116, @LucasWerkmeister wrote: > 1. The password of FNBot on Wikimedia wikis. (Not a bot password!) I expect that, as in T392781, we’ll want to remove this from t...
[21:39:49] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10774663 (10BTullis) Thanks @fnegri for all of your har...