[00:11:28] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[00:14:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[00:16:28] <wmcs-alerts>	 (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:16:28] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[00:21:28] <wmcs-alerts>	 (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:24:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[00:43:16] <wikibugs>	 10Tool-extjsonuploader, 10MediaWiki-Documentation: Display autogenerated extension information in Template:Extension - https://phabricator.wikimedia.org/T222479#9667687 (10Pppery)
[01:24:28] <wmcs-alerts>	 (PuppetSyncFailure) firing: Failed to update Puppet repository /srv/git/labs/private on instance toolsbeta-puppetserver-1 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure
[01:40:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:45:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:50:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:55:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) resolved: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:53:16] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[03:00:22] <jinxer-wm>	 (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[04:40:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:45:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:50:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:55:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) resolved: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[06:53:16] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[07:00:38] <jinxer-wm>	 (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[07:40:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[07:45:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[07:50:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[07:55:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[10:53:16] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[11:00:38] <jinxer-wm>	 (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[11:02:42] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
[11:08:54] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
[11:09:28] <wmcs-alerts>	 (PuppetSyncFailure) resolved: Failed to update Puppet repository /srv/git/labs/private on instance toolsbeta-puppetserver-1 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure
[11:13:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[11:23:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[11:43:11] <wikibugs>	 10Cloud-VPS, 10MediaWiki-Vagrant, 13Patch-For-Review: 14Update Vagrant puppet role  to work on Bookworm. - 14https://phabricator.wikimedia.org/T356551#9668845 (10taavi) 05Open→03Declined
[11:45:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[11:55:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[12:12:19] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
[12:12:26] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
[12:13:47] <wikibugs>	 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Synthesise user testing results - https://phabricator.wikimedia.org/T358098#9668962 (10KColeman-WMF)
[12:14:28] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-docker-registry-8 in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[12:27:12] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1015312 (owner: 10L10n-bot)
[12:27:13] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1015313 (owner: 10L10n-bot)
[12:38:31] <wmcs-alerts>	 (ToolsToolsDBReplicationLagIsTooHigh) firing: ToolsDB replication on tools-db-3 is lagging behind the primary, the current lag is 70344 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh
[12:39:28] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tools-docker-registry-8 in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[12:40:28] <wmcs-alerts>	 (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 2 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[12:40:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[12:44:00] <wikibugs>	 10Toolforge (Toolforge iteration 07): 14dbreps job pending to start for 2d16h on Toolforge - 14https://phabricator.wikimedia.org/T358175#9668998 (10taavi) 05In progress→03Resolved
[12:45:41] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[12:58:56] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
[13:00:28] <wmcs-alerts>	 (PuppetStaleCertificates) resolved: Found non-revoked Puppet certificates for 2 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[13:02:56] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy'
[13:05:31] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
[13:06:39] <wikibugs>	 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10XTools, 07Design, and 2 others: [Design] Communicate to users why there gaps in IP data on Special:Contributions - https://phabricator.wikimedia.org/T360928#9669055 (10KColeman-WMF)
[13:06:55] <wikibugs>	 10Cloud-VPS, 10Spicerack, 10SRE-tools: spicerack.puppet.PuppetHostsError: Unable to find CSR fingerprints for all hosts, detected errors are: Another puppet instance is already running and the waitforlock setting is set to 0; exiting - https://phabricator.wikimedia.org/T361218 (10taavi) 03NEW
[13:07:11] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud
[13:07:55] <wikibugs>	 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: 14[Design] Synthesise user testing results - 14https://phabricator.wikimedia.org/T358098#9669069 (10KColeman-WMF) 05In progress→03Resolved
[13:10:45] <wikibugs>	 10Cloud-VPS: Drop 68.10.in-addr.arpa. from Designate - https://phabricator.wikimedia.org/T361220 (10taavi) 03NEW
[13:11:48] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy'
[13:16:32] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy'
[13:16:58] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4
[13:17:17] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4
[13:18:31] <wikibugs>	 10Toolforge: Upgrade Toolforge front proxies to Bookworm - https://phabricator.wikimedia.org/T361223 (10taavi) 03NEW
[13:18:39] <wikibugs>	 10Toolforge: Upgrade Toolforge front proxies to Bookworm - https://phabricator.wikimedia.org/T361223#9669155 (10taavi)
[13:18:43] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9669156 (10taavi)
[13:19:22] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
[13:20:27] <wmcs-alerts>	 (ProbeDown) firing: Service toolsbeta-proxy-4:443 has failed probes (http_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-proxy-4:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:25:12] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
[13:30:50] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
[13:35:22] <jinxer-wm>	 (HAProxyBackendUnavailable) resolved: HAProxy service nova-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[13:36:30] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
[13:41:15] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q#:rack/setup/install (2) cloudbackup hosts - https://phabricator.wikimedia.org/T356216#9669229 (10Andrew) I suspect that on re-run partman isn't properly zeroing out the partition table before it starts so we're accumula...
[13:41:28] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-proxy-8 in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[13:50:01] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests): 14Temporarily increase quota for dwl Buster migration - 14https://phabricator.wikimedia.org/T360788#9669262 (10Andrew) 05Open→03Resolved a:03Andrew 14thanks! I lowered quotas but left a little wiggle room.
[13:50:27] <wmcs-alerts>	 (ProbeDown) resolved: Service toolsbeta-proxy-4:443 has failed probes (http_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-proxy-4:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:56:28] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tools-proxy-8 in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[14:01:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3
[14:01:46] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3
[14:05:27] <wmcs-alerts>	 (ProbeDown) firing: Service toolsbeta-proxy-3:443 has failed probes (http_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-proxy-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:05:28] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-etcd-24 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[14:15:27] <wmcs-alerts>	 (ProbeDown) resolved: Service toolsbeta-proxy-3:443 has failed probes (http_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-proxy-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:15:28] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-etcd-24 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[14:25:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (T360699)
[14:25:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
[14:25:31] <stashbot>	 T360699: Toolsbeta: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T360699
[14:26:39] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
[14:27:30] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
[14:27:47] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (T360699)
[14:30:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
[14:30:41] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Kubernetes: [infra] Upgrade Toolforge K8s etcd nodes to Bookworm - https://phabricator.wikimedia.org/T361237 (10taavi) 03NEW
[14:31:02] <wmcs-alerts>	 (ProbeDown) firing: Service tools-proxy-05:443 has failed probes (http_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-proxy-05:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:32:56] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (T360699)
[14:33:00] <stashbot>	 T360699: Toolsbeta: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T360699
[14:33:16] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
[14:36:02] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1015313 (owner: 10L10n-bot)
[14:36:47] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1015312 (owner: 10L10n-bot)
[14:41:02] <wmcs-alerts>	 (ProbeDown) resolved: Service tools-proxy-05:443 has failed probes (http_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-proxy-05:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:41:17] <wmcs-alerts>	 (ProbeDown) firing: Service tools-proxy-05:443 has failed probes (http_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-proxy-05:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:42:32] <wmcs-alerts>	 (ProbeDown) resolved: Service tools-proxy-05:443 has failed probes (http_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-proxy-05:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:42:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (T349207)
[14:42:50] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[14:42:51] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) (T349207)
[14:43:14] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (T349207)
[14:44:04] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
[14:46:21] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (T349207)
[14:46:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) (T349207)
[14:47:32] <wmcs-alerts>	 (ProbeDown) firing: Service tools-proxy-05:443 has failed probes (http_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-proxy-05:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:48:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (T349207)
[14:48:15] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[14:48:26] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99)
[14:48:47] <wmcs-alerts>	 (ProbeDown) resolved: Service tools-proxy-05:443 has failed probes (http_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-proxy-05:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:49:01] <wmcs-alerts>	 (ToolsToolsDBReplicationLagIsTooHigh) resolved: ToolsDB replication on tools-db-3 is lagging behind the primary, the current lag is 4602 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh
[14:52:32] <wmcs-alerts>	 (ProbeDown) firing: Service tools-proxy-05:443 has failed probes (http_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-proxy-05:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:53:16] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[15:02:32] <wmcs-alerts>	 (ProbeDown) resolved: Service tools-proxy-05:443 has failed probes (http_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-proxy-05:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:19:12] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (T349207)
[15:19:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) (T349207)
[15:19:18] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[15:23:13] <wikibugs>	 06cloud-services-team: Supporting AI, LLM, and data models on WMCS - https://phabricator.wikimedia.org/T336905#9669812 (10fnegri) There's no pending discussion at the moment, so I'm moving this task out of "Needs discussion" column and back to the inbox. Feel free to leave a comment if you would like this to be...
[15:24:10] <wikibugs>	 06cloud-services-team: Supporting AI, LLM, and data models on WMCS - https://phabricator.wikimedia.org/T336905#9669819 (10fnegri) p:05Triage→03Low
[15:24:43] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Infrastructure Security: WMCS-roots wiki replica access - https://phabricator.wikimedia.org/T337848#9669814 (10fnegri) There's no pending discussion at the moment, so I'm moving this task out of "Needs discussion" column and back to the inbox. Feel free to leave a co...
[15:26:16] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Infrastructure Security: WMCS-roots wiki replica access - https://phabricator.wikimedia.org/T337848#9669839 (10fnegri)
[15:29:11] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
[15:29:15] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[15:30:01] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
[15:35:55] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
[15:35:59] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[15:36:44] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
[15:47:21] <wikibugs>	 10Tool-ducttape, 06Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator, 03Abstract Wikipedia Fix-It tasks: Improve CI for Wikifunctions services to better test like reality - https://phabricator.wikimedia.org/T350284#9669938 (10Jdforrester-WMF) p:05Triage→03Medium a:03Jdforrester-...
[15:49:01] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
[15:49:05] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[15:49:49] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
[15:53:44] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
[15:53:53] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q#:rack/setup/install (2) cloudbackup hosts - https://phabricator.wikimedia.org/T356216#9669968 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudbackup2003.codfw....
[15:54:37] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
[16:09:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
[16:09:18] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[16:12:44] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: 14WMCS cookbooks: provide shared hosts for people without global root privileges - 14https://phabricator.wikimedia.org/T343330#9670070 (10fnegri) 05Open→03Resolved a:03fnegri 14There are still some gaps ({T344412}) but I think the main requirem...
[16:13:36] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
[16:15:21] <wikibugs>	 10Tool-refill: 14Refill tool stuck "waiting for an available worker" - 14https://phabricator.wikimedia.org/T361091#9670086 (10TheresNoTime) 05Open→03Resolved
[16:23:32] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Infrastructure Security: 14WMCS-roots wiki replica access - 14https://phabricator.wikimedia.org/T337848#9670115 (10fnegri) 05Open→03Resolved a:03fnegri 14I updated the description of this task noting which parts have been fixed since the description was wr...
[16:24:16] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
[16:24:19] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[16:26:28] <wmcs-alerts>	 (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-test-localdisk on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:33:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
[16:42:28] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Infrastructure Security, 10Puppet (Puppet 7.0): cloudcumin: support reimage and other operations - https://phabricator.wikimedia.org/T344412#9670220 (10fnegri)
[16:45:56] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (5) Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[16:47:25] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q#:rack/setup/install (2) cloudbackup hosts - https://phabricator.wikimedia.org/T356216#9670266 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudbackup2003.codfw.wmne...
[16:47:49] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q#:rack/setup/install (2) cloudbackup hosts - https://phabricator.wikimedia.org/T356216#9670267 (10Jhancock.wm)
[16:50:13] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: 14Q#:rack/setup/install (2) cloudbackup hosts - 14https://phabricator.wikimedia.org/T356216#9670269 (10Jhancock.wm) 05Open→03Resolved 14issue fixed and ready to go @Andrew 
[16:50:48] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
[16:56:34] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
[16:58:54] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
[17:04:29] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
[17:13:14] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Infrastructure Security, 10Puppet (Puppet 7.0): cloudcumin: support reimage and other operations - https://phabricator.wikimedia.org/T344412#9670399 (10fnegri) Note that all `sre.*` cookbooks are no longer installed in cloudcumins (see the discussion in T343894)....
[17:13:16] <wikibugs>	 10Toolforge, 07Documentation: Consolidate information about tool memory, resources, and quota into one doc - https://phabricator.wikimedia.org/T347887#9670403 (10TBurmeister) The new document I suggested creating when I filed this would be a how-to page focused on the task of managing tool quota and resources....
[17:13:32] <wikibugs>	 (03PS1) 10Krinkle: frontend: change Issue tracker link from "Create form" to "workboard" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015355
[17:42:00] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
[17:42:04] <stashbot>	 T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207
[17:51:35] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
[18:53:16] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[19:34:04] <wikibugs>	 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Migrate per-project Puppet servers to Puppet 7 - https://phabricator.wikimedia.org/T351452#9671029 (10Andrew)
[19:35:16] <wikibugs>	 06cloud-services-team, 10VPS-Projects, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Puppet (Puppet 7.0): 14Update Integration project puppetmaster - 14https://phabricator.wikimedia.org/T360461#9671030 (10Andrew) 05Open→03Resolved a:03Andrew
[19:37:41] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Move Cloud VPS auth.logs to central logging - https://phabricator.wikimedia.org/T127717#9671034 (10Andrew) @Southparkfan We're trying to reduce use of Buster in cloud-vps, and two servers in 'auditlogging' are running Buster:  syslog-s...
[19:49:44] <wikibugs>	 (03PS1) 10Dzahn: delete contint.wikimedia.org dummy key, migrated to cfssl [labs/private] - 10https://gerrit.wikimedia.org/r/1015377 (https://phabricator.wikimedia.org/T360413)
[19:51:28] <wikibugs>	 (03PS2) 10Dzahn: delete contint.wikimedia.org dummy key, migrated to cfssl [labs/private] - 10https://gerrit.wikimedia.org/r/1015377 (https://phabricator.wikimedia.org/T360413)
[19:54:19] <wikibugs>	 (03CR) 10Dzahn: [V:03+2 C:03+2] delete contint.wikimedia.org dummy key, migrated to cfssl [labs/private] - 10https://gerrit.wikimedia.org/r/1015377 (https://phabricator.wikimedia.org/T360413) (owner: 10Dzahn)
[20:15:14] <wikibugs>	 10Data-Services, 06DBA, 10MediaWiki-Logevents, 07Security: Make (redacted) log_search table available on Wiki Replicas - https://phabricator.wikimedia.org/T85756#9671281 (10Pppery)
[20:16:25] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] frontend: change Issue tracker link from "Create form" to "workboard" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015355 (owner: 10Krinkle)
[20:17:19] <wikibugs>	 (03Merged) 10jenkins-bot: frontend: change Issue tracker link from "Create form" to "workboard" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015355 (owner: 10Krinkle)
[20:21:54] <wikibugs>	 (03PS1) 10Krinkle: write_config: index `mediawiki/gadgets/*` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015384
[20:22:15] <wikibugs>	 (03PS2) 10Krinkle: write_config: index `mediawiki/gadgets/*` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015384
[20:22:40] <wikibugs>	 (03PS3) 10Krinkle: write_config: index `mediawiki/gadgets/*` and `wikipedia/gadgets/*` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015384
[20:33:01] <jinxer-wm>	 (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[20:45:56] <jinxer-wm>	 (CloudVPSDesignateLeaks) firing: (5) Detected 14 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[20:54:05] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Move Cloud VPS auth.logs to central logging - https://phabricator.wikimedia.org/T127717#9671489 (10Southparkfan) >>! In T127717#9671034, @Andrew wrote: > @Southparkfan We're trying to reduce use of Buster in cloud-vps, and two servers...
[21:04:28] <wmcs-alerts>	 (PuppetAgentNoResources) firing: No Puppet resources found on instance runner-1025 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[21:08:37] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Move Cloud VPS auth.logs to central logging - https://phabricator.wikimedia.org/T127717#9671526 (10Andrew) >>! In T127717#9671489, @Southparkfan wrote: >>>! In T127717#9671034, @Andrew wrote: >> @Southparkfan We're trying to reduce use...
[21:19:28] <wmcs-alerts>	 (PuppetAgentNoResources) resolved: No Puppet resources found on instance runner-1025 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[21:30:10] <wikibugs>	 (03PS1) 10Krinkle: build: Replace grunt-git-authors dependency with simple git-log command [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015396
[21:30:10] <wikibugs>	 (03PS1) 10Krinkle: Tag v2.3.5 [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015397
[21:30:19] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] build: Replace grunt-git-authors dependency with simple git-log command [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015396 (owner: 10Krinkle)
[21:30:58] <wikibugs>	 (03Merged) 10jenkins-bot: build: Replace grunt-git-authors dependency with simple git-log command [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015396 (owner: 10Krinkle)
[21:31:21] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Tag v2.3.5 [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015397 (owner: 10Krinkle)
[21:31:59] <wikibugs>	 (03Merged) 10jenkins-bot: Tag v2.3.5 [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015397 (owner: 10Krinkle)
[21:39:04] <wikibugs>	 (03PS1) 10Krinkle: core: Remove unused `Intuition->version` property [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015398
[21:39:26] <wikibugs>	 (03CR) 10CI reject: [V:04-1] core: Remove unused `Intuition->version` property [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015398 (owner: 10Krinkle)
[21:40:05] <wikibugs>	 (03PS2) 10Krinkle: core: Remove unused `Intuition->version` property [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015398
[21:40:14] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] core: Remove unused `Intuition->version` property [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015398 (owner: 10Krinkle)
[21:40:28] <wikibugs>	 (03CR) 10CI reject: [V:04-1] core: Remove unused `Intuition->version` property [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015398 (owner: 10Krinkle)
[21:41:20] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] write_config: index `mediawiki/gadgets/*` and `wikipedia/gadgets/*` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015384 (owner: 10Krinkle)
[21:42:07] <wikibugs>	 (03Merged) 10jenkins-bot: write_config: index `mediawiki/gadgets/*` and `wikipedia/gadgets/*` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015384 (owner: 10Krinkle)
[21:47:14] <wikibugs>	 (03CR) 10Samtar: [C:03+2] Handle expiry better [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013683 (owner: 10Operator873)
[21:47:51] <wikibugs>	 (03CR) 10JJMC89: Handle expiry better (031 comment) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013683 (owner: 10Operator873)
[21:48:42] <wikibugs>	 (03PS3) 10Krinkle: core: Remove unused `Intuition->version` property [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015398
[21:50:46] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] core: Remove unused `Intuition->version` property [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015398 (owner: 10Krinkle)
[21:54:28] <wikibugs>	 (03Merged) 10jenkins-bot: Handle expiry better [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1013683 (owner: 10Operator873)
[21:54:29] <wikibugs>	 (03Merged) 10jenkins-bot: core: Remove unused `Intuition->version` property [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015398 (owner: 10Krinkle)
[22:32:38] <wikibugs>	 (03PS1) 10Krinkle: Tag v2.3.6 [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015407
[22:32:53] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Tag v2.3.6 [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015407 (owner: 10Krinkle)
[22:35:10] <wikibugs>	 (03Merged) 10jenkins-bot: Tag v2.3.6 [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1015407 (owner: 10Krinkle)
[22:48:25] <wikibugs>	 10VPS-project-Codesearch: Sort repo search results alphabetically - https://phabricator.wikimedia.org/T339191#9671707 (10Krinkle) The list is intentionally sorted by relevance based on fuzzy match. This is incompatible with an alphabetic search as that would potentially cut off the most relevant entry to outside...
[22:56:34] <wikibugs>	 (03PS1) 10Krinkle: write_config: Change git.legoktm.com url-pattern to `/src/` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015424 (https://phabricator.wikimedia.org/T304879)
[22:58:00] <wikibugs>	 (03CR) 10CI reject: [V:04-1] write_config: Change git.legoktm.com url-pattern to `/src/` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015424 (https://phabricator.wikimedia.org/T304879) (owner: 10Krinkle)
[22:59:29] <wikibugs>	 10VPS-project-Codesearch, 13Patch-For-Review: Codesearch: Fix ShoutHow repository viewer links to link to a viewable version of the matching file instead of prompting the user to download the file - https://phabricator.wikimedia.org/T304879#9671737 (10Krinkle) a:03Krinkle File browser URLs work in Gogs over...
[23:05:12] <wikibugs>	 (03PS2) 10Krinkle: write_config: Change git.legoktm.com url-pattern to `/src/` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015424 (https://phabricator.wikimedia.org/T304879)
[23:21:41] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] write_config: Change git.legoktm.com url-pattern to `/src/` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015424 (https://phabricator.wikimedia.org/T304879) (owner: 10Krinkle)
[23:21:52] <wikibugs>	 10wikitech.wikimedia.org, 10DiscussionTools: Page state routing triggers DiscussionTools warning, e.g. #!/deploycal/current - https://phabricator.wikimedia.org/T361322 (10Krinkle) 03NEW
[23:22:25] <wikibugs>	 10wikitech.wikimedia.org, 10DiscussionTools: Page state routing triggers DiscussionTools warning, e.g. #!/deploycal/current - https://phabricator.wikimedia.org/T361322#9671869 (10Krinkle)
[23:22:35] <wikibugs>	 (03Merged) 10jenkins-bot: write_config: Change git.legoktm.com url-pattern to `/src/` [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015424 (https://phabricator.wikimedia.org/T304879) (owner: 10Krinkle)
[23:29:15] <wikibugs>	 10wikitech.wikimedia.org, 10DiscussionTools, 10Editing-team (Kanban Board), 13Patch-For-Review: Page state routing triggers DiscussionTools warning, e.g. #!/deploycal/current - https://phabricator.wikimedia.org/T361322#9671884 (10DLynch) a:03DLynch
[23:35:41] <wikibugs>	 06cloud-services-team, 10VPS-Projects, 06collaboration-services, 10Puppet (Puppet 7.0): Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9671908 (10brennen) I configured a new `puppetmaster-1003` through to step 7 above.  Ran into some issues (detailed below) switch `phabric...
[23:43:30] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Move Cloud VPS auth.logs to central logging - https://phabricator.wikimedia.org/T127717#9671931 (10Southparkfan) >>! In T127717#9671526, @Andrew wrote: >>>! In T127717#9671489, @Southparkfan wrote: >>>>! In T127717#9671034, @Andrew wro...
[23:51:26] <wikibugs>	 (03PS1) 10Krinkle: build: Add .gitreview file [labs/tools/editorinteract] - 10https://gerrit.wikimedia.org/r/1015431
[23:51:34] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] build: Add .gitreview file [labs/tools/editorinteract] - 10https://gerrit.wikimedia.org/r/1015431 (owner: 10Krinkle)
[23:58:58] <wikibugs>	 06cloud-services-team, 10VPS-Projects, 06collaboration-services, 10Puppet (Puppet 7.0): Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9671959 (10brennen) That did the trick. Second run:  ` brennen@phabricator-bullseye:/etc/puppet$ sudo run-puppet-agent Info: Using configu...