[00:13:08] 06cloud-services-team, 10Toolforge: Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127 (10bd808) 03NEW [00:20:42] FIRING: CloudVPSDesignateLeaks: Detected 40 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [02:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:40:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.095% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:20:42] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [05:23:07] (03open) 10bd808: Rate limit channel joins [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/7 (https://phabricator.wikimedia.org/T380124) [05:25:52] (03merge) 10bd808: Rate limit channel joins [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/7 (https://phabricator.wikimedia.org/T380124) [05:46:30] (03open) 10bd808: Increase channel join wait [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/8 [05:48:26] (03merge) 10bd808: Increase channel join wait [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/8 [06:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [07:40:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.667% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:20:42] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [10:01:31] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-4 is lagging behind the primary, the current lag is 3699 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [10:11:31] RESOLVED: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-4 is lagging behind the primary, the current lag is 4119 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [10:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:52:12] (03PS1) 10Urbanecm: cswiki: announcers/zops: Make the bot less sensitive [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1091881 [10:52:22] (03CR) 10Urbanecm: [C:03+2] cswiki: announcers/zops: Make the bot less sensitive [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1091881 (owner: 10Urbanecm) [10:53:31] (03Merged) 10jenkins-bot: cswiki: announcers/zops: Make the bot less sensitive [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1091881 (owner: 10Urbanecm) [11:04:43] PROBLEM - Disk space on cloudbackup1004 is CRITICAL: DISK CRITICAL - free space: /srv 649778MiB (3% inode=99%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1004&var-datasource=eqiad+prometheus/ops [11:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:40:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 3.909% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:20:42] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [13:33:29] (03open) 10bd808: Wait 2.5s between channel joins [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/9 (https://phabricator.wikimedia.org/T380124) [13:35:38] (03merge) 10bd808: Wait 2.5s between channel joins [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/9 (https://phabricator.wikimedia.org/T380124) [14:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:40:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.887% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:44:43] RECOVERY - Disk space on cloudbackup1004 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1004&var-datasource=eqiad+prometheus/ops [15:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [16:20:42] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [17:57:31] (03open) 10bd808: Allow tuning join delay with ISS_JOIN_DELAY_MILLIS envvar [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/10 (https://phabricator.wikimedia.org/T380124) [18:04:32] (03update) 10bd808: Allow tuning join delay with ISS_JOIN_DELAY_MILLIS envvar [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/10 (https://phabricator.wikimedia.org/T380124) [18:16:19] 10VPS-project-Codesearch, 10dev-images: Add releng/dev-images to codesearch - https://phabricator.wikimedia.org/T380133 (10Daimona) 03NEW [18:19:38] (03PS1) 10Daimona Eaytoy: Index releng/dev-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1091896 (https://phabricator.wikimedia.org/T380133) [18:19:58] 10VPS-project-Codesearch, 10dev-images, 13Patch-For-Review: Add releng/dev-images to codesearch - https://phabricator.wikimedia.org/T380133#10329690 (10Daimona) a:03Daimona [18:20:00] (03merge) 10bd808: Allow tuning join delay with ISS_JOIN_DELAY_MILLIS envvar [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/10 (https://phabricator.wikimedia.org/T380124) [18:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [20:00:28] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate wikilabels-backups-01.wikilabels.eqiad.wmflabs is about to expire in 26d 23h 54m 26s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [20:17:42] 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10329764 (10waldyrious) Shouldn't the new members be added (or add themselves) to the [[https://phabricator.wikimedia.org/project/members/2457/|Toolforge-standards... [20:20:42] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [21:33:55] 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10329802 (10bd808) >>! In T370474#10329764, @waldyrious wrote: > Shouldn't the new members be added (or add themselves) to the [[https://phabricator.wikimedia.org/... [22:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange