[00:08:20] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1175225 [00:08:20] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1175225 (owner: 10TrainBranchBot) [00:32:13] (03CR) 10CI reject: [V:04-1] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1175225 (owner: 10TrainBranchBot) [00:55:34] 06SRE, 10LDAP-Access-Requests: Logstash Access for gergesshamon - https://phabricator.wikimedia.org/T399421#11055775 (10Gerges) @CDobbins, Yes, I still need access to Logstash. I do need an NDA, but please note that I’m working independently as a volunteer, without support from team wmf. [01:41:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:05:48] FIRING: ThumborHighHaproxyErrorRate: Thumbor haproxy error rate for pod thumbor-main-6d8d7547b7-fhjq8 - eqiad - https://wikitech.wikimedia.org/wiki/Thumbor - https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor - https://alerts.wikimedia.org/?q=alertname%3DThumborHighHaproxyErrorRate [03:09:29] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate thanos-query.discovery.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [03:38:59] FIRING: KubernetesDeploymentUnavailableReplicas: ... [03:38:59] Deployment thumbor-main in thumbor at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=thumbor&var-deployment=thumbor-main - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplica [04:25:48] RESOLVED: ThumborHighHaproxyErrorRate: Thumbor haproxy error rate for pod thumbor-main-6d8d7547b7-fhjq8 - eqiad - https://wikitech.wikimedia.org/wiki/Thumbor - https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor - https://alerts.wikimedia.org/?q=alertname%3DThumborHighHaproxyErrorRate [05:09:44] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:40:32] PROBLEM - Disk space on an-worker1127 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/e 149772 MB (3% inode=99%): /var/lib/hadoop/data/g 156683 MB (4% inode=99%): /var/lib/hadoop/data/j 156279 MB (4% inode=99%): /var/lib/hadoop/data/c 160072 MB (4% inode=99%): /var/lib/hadoop/data/b 159794 MB (4% inode=99%): /var/lib/hadoop/data/l 153903 MB (4% inode=99%): /var/lib/hadoop/data/k 152920 MB (4% inode=99%): /var/lib/hadoop/data [05:40:32] 1 MB (4% inode=99%): /var/lib/hadoop/data/i 155642 MB (4% inode=99%): /var/lib/hadoop/data/m 163287 MB (4% inode=99%): /var/lib/hadoop/data/d 160194 MB (4% inode=99%): /var/lib/hadoop/data/h 154639 MB (4% inode=99%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=an-worker1127&var-datasource=eqiad+prometheus/ops [06:09:44] RESOLVED: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:20:32] PROBLEM - Disk space on an-worker1127 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/e 151245 MB (4% inode=99%): /var/lib/hadoop/data/g 153880 MB (4% inode=99%): /var/lib/hadoop/data/j 153759 MB (4% inode=99%): /var/lib/hadoop/data/c 157597 MB (4% inode=99%): /var/lib/hadoop/data/b 159237 MB (4% inode=99%): /var/lib/hadoop/data/l 150619 MB (4% inode=99%): /var/lib/hadoop/data/k 149166 MB (3% inode=99%): /var/lib/hadoop/data [06:20:32] 5 MB (4% inode=99%): /var/lib/hadoop/data/i 153610 MB (4% inode=99%): /var/lib/hadoop/data/m 162058 MB (4% inode=99%): /var/lib/hadoop/data/d 161929 MB (4% inode=99%): /var/lib/hadoop/data/h 151980 MB (4% inode=99%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=an-worker1127&var-datasource=eqiad+prometheus/ops [06:51:10] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [06:53:18] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [06:58:28] PROBLEM - mailman list info ssl expiry on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250803T0700) [07:00:24] RECOVERY - mailman list info ssl expiry on lists1004 is OK: OK - Certificate lists.wikimedia.org will expire on Mon 06 Oct 2025 08:56:14 PM GMT +0000. https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [07:03:00] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 8922 bytes in 0.187 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [07:03:08] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 54369 bytes in 0.217 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [07:09:29] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate thanos-query.discovery.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [07:17:10] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [07:17:18] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [07:19:00] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 8922 bytes in 0.174 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [07:19:08] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 54368 bytes in 0.068 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [07:38:59] FIRING: KubernetesDeploymentUnavailableReplicas: ... [07:38:59] Deployment thumbor-main in thumbor at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=thumbor&var-deployment=thumbor-main - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplica [07:40:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on wdqs2025:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:55:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on wdqs2025:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:02:10] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [08:03:00] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 8923 bytes in 1.087 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [09:40:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on wdqs2025:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:55:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on wdqs2025:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:09:29] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate thanos-query.discovery.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [11:20:32] PROBLEM - Disk space on an-worker1127 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/e 160319 MB (4% inode=99%): /var/lib/hadoop/data/g 156337 MB (4% inode=99%): /var/lib/hadoop/data/j 156485 MB (4% inode=99%): /var/lib/hadoop/data/c 157472 MB (4% inode=99%): /var/lib/hadoop/data/b 160535 MB (4% inode=99%): /var/lib/hadoop/data/l 153745 MB (4% inode=99%): /var/lib/hadoop/data/k 149721 MB (3% inode=99%): /var/lib/hadoop/data [11:20:32] 8 MB (4% inode=99%): /var/lib/hadoop/data/i 157994 MB (4% inode=99%): /var/lib/hadoop/data/m 157585 MB (4% inode=99%): /var/lib/hadoop/data/d 161609 MB (4% inode=99%): /var/lib/hadoop/data/h 160599 MB (4% inode=99%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=an-worker1127&var-datasource=eqiad+prometheus/ops [11:25:48] FIRING: ThumborHighHaproxyErrorRate: Thumbor haproxy error rate for pod thumbor-main-6d8d7547b7-4bfqm - eqiad - https://wikitech.wikimedia.org/wiki/Thumbor - https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor - https://alerts.wikimedia.org/?q=alertname%3DThumborHighHaproxyErrorRate [11:26:14] PROBLEM - Disk space on an-worker1129 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/b 177674 MB (4% inode=99%): /var/lib/hadoop/data/l 168972 MB (4% inode=99%): /var/lib/hadoop/data/k 154854 MB (4% inode=99%): /var/lib/hadoop/data/c 146933 MB (3% inode=99%): /var/lib/hadoop/data/d 174395 MB (4% inode=99%): /var/lib/hadoop/data/e 175746 MB (4% inode=99%): /var/lib/hadoop/data/g 179784 MB (4% inode=99%): /var/lib/hadoop/data [11:26:14] 4 MB (4% inode=99%): /var/lib/hadoop/data/i 155170 MB (4% inode=99%): /var/lib/hadoop/data/j 154206 MB (4% inode=99%): /var/lib/hadoop/data/m 184369 MB (4% inode=99%): /var/lib/hadoop/data/f 178583 MB (4% inode=99%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=an-worker1129&var-datasource=eqiad+prometheus/ops [11:38:59] FIRING: KubernetesDeploymentUnavailableReplicas: ... [11:38:59] Deployment thumbor-main in thumbor at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=thumbor&var-deployment=thumbor-main - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplica [12:03:21] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance [12:03:39] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [12:03:47] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1156 (T400854)', diff saved to https://phabricator.wikimedia.org/P80547 and previous config saved to /var/cache/conftool/dbconfig/20250803-120346-ladsgroup.json [12:03:49] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [12:06:30] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1156 (T400854)', diff saved to https://phabricator.wikimedia.org/P80548 and previous config saved to /var/cache/conftool/dbconfig/20250803-120629-ladsgroup.json [12:21:37] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P80549 and previous config saved to /var/cache/conftool/dbconfig/20250803-122136-ladsgroup.json [12:36:45] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P80550 and previous config saved to /var/cache/conftool/dbconfig/20250803-123644-ladsgroup.json [12:41:58] 06SRE, 10MediaWiki-extensions-QuickInstantCommons, 10MediaWiki-File-management, 06Traffic: Make InstantCommons and other uses of ForeignApiRepo use WMF policy-compliant user agents - https://phabricator.wikimedia.org/T400881#11055956 (10Tgr) When the images are hotlinked (but the downstream wiki still need... [12:51:52] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1156 (T400854)', diff saved to https://phabricator.wikimedia.org/P80551 and previous config saved to /var/cache/conftool/dbconfig/20250803-125152-ladsgroup.json [12:51:55] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [12:52:07] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance [12:52:15] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1162 (T400854)', diff saved to https://phabricator.wikimedia.org/P80552 and previous config saved to /var/cache/conftool/dbconfig/20250803-125214-ladsgroup.json [12:54:45] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1162 (T400854)', diff saved to https://phabricator.wikimedia.org/P80553 and previous config saved to /var/cache/conftool/dbconfig/20250803-125444-ladsgroup.json [13:09:53] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P80554 and previous config saved to /var/cache/conftool/dbconfig/20250803-130952-ladsgroup.json [13:25:01] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P80555 and previous config saved to /var/cache/conftool/dbconfig/20250803-132500-ladsgroup.json [13:40:09] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1162 (T400854)', diff saved to https://phabricator.wikimedia.org/P80556 and previous config saved to /var/cache/conftool/dbconfig/20250803-134008-ladsgroup.json [13:40:12] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [13:40:13] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance [13:40:20] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1182 (T400854)', diff saved to https://phabricator.wikimedia.org/P80557 and previous config saved to /var/cache/conftool/dbconfig/20250803-134019-ladsgroup.json [13:43:01] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1182 (T400854)', diff saved to https://phabricator.wikimedia.org/P80558 and previous config saved to /var/cache/conftool/dbconfig/20250803-134300-ladsgroup.json [13:45:48] RESOLVED: ThumborHighHaproxyErrorRate: Thumbor haproxy error rate for pod thumbor-main-6d8d7547b7-4bfqm - eqiad - https://wikitech.wikimedia.org/wiki/Thumbor - https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor - https://alerts.wikimedia.org/?q=alertname%3DThumborHighHaproxyErrorRate [13:58:09] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P80559 and previous config saved to /var/cache/conftool/dbconfig/20250803-135808-ladsgroup.json [14:13:17] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P80560 and previous config saved to /var/cache/conftool/dbconfig/20250803-141316-ladsgroup.json [14:28:24] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1182 (T400854)', diff saved to https://phabricator.wikimedia.org/P80561 and previous config saved to /var/cache/conftool/dbconfig/20250803-142824-ladsgroup.json [14:28:27] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [14:28:40] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance [14:28:47] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1188 (T400854)', diff saved to https://phabricator.wikimedia.org/P80562 and previous config saved to /var/cache/conftool/dbconfig/20250803-142847-ladsgroup.json [14:31:18] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1188 (T400854)', diff saved to https://phabricator.wikimedia.org/P80563 and previous config saved to /var/cache/conftool/dbconfig/20250803-143117-ladsgroup.json [14:46:25] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P80564 and previous config saved to /var/cache/conftool/dbconfig/20250803-144624-ladsgroup.json [15:01:33] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P80565 and previous config saved to /var/cache/conftool/dbconfig/20250803-150132-ladsgroup.json [15:05:28] (03PS1) 10DLynch: GutterSidebarEditCheckDialog: Guard against null bounding rects [extensions/VisualEditor] (wmf/1.45.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1175253 [15:09:29] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate thanos-query.discovery.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:09:44] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:16:40] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1188 (T400854)', diff saved to https://phabricator.wikimedia.org/P80566 and previous config saved to /var/cache/conftool/dbconfig/20250803-151639-ladsgroup.json [15:16:43] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [15:16:55] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance [15:17:02] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1197 (T400854)', diff saved to https://phabricator.wikimedia.org/P80567 and previous config saved to /var/cache/conftool/dbconfig/20250803-151702-ladsgroup.json [15:19:32] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1197 (T400854)', diff saved to https://phabricator.wikimedia.org/P80568 and previous config saved to /var/cache/conftool/dbconfig/20250803-151932-ladsgroup.json [15:19:44] RESOLVED: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:34:40] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P80569 and previous config saved to /var/cache/conftool/dbconfig/20250803-153439-ladsgroup.json [15:38:59] FIRING: KubernetesDeploymentUnavailableReplicas: ... [15:38:59] Deployment thumbor-main in thumbor at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=thumbor&var-deployment=thumbor-main - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplica [15:49:48] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P80570 and previous config saved to /var/cache/conftool/dbconfig/20250803-154947-ladsgroup.json [16:04:55] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1197 (T400854)', diff saved to https://phabricator.wikimedia.org/P80571 and previous config saved to /var/cache/conftool/dbconfig/20250803-160455-ladsgroup.json [16:04:58] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [16:05:11] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance [16:06:09] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance [16:06:18] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1229 (T400854)', diff saved to https://phabricator.wikimedia.org/P80572 and previous config saved to /var/cache/conftool/dbconfig/20250803-160616-ladsgroup.json [16:08:47] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1229 (T400854)', diff saved to https://phabricator.wikimedia.org/P80573 and previous config saved to /var/cache/conftool/dbconfig/20250803-160846-ladsgroup.json [16:16:50] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.8 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:18:48] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 37, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:23:54] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P80574 and previous config saved to /var/cache/conftool/dbconfig/20250803-162354-ladsgroup.json [16:39:02] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P80575 and previous config saved to /var/cache/conftool/dbconfig/20250803-163901-ladsgroup.json [16:49:48] (03PS4) 10Krinkle: mediawiki: Fix non-redirecting download.mediawiki.org domain [puppet] - 10https://gerrit.wikimedia.org/r/1174872 (https://phabricator.wikimedia.org/T80222) [16:49:56] (03CR) 10Krinkle: mediawiki: Fix non-redirecting download.mediawiki.org domain (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1174872 (https://phabricator.wikimedia.org/T80222) (owner: 10Krinkle) [16:54:09] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1229 (T400854)', diff saved to https://phabricator.wikimedia.org/P80576 and previous config saved to /var/cache/conftool/dbconfig/20250803-165409-ladsgroup.json [16:54:13] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [16:54:25] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1233.eqiad.wmnet with reason: Maintenance [16:54:33] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1233 (T400854)', diff saved to https://phabricator.wikimedia.org/P80577 and previous config saved to /var/cache/conftool/dbconfig/20250803-165432-ladsgroup.json [16:57:11] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1233 (T400854)', diff saved to https://phabricator.wikimedia.org/P80578 and previous config saved to /var/cache/conftool/dbconfig/20250803-165710-ladsgroup.json [17:11:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on wdqs2025:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [17:12:20] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P80579 and previous config saved to /var/cache/conftool/dbconfig/20250803-171218-ladsgroup.json [17:27:28] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P80580 and previous config saved to /var/cache/conftool/dbconfig/20250803-172727-ladsgroup.json [17:42:35] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1233 (T400854)', diff saved to https://phabricator.wikimedia.org/P80581 and previous config saved to /var/cache/conftool/dbconfig/20250803-174235-ladsgroup.json [17:42:38] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [17:42:48] FIRING: ThumborHighHaproxyErrorRate: Thumbor haproxy error rate for pod thumbor-main-6d8d7547b7-5vrjf - eqiad - https://wikitech.wikimedia.org/wiki/Thumbor - https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor - https://alerts.wikimedia.org/?q=alertname%3DThumborHighHaproxyErrorRate [17:42:50] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance [17:43:47] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1254.eqiad.wmnet with reason: Maintenance [17:43:54] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1254 (T400854)', diff saved to https://phabricator.wikimedia.org/P80582 and previous config saved to /var/cache/conftool/dbconfig/20250803-174354-ladsgroup.json [17:50:34] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1254 (T400854)', diff saved to https://phabricator.wikimedia.org/P80583 and previous config saved to /var/cache/conftool/dbconfig/20250803-175034-ladsgroup.json [17:50:38] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [17:56:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on wdqs2025:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:05:47] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20250803-180541-ladsgroup.json [18:20:54] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P80585 and previous config saved to /var/cache/conftool/dbconfig/20250803-182054-ladsgroup.json [18:36:02] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1254 (T400854)', diff saved to https://phabricator.wikimedia.org/P80586 and previous config saved to /var/cache/conftool/dbconfig/20250803-183601-ladsgroup.json [18:36:05] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [18:36:17] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1259.eqiad.wmnet with reason: Maintenance [18:36:25] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db1259 (T400854)', diff saved to https://phabricator.wikimedia.org/P80587 and previous config saved to /var/cache/conftool/dbconfig/20250803-183624-ladsgroup.json [18:39:04] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1259 (T400854)', diff saved to https://phabricator.wikimedia.org/P80588 and previous config saved to /var/cache/conftool/dbconfig/20250803-183904-ladsgroup.json [18:54:12] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P80589 and previous config saved to /var/cache/conftool/dbconfig/20250803-185411-ladsgroup.json [19:09:19] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P80590 and previous config saved to /var/cache/conftool/dbconfig/20250803-190919-ladsgroup.json [19:09:29] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate thanos-query.discovery.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [19:12:48] FIRING: [2x] ThumborHighHaproxyErrorRate: Thumbor haproxy error rate for pod thumbor-main-6d8d7547b7-5vrjf - eqiad - https://wikitech.wikimedia.org/wiki/Thumbor - https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor - https://alerts.wikimedia.org/?q=alertname%3DThumborHighHaproxyErrorRate [19:24:27] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1259 (T400854)', diff saved to https://phabricator.wikimedia.org/P80591 and previous config saved to /var/cache/conftool/dbconfig/20250803-192426-ladsgroup.json [19:24:30] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [19:24:42] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [19:25:44] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance [19:25:52] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2148 (T400854)', diff saved to https://phabricator.wikimedia.org/P80592 and previous config saved to /var/cache/conftool/dbconfig/20250803-192551-ladsgroup.json [19:28:50] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2148 (T400854)', diff saved to https://phabricator.wikimedia.org/P80593 and previous config saved to /var/cache/conftool/dbconfig/20250803-192846-ladsgroup.json [19:38:59] FIRING: KubernetesDeploymentUnavailableReplicas: ... [19:38:59] Deployment thumbor-main in thumbor at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=thumbor&var-deployment=thumbor-main - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplica [19:43:57] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P80594 and previous config saved to /var/cache/conftool/dbconfig/20250803-194357-ladsgroup.json [19:59:05] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P80595 and previous config saved to /var/cache/conftool/dbconfig/20250803-195904-ladsgroup.json [20:02:48] FIRING: [2x] ThumborHighHaproxyErrorRate: Thumbor haproxy error rate for pod thumbor-main-6d8d7547b7-5vrjf - eqiad - https://wikitech.wikimedia.org/wiki/Thumbor - https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor - https://alerts.wikimedia.org/?q=alertname%3DThumborHighHaproxyErrorRate [20:14:12] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2148 (T400854)', diff saved to https://phabricator.wikimedia.org/P80596 and previous config saved to /var/cache/conftool/dbconfig/20250803-201412-ladsgroup.json [20:14:15] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [20:14:28] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance [20:14:36] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2175 (T400854)', diff saved to https://phabricator.wikimedia.org/P80597 and previous config saved to /var/cache/conftool/dbconfig/20250803-201435-ladsgroup.json [20:17:31] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2175 (T400854)', diff saved to https://phabricator.wikimedia.org/P80598 and previous config saved to /var/cache/conftool/dbconfig/20250803-201730-ladsgroup.json [20:32:40] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P80599 and previous config saved to /var/cache/conftool/dbconfig/20250803-203238-ladsgroup.json [20:47:48] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P80600 and previous config saved to /var/cache/conftool/dbconfig/20250803-204747-ladsgroup.json [21:02:56] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2175 (T400854)', diff saved to https://phabricator.wikimedia.org/P80601 and previous config saved to /var/cache/conftool/dbconfig/20250803-210255-ladsgroup.json [21:02:59] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [21:03:11] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2189.codfw.wmnet with reason: Maintenance [21:03:19] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2189 (T400854)', diff saved to https://phabricator.wikimedia.org/P80602 and previous config saved to /var/cache/conftool/dbconfig/20250803-210318-ladsgroup.json [21:06:09] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2189 (T400854)', diff saved to https://phabricator.wikimedia.org/P80603 and previous config saved to /var/cache/conftool/dbconfig/20250803-210608-ladsgroup.json [21:21:16] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P80604 and previous config saved to /var/cache/conftool/dbconfig/20250803-212116-ladsgroup.json [21:27:48] RESOLVED: ThumborHighHaproxyErrorRate: Thumbor haproxy error rate for pod thumbor-main-6d8d7547b7-nzbvt - eqiad - https://wikitech.wikimedia.org/wiki/Thumbor - https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor - https://alerts.wikimedia.org/?q=alertname%3DThumborHighHaproxyErrorRate [21:36:24] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P80605 and previous config saved to /var/cache/conftool/dbconfig/20250803-213623-ladsgroup.json [21:51:32] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2189 (T400854)', diff saved to https://phabricator.wikimedia.org/P80606 and previous config saved to /var/cache/conftool/dbconfig/20250803-215131-ladsgroup.json [21:51:35] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [21:51:47] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance [21:53:28] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2207.codfw.wmnet with reason: Maintenance [21:53:35] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2207 (T400854)', diff saved to https://phabricator.wikimedia.org/P80607 and previous config saved to /var/cache/conftool/dbconfig/20250803-215335-ladsgroup.json [21:56:25] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2207 (T400854)', diff saved to https://phabricator.wikimedia.org/P80608 and previous config saved to /var/cache/conftool/dbconfig/20250803-215625-ladsgroup.json [22:11:33] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P80609 and previous config saved to /var/cache/conftool/dbconfig/20250803-221132-ladsgroup.json [22:26:40] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P80610 and previous config saved to /var/cache/conftool/dbconfig/20250803-222640-ladsgroup.json [22:41:48] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2207 (T400854)', diff saved to https://phabricator.wikimedia.org/P80611 and previous config saved to /var/cache/conftool/dbconfig/20250803-224147-ladsgroup.json [22:41:51] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [22:41:52] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2225.codfw.wmnet with reason: Maintenance [22:42:00] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2225 (T400854)', diff saved to https://phabricator.wikimedia.org/P80612 and previous config saved to /var/cache/conftool/dbconfig/20250803-224159-ladsgroup.json [22:44:51] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2225 (T400854)', diff saved to https://phabricator.wikimedia.org/P80613 and previous config saved to /var/cache/conftool/dbconfig/20250803-224450-ladsgroup.json [22:59:58] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P80614 and previous config saved to /var/cache/conftool/dbconfig/20250803-225957-ladsgroup.json [23:09:29] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate thanos-query.discovery.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:15:06] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P80615 and previous config saved to /var/cache/conftool/dbconfig/20250803-231505-ladsgroup.json [23:30:15] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2225 (T400854)', diff saved to https://phabricator.wikimedia.org/P80616 and previous config saved to /var/cache/conftool/dbconfig/20250803-233013-ladsgroup.json [23:30:18] T400854: Add rc_source_name_timestamp index to recentchanges table in wmf production - https://phabricator.wikimedia.org/T400854 [23:30:30] !log ladsgroup@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2226.codfw.wmnet with reason: Maintenance [23:30:39] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2226 (T400854)', diff saved to https://phabricator.wikimedia.org/P80617 and previous config saved to /var/cache/conftool/dbconfig/20250803-233037-ladsgroup.json [23:33:22] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2226 (T400854)', diff saved to https://phabricator.wikimedia.org/P80618 and previous config saved to /var/cache/conftool/dbconfig/20250803-233322-ladsgroup.json [23:38:09] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1175270 [23:38:09] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1175270 (owner: 10TrainBranchBot) [23:38:59] FIRING: KubernetesDeploymentUnavailableReplicas: ... [23:38:59] Deployment thumbor-main in thumbor at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=thumbor&var-deployment=thumbor-main - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplica [23:48:30] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P80619 and previous config saved to /var/cache/conftool/dbconfig/20250803-234829-ladsgroup.json [23:51:23] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1175270 (owner: 10TrainBranchBot)