[00:38:57] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1003858 [00:39:03] (03CR) 10TrainBranchBot: [C: 03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1003858 (owner: 10TrainBranchBot) [01:00:28] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1003858 (owner: 10TrainBranchBot) [02:08:35] (PuppetZeroResources) firing: Puppet has failed generate resources on ncmonitor1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:09:25] (SystemdUnitFailed) firing: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:10:14] PROBLEM - Check unit status of httpbb_hourly_appserver on cumin1002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_hourly_appserver https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [02:38:36] (JobUnavailable) firing: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:08:36] (JobUnavailable) resolved: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:09:25] (SystemdUnitFailed) resolved: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:10:14] RECOVERY - Check unit status of httpbb_hourly_appserver on cumin1002 is OK: OK: Status of the systemd unit httpbb_hourly_appserver https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [03:55:22] !log ladsgroup@cumin1002 START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance [03:55:36] !log ladsgroup@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance [03:55:42] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2140 (T352010)', diff saved to https://phabricator.wikimedia.org/P56930 and previous config saved to /var/cache/conftool/dbconfig/20240218-035542-ladsgroup.json [03:55:47] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010 [06:08:35] (PuppetZeroResources) firing: Puppet has failed generate resources on ncmonitor1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:19:54] PROBLEM - snapshot of x1 in codfw on backupmon1001 is CRITICAL: snapshot for x1 at codfw (db2097) taken more than 3 days ago: Most recent backup 2024-02-15 06:01:50 https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [07:04:21] (PoolcounterFullQueues) firing: Full queues for poolcounter1004:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [07:09:21] (PoolcounterFullQueues) resolved: Full queues for poolcounter1004:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [08:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240218T0800) [08:47:42] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2140 (T352010)', diff saved to https://phabricator.wikimedia.org/P56931 and previous config saved to /var/cache/conftool/dbconfig/20240218-084741-ladsgroup.json [08:47:47] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010 [09:02:48] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P56932 and previous config saved to /var/cache/conftool/dbconfig/20240218-090248-ladsgroup.json [09:17:55] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P56933 and previous config saved to /var/cache/conftool/dbconfig/20240218-091754-ladsgroup.json [09:31:42] PROBLEM - mailman list info on lists1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [09:32:04] PROBLEM - mailman archives on lists1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [09:33:02] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2140 (T352010)', diff saved to https://phabricator.wikimedia.org/P56934 and previous config saved to /var/cache/conftool/dbconfig/20240218-093301-ladsgroup.json [09:33:04] !log ladsgroup@cumin1002 START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance [09:33:06] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010 [09:33:17] !log ladsgroup@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance [09:33:24] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P56935 and previous config saved to /var/cache/conftool/dbconfig/20240218-093323-ladsgroup.json [09:34:44] RECOVERY - mailman list info on lists1001 is OK: HTTP OK: HTTP/1.1 200 OK - 8572 bytes in 9.947 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [09:34:56] RECOVERY - mailman archives on lists1001 is OK: HTTP OK: HTTP/1.1 200 OK - 51451 bytes in 0.071 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [10:08:35] (PuppetZeroResources) firing: Puppet has failed generate resources on ncmonitor1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:33:55] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P56936 and previous config saved to /var/cache/conftool/dbconfig/20240218-103355-ladsgroup.json [10:34:04] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010 [10:49:02] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P56937 and previous config saved to /var/cache/conftool/dbconfig/20240218-104901-ladsgroup.json [11:04:08] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P56938 and previous config saved to /var/cache/conftool/dbconfig/20240218-110408-ladsgroup.json [11:15:32] 10ops-codfw, 10serviceops: Degraded RAID on mw2442 - https://phabricator.wikimedia.org/T357380#9553521 (10JMeybohm) [11:15:42] 10ops-codfw, 10serviceops: Degraded RAID on mw2442 - https://phabricator.wikimedia.org/T357380#9553520 (10JMeybohm) Sorry for the late reply. I'm not sure what you're asking thb. As I understand it the disk most likely broke, so the "replace the PD" option would be the way to go here. From the kern.log: ` Feb... [11:19:15] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P56939 and previous config saved to /var/cache/conftool/dbconfig/20240218-111915-ladsgroup.json [11:19:18] !log ladsgroup@cumin1002 START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance [11:19:20] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010 [11:19:32] !log ladsgroup@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance [11:19:34] !log ladsgroup@cumin1002 START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [11:19:48] !log ladsgroup@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [11:19:54] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2155 (T352010)', diff saved to https://phabricator.wikimedia.org/P56940 and previous config saved to /var/cache/conftool/dbconfig/20240218-111954-ladsgroup.json [13:42:06] (MediaWikiEditFailures) firing: (2) Elevated MediaWiki edit failures (session_loss) for cluster appserver - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [13:47:06] (MediaWikiEditFailures) resolved: (2) Elevated MediaWiki edit failures (session_loss) for cluster appserver - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [14:08:35] (PuppetZeroResources) firing: Puppet has failed generate resources on ncmonitor1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:38:36] (JobUnavailable) firing: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:58:36] (JobUnavailable) resolved: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:27:36] (03PS2) 10Reedy: Fix casing of Mediawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/993738 [16:29:42] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2155 (T352010)', diff saved to https://phabricator.wikimedia.org/P56941 and previous config saved to /var/cache/conftool/dbconfig/20240218-162942-ladsgroup.json [16:29:47] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010 [16:44:49] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P56942 and previous config saved to /var/cache/conftool/dbconfig/20240218-164448-ladsgroup.json [16:59:56] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P56943 and previous config saved to /var/cache/conftool/dbconfig/20240218-165955-ladsgroup.json [17:15:02] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2155 (T352010)', diff saved to https://phabricator.wikimedia.org/P56944 and previous config saved to /var/cache/conftool/dbconfig/20240218-171502-ladsgroup.json [17:15:04] !log ladsgroup@cumin1002 START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance [17:15:08] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010 [17:15:19] !log ladsgroup@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance [17:15:26] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Depooling db2172 (T352010)', diff saved to https://phabricator.wikimedia.org/P56945 and previous config saved to /var/cache/conftool/dbconfig/20240218-171526-ladsgroup.json [17:19:23] (03PS5) 10ArielGlenn: Remove incorrect comments from the dumptextpass backfill script [dumps] - 10https://gerrit.wikimedia.org/r/947336 (https://phabricator.wikimedia.org/T343882) [17:19:25] (03PS1) 10ArielGlenn: move usage message into its own method in monitor.py [dumps] - 10https://gerrit.wikimedia.org/r/1004328 [17:19:27] (03PS1) 10ArielGlenn: pylint cleanup for monitor.py [dumps] - 10https://gerrit.wikimedia.org/r/1004329 (https://phabricator.wikimedia.org/T343882) [17:19:29] (03PS1) 10ArielGlenn: make monitor.py use proper options [dumps] - 10https://gerrit.wikimedia.org/r/1004330 [17:19:31] (03PS1) 10ArielGlenn: allow monitor script to skip writing top level index html [dumps] - 10https://gerrit.wikimedia.org/r/1004331 [17:19:33] (03PS1) 10ArielGlenn: Dumps: have page content backfill script write files into temp dir [dumps] - 10https://gerrit.wikimedia.org/r/1004332 (https://phabricator.wikimedia.org/T252396) [17:19:35] (03CR) 10CI reject: [V: 04-1] move usage message into its own method in monitor.py [dumps] - 10https://gerrit.wikimedia.org/r/1004328 (owner: 10ArielGlenn) [17:19:37] (03CR) 10CI reject: [V: 04-1] pylint cleanup for monitor.py [dumps] - 10https://gerrit.wikimedia.org/r/1004329 (https://phabricator.wikimedia.org/T343882) (owner: 10ArielGlenn) [17:19:39] (03CR) 10CI reject: [V: 04-1] make monitor.py use proper options [dumps] - 10https://gerrit.wikimedia.org/r/1004330 (owner: 10ArielGlenn) [17:19:41] (03CR) 10CI reject: [V: 04-1] Dumps: have page content backfill script write files into temp dir [dumps] - 10https://gerrit.wikimedia.org/r/1004332 (https://phabricator.wikimedia.org/T252396) (owner: 10ArielGlenn) [17:36:24] (03PS6) 10ArielGlenn: Remove incorrect comments from the dumptextpass backfill script [dumps] - 10https://gerrit.wikimedia.org/r/947336 (https://phabricator.wikimedia.org/T343882) [17:36:26] (03PS2) 10ArielGlenn: Dumps: have page content backfill script write files into temp dir [dumps] - 10https://gerrit.wikimedia.org/r/1004332 (https://phabricator.wikimedia.org/T252396) [17:36:31] (03CR) 10CI reject: [V: 04-1] Dumps: have page content backfill script write files into temp dir [dumps] - 10https://gerrit.wikimedia.org/r/1004332 (https://phabricator.wikimedia.org/T252396) (owner: 10ArielGlenn) [18:08:36] (PuppetZeroResources) firing: Puppet has failed generate resources on ncmonitor1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:13:36] (JobUnavailable) firing: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:18:36] (JobUnavailable) resolved: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:23:36] (JobUnavailable) firing: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:28:36] (JobUnavailable) resolved: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:54:58] PROBLEM - BGP status on cr1-esams is CRITICAL: BGP CRITICAL - AS6939/IPv4: Connect - HE https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status [20:03:04] PROBLEM - Host mr1-esams.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [20:17:32] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw IPv6 is CRITICAL: CRITICAL - failed 138 probes of 737 (alerts on 90) - https://atlas.ripe.net/measurements/32390541/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [20:19:42] PROBLEM - IPv6 ping to eqsin on ripe-atlas-eqsin IPv6 is CRITICAL: CRITICAL - failed 106 probes of 735 (alerts on 90) - https://atlas.ripe.net/measurements/11645088/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [20:20:02] PROBLEM - IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 is CRITICAL: CRITICAL - failed 147 probes of 731 (alerts on 90) - https://atlas.ripe.net/measurements/1790947/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [20:21:40] PROBLEM - IPv6 ping to esams on ripe-atlas-esams IPv6 is CRITICAL: CRITICAL - failed 112 probes of 737 (alerts on 90) - https://atlas.ripe.net/measurements/59935539/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [20:39:42] RECOVERY - IPv6 ping to eqsin on ripe-atlas-eqsin IPv6 is OK: OK - failed 48 probes of 735 (alerts on 90) - https://atlas.ripe.net/measurements/11645088/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [20:46:40] RECOVERY - IPv6 ping to esams on ripe-atlas-esams IPv6 is OK: OK - failed 82 probes of 737 (alerts on 90) - https://atlas.ripe.net/measurements/59935539/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [20:54:54] RECOVERY - Host mr1-esams.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 88.00 ms [20:57:32] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw IPv6 is OK: OK - failed 41 probes of 737 (alerts on 90) - https://atlas.ripe.net/measurements/32390541/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [21:00:02] RECOVERY - IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 is OK: OK - failed 39 probes of 731 (alerts on 90) - https://atlas.ripe.net/measurements/1790947/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [22:08:35] (PuppetZeroResources) firing: Puppet has failed generate resources on ncmonitor1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:25:43] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2172 (T352010)', diff saved to https://phabricator.wikimedia.org/P56947 and previous config saved to /var/cache/conftool/dbconfig/20240218-222543-ladsgroup.json [22:25:49] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010 [22:40:50] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P56948 and previous config saved to /var/cache/conftool/dbconfig/20240218-224049-ladsgroup.json [22:55:57] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P56949 and previous config saved to /var/cache/conftool/dbconfig/20240218-225556-ladsgroup.json [23:11:03] !log ladsgroup@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2172 (T352010)', diff saved to https://phabricator.wikimedia.org/P56950 and previous config saved to /var/cache/conftool/dbconfig/20240218-231102-ladsgroup.json [23:11:10] T352010: Gradually drop old pagelinks columns - https://phabricator.wikimedia.org/T352010