[00:06:21] 10Cloud-VPS, 10cloud-services-team: Rescue DBapp trove instance in glamwikidashboard project - https://phabricator.wikimedia.org/T355138 (10YonatanWMIL) I can stop the DB from growing by stopping the daily data insertion. I just didn't expect it to fill up so fast after doubling the disk space. Going to have t... [00:15:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:16:19] 10Grid-Engine-to-K8s-Migration: Migrate wordpile from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320184 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:16:22] 10Grid-Engine-to-K8s-Migration: Migrate title-search from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320087 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:16:24] 10Grid-Engine-to-K8s-Migration: Migrate shrinitools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320037 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:16:26] 10Grid-Engine-to-K8s-Migration: Migrate render from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320001 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:16:28] 10Grid-Engine-to-K8s-Migration: Migrate noclaims from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319927 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:16:30] 10Grid-Engine-to-K8s-Migration: Migrate mgp-cewbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319890 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:16:33] 10Grid-Engine-to-K8s-Migration: Migrate map-search from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319875 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:16:35] 10Grid-Engine-to-K8s-Migration: Migrate labelimgohs from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319850 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:16:37] 10Grid-Engine-to-K8s-Migration: Migrate laaknortools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319849 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:16:39] 10Grid-Engine-to-K8s-Migration: Migrate krdbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319847 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:16:41] 10Grid-Engine-to-K8s-Migration: Migrate kolbert from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319846 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:16:43] 10Grid-Engine-to-K8s-Migration: Migrate khanamalumat from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319842 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:16:45] 10Grid-Engine-to-K8s-Migration: Migrate kasparbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319841 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:16:47] 10Grid-Engine-to-K8s-Migration: Migrate karsilayici from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319840 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:16:49] 10Grid-Engine-to-K8s-Migration: Migrate kanzatimagerequests from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319838 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it o... [00:16:51] 10Grid-Engine-to-K8s-Migration: Migrate kanzatcopyvio from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319837 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:16:53] 10Grid-Engine-to-K8s-Migration: Migrate kaleem-bot-i from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319835 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:16:55] 10Grid-Engine-to-K8s-Migration: Migrate kaleem-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319834 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:16:57] 10Grid-Engine-to-K8s-Migration: Migrate jpxg-test from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319833 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:16:59] 10Grid-Engine-to-K8s-Migration: Migrate jogobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319831 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:02] 10Grid-Engine-to-K8s-Migration: Migrate jitrixis-test from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319830 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:17:04] 10Grid-Engine-to-K8s-Migration: Migrate jawi from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319827 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, pl... [00:17:06] 10Grid-Engine-to-K8s-Migration: Migrate itemfinder from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319821 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:17:08] 10Grid-Engine-to-K8s-Migration: Migrate isbn from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319819 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, pl... [00:17:10] 10Grid-Engine-to-K8s-Migration: Migrate ipp from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319817 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, ple... [00:17:12] 10Grid-Engine-to-K8s-Migration: Migrate ipinfo from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319815 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:14] 10Grid-Engine-to-K8s-Migration: Migrate integraality from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319813 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:17:16] 10Grid-Engine-to-K8s-Migration: Migrate imagery from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319807 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:18] 10Grid-Engine-to-K8s-Migration: Migrate igloo from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319806 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, p... [00:17:20] 10Grid-Engine-to-K8s-Migration: Migrate ideasbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319805 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:17:22] 10Grid-Engine-to-K8s-Migration: Migrate iacrop from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319804 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:26] 10Grid-Engine-to-K8s-Migration: Migrate huntleybots from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319802 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:17:28] 10Grid-Engine-to-K8s-Migration: Migrate hunsbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319801 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:30] 10Grid-Engine-to-K8s-Migration: Migrate htools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319796 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:32] 10Grid-Engine-to-K8s-Migration: Migrate hsfbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319795 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:34] 10Grid-Engine-to-K8s-Migration: Migrate hrwiki from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319794 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:36] 10Grid-Engine-to-K8s-Migration: Migrate hostbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319792 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:17:38] 10Grid-Engine-to-K8s-Migration: Migrate honeypot95 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319791 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:17:40] 10Grid-Engine-to-K8s-Migration: Migrate himo from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319789 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, pl... [00:17:42] 10Grid-Engine-to-K8s-Migration: Migrate herculebot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319786 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:17:44] 10Grid-Engine-to-K8s-Migration: Migrate hazard-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319785 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:17:46] 10Grid-Engine-to-K8s-Migration: Migrate hashtagwatcher from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319784 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off th... [00:17:48] 10Grid-Engine-to-K8s-Migration: Migrate hamishbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319783 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:17:50] 10Grid-Engine-to-K8s-Migration: Migrate grapedog from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319780 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:17:52] 10Grid-Engine-to-K8s-Migration: Migrate gorlingor from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319777 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:17:54] 10Grid-Engine-to-K8s-Migration: Migrate gnubotmarcoo from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319776 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:17:56] 10Grid-Engine-to-K8s-Migration: Migrate gns from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319775 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, ple... [00:17:58] 10Grid-Engine-to-K8s-Migration: Migrate globalusagecount from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319774 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off... [00:18:00] 10Grid-Engine-to-K8s-Migration: Migrate glamify from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319772 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:18:02] 10Grid-Engine-to-K8s-Migration: Migrate germancon-mobile from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319769 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off... [00:18:04] 10Grid-Engine-to-K8s-Migration: Migrate gerakitools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319768 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:18:06] 10Grid-Engine-to-K8s-Migration: Migrate gerakibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319767 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:18:08] 10Grid-Engine-to-K8s-Migration: Migrate geophotoreq from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319766 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:18:10] 10Grid-Engine-to-K8s-Migration: Migrate geograph from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319765 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:18:12] 10Grid-Engine-to-K8s-Migration: Migrate gendergapdashboard from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319764 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it of... [00:18:14] 10Grid-Engine-to-K8s-Migration: Migrate g13bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319761 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:18:16] 10Grid-Engine-to-K8s-Migration: Migrate fvcbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319760 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:18:18] 10Grid-Engine-to-K8s-Migration: Migrate furutani from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319759 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:18:20] 10Grid-Engine-to-K8s-Migration: Migrate fshbibbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319757 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:18:22] 10Grid-Engine-to-K8s-Migration: Migrate fscbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319756 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:18:24] 10Grid-Engine-to-K8s-Migration: Migrate friskobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319755 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:18:26] 10Grid-Engine-to-K8s-Migration: Migrate freddy2001 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319754 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:18:28] 10Grid-Engine-to-K8s-Migration: Migrate fr-wikiversity-ns from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319753 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off... [00:18:30] 10Grid-Engine-to-K8s-Migration: Migrate footygen from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319749 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:18:32] 10Grid-Engine-to-K8s-Migration: Migrate flossbrowser from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319747 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:18:34] 10Grid-Engine-to-K8s-Migration: Migrate fireflybot2 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319742 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:18:36] 10Grid-Engine-to-K8s-Migration: Migrate family from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319739 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:18:38] 10Grid-Engine-to-K8s-Migration: Migrate ext-lnk-discover from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319736 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off... [00:18:40] 10Grid-Engine-to-K8s-Migration: Migrate expose-data from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319735 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:18:42] 10Grid-Engine-to-K8s-Migration: Migrate chie-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319625 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:18:44] 10Grid-Engine-to-K8s-Migration: Migrate cewbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319622 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:18:47] 10Grid-Engine-to-K8s-Migration: Migrate botwikiawk from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319607 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:18:49] 10Grid-Engine-to-K8s-Migration: Migrate botleo from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319602 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:18:51] 10Grid-Engine-to-K8s-Migration: Migrate blogconverter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319597 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the... [00:18:53] 10Grid-Engine-to-K8s-Migration: Migrate blahma from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319591 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:18:55] 10Grid-Engine-to-K8s-Migration: Migrate bibleversefinder2 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319589 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off... [00:18:57] 10Grid-Engine-to-K8s-Migration: Migrate bibleversefinder from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319588 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off... [00:18:59] 10Grid-Engine-to-K8s-Migration: Migrate betacommand-dev from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319587 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off t... [00:19:01] 10Grid-Engine-to-K8s-Migration: Migrate bene from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319586 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, pl... [00:19:03] 10Grid-Engine-to-K8s-Migration: Migrate bays from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319585 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, pl... [00:19:05] 10Grid-Engine-to-K8s-Migration: Migrate basebot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319583 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:19:07] 10Grid-Engine-to-K8s-Migration: Migrate avicbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319582 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:19:09] 10Grid-Engine-to-K8s-Migration: Migrate autopromote-status from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319581 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it of... [00:19:11] 10Grid-Engine-to-K8s-Migration: Migrate ato from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319580 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, ple... [00:19:13] 10Grid-Engine-to-K8s-Migration: Migrate ashbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319576 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:19:15] 10Grid-Engine-to-K8s-Migration: Migrate ash-dev from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319575 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid,... [00:19:17] 10Grid-Engine-to-K8s-Migration: Migrate as-info-dev from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319574 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:19:19] 10Grid-Engine-to-K8s-Migration: Migrate articleplaceholderwiki from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319572 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating i... [00:19:21] 10Grid-Engine-to-K8s-Migration: Migrate artemisia from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319571 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:19:23] 10Grid-Engine-to-K8s-Migration: Migrate arnaubot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319570 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid... [00:19:25] 10Grid-Engine-to-K8s-Migration: Migrate archive-things-4 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319566 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off... [00:19:27] 10Grid-Engine-to-K8s-Migration: Migrate archive-things-1 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319565 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off... [00:19:29] 10Grid-Engine-to-K8s-Migration: Migrate archive-things from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319564 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off th... [00:19:31] 10Grid-Engine-to-K8s-Migration: Migrate antigng-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319560 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the g... [00:19:33] 10Grid-Engine-to-K8s-Migration: Migrate ancestors2 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319554 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:19:35] 10Grid-Engine-to-K8s-Migration: Migrate analytics from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319553 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gri... [00:19:37] 10Grid-Engine-to-K8s-Migration: Migrate analytalks from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319552 (10komla) This tool has been disabled from running on the Grid. If you are the maintainer and you want this re-enabled so that you can work on migrating it off the gr... [00:20:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:36:38] (ProbeDown) firing: (2) Service toolsbeta-test-k8s-haproxy-3:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:00:58] (03PS1) 10Amire80: WIP Merge lego footer messages into one [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003574 (https://phabricator.wikimedia.org/T355011) [01:01:05] (03CR) 10CI reject: [V: 04-1] WIP Merge lego footer messages into one [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003574 (https://phabricator.wikimedia.org/T355011) (owner: 10Amire80) [01:01:53] (03PS2) 10Amire80: WIP Merge lego footer messages into one [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003574 (https://phabricator.wikimedia.org/T355011) [01:02:01] (03CR) 10CI reject: [V: 04-1] WIP Merge lego footer messages into one [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003574 (https://phabricator.wikimedia.org/T355011) (owner: 10Amire80) [01:02:59] (03PS3) 10Amire80: WIP Merge lego footer messages into one [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003574 (https://phabricator.wikimedia.org/T355011) [01:18:44] 10Toolforge (Toolforge iteration 05), 10Toolforge Build Service, 10Patch-For-Review: [tbs] cleanup robot account related code - https://phabricator.wikimedia.org/T352763 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/34 [builds-build... [01:19:24] 10Toolforge (Toolforge iteration 05), 10Toolforge Build Service, 10Patch-For-Review: [tbs] cleanup robot account related code - https://phabricator.wikimedia.org/T352763 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/78 [builds-api] use... [01:19:51] 10Toolforge (Toolforge iteration 05), 10Toolforge Build Service, 10Patch-For-Review: [tbs] cleanup robot account related code - https://phabricator.wikimedia.org/T352763 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/196 [toolforge... [01:24:49] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:08:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1055.eqiad.wmnet}' [03:08:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}' [03:21:36] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}' [03:23:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}' [03:30:15] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1055.eqiad.wmnet}' [03:30:50] (NeutronAgentDown) firing: Neutron neutron-linuxbridge-agent on cloudvirt1055 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [03:30:55] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}' [03:39:33] (ProbeDown) firing: (2) Service toolsbeta-test-k8s-haproxy-3:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:43:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}' [03:43:30] 10Grid-Engine-to-K8s-Migration: Migrate bawolff from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319584 (10Bawolff) 05Open→03Resolved I ended up just removing that part of the tool. It largely wasn't working for other reasons. [03:47:12] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}' [03:47:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1052.eqiad.wmnet}' [04:02:20] (NeutronAgentDown) resolved: Neutron neutron-linuxbridge-agent on cloudvirt1055 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:03:34] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1053.eqiad.wmnet}' [04:05:50] (NeutronAgentDown) firing: Neutron neutron-linuxbridge-agent on cloudvirt1053 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:09:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1052.eqiad.wmnet}' [04:10:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1050.eqiad.wmnet}' [04:10:25] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1051.eqiad.wmnet}' [04:10:50] (NeutronAgentDown) firing: (2) Neutron neutron-linuxbridge-agent on cloudvirt1052 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:30:46] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1050.eqiad.wmnet}' [04:30:50] (NeutronAgentDown) firing: Neutron neutron-linuxbridge-agent on cloudvirt1050 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:32:20] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1051.eqiad.wmnet}' [04:33:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1048.eqiad.wmnet}' [04:33:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}' [04:35:50] (NeutronAgentDown) firing: (2) Neutron neutron-linuxbridge-agent on cloudvirt1050 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:40:50] (NeutronAgentDown) resolved: (2) Neutron neutron-linuxbridge-agent on cloudvirt1052 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:44:11] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}' [04:45:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}' [04:48:06] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1048.eqiad.wmnet}' [04:50:09] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}' [04:50:50] (NeutronAgentDown) firing: Neutron neutron-linuxbridge-agent on cloudvirt1048 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:51:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1046.eqiad.wmnet[B}' [04:52:00] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1046.eqiad.wmnet[B}' [04:52:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1047.eqiad.wmnet}' [05:00:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1046.eqiad.wmnet}' [05:05:50] (NeutronAgentDown) firing: (4) Neutron neutron-linuxbridge-agent on cloudvirt1048 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [05:10:50] (NeutronAgentDown) resolved: (3) Neutron neutron-linuxbridge-agent on cloudvirt1048 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [05:15:06] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1046.eqiad.wmnet}' [05:15:34] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1045.eqiad.wmnet}' [05:18:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1047.eqiad.wmnet}' [05:19:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1044.eqiad.wmnet}' [05:25:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [05:41:20] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1045.eqiad.wmnet}' [05:41:50] (NeutronAgentDown) firing: Neutron neutron-linuxbridge-agent on cloudvirt1045 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [05:51:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1044.eqiad.wmnet}' [05:52:44] (InterfaceSpeedError) firing: brq7425e328-56 on cloudvirt1044:9100 has the wrong speed: 1.25e+06. - https://wikitech.wikimedia.org/wiki/Monitoring/check_eth - https://grafana.wikimedia.org/d/000000562 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceSpeedError [05:52:49] 10cloud-services-team: InterfaceSpeedError brq7425e328-56 on cloudvirt1044:9100 has the wrong speed: 1.25e+06. - https://phabricator.wikimedia.org/T357604 (10phaultfinder) [05:57:44] (InterfaceSpeedError) resolved: brq7425e328-56 on cloudvirt1044:9100 has the wrong speed: 1.25e+06. - https://wikitech.wikimedia.org/wiki/Monitoring/check_eth - https://grafana.wikimedia.org/d/000000562 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceSpeedError [06:13:20] (NeutronAgentDown) resolved: Neutron neutron-linuxbridge-agent on cloudvirt1045 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [06:36:41] (ProbeDown) firing: (2) Service toolsbeta-test-k8s-haproxy-3:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:41:41] (CloudVPSDesignateLeaks) firing: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:46:41] (CloudVPSDesignateLeaks) firing: (2) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:23:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/ 5.487% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:53:30] 10Grid-Engine-to-K8s-Migration: Migrate smallem from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320048 (10Klein) 05Open→03Resolved As I'm writing this, all the 4 tasks have started and 3 of them have completed successfully (the last one takes more than one day to comple... [09:00:19] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4), 10Epic, 10Goal, 10User-aborrero: openstack eqiad1: introduce cloud-private and cloudlb - https://phabricator.wikimedia.org/T341060 (10aborrero) [09:25:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:36:41] (ProbeDown) firing: (2) Service toolsbeta-test-k8s-haproxy-3:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:40:25] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster [10:44:45] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster [10:45:47] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 [10:46:08] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 [10:46:45] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster [10:53:06] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster [10:55:56] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 [10:56:15] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 [10:57:55] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 [10:57:58] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 [10:58:16] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster [11:06:17] !log taavi@cloudcumin1001 toolsbeta Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster [11:06:17] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster [11:08:12] (CloudVPSDesignateLeaks) resolved: (2) Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:08:27] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 [11:08:36] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5 [11:09:49] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 [11:09:49] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5 [11:11:44] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 [11:12:17] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5 [11:24:54] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster [11:29:19] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster [11:29:44] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7 [11:30:05] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7 [11:30:11] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster [11:33:53] 10Toolforge, 10cloud-services-team, 10Patch-For-Review: Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665 (10taavi) [11:34:13] 10Toolforge, 10cloud-services-team: Upgrade Toolforge Kubernetes to version 1.24 - https://phabricator.wikimedia.org/T307651 (10taavi) [11:34:15] 10Toolforge, 10cloud-services-team, 10Patch-For-Review: Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665 (10taavi) [11:35:06] 10Toolforge, 10cloud-services-team, 10Patch-For-Review: Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665 (10taavi) a:03taavi [11:37:56] !log taavi@cloudcumin1001 tools Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster [11:37:57] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster [11:41:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:46:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:48:37] 10Tool-toolwatch, 10Technical-Tool-Request: Tool Request: ToolForge Health Dashboard Tool (ToolWatch) - https://phabricator.wikimedia.org/T341379 (10fnegri) Given that https://tool-watch.toolforge.org/ is up and running, and there is an associated Phab tag #tool-toolwatch, can we mark this task as Resolved, an... [11:52:02] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' (T319184) [12:05:25] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' (T319184) [12:05:31] T319184: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 [12:08:20] 10Toolforge, 10cloud-services-team: Migrate remaining tools off Gridengine - https://phabricator.wikimedia.org/T313405 (10taavi) [12:08:22] 10Toolforge, 10cloud-services-team: Make Grid Engine tooling emit deprecation warnings - https://phabricator.wikimedia.org/T316124 (10taavi) 05Open→03Declined We did not end up implementing this. [12:11:44] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE, 10netops, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1031.eqiad.wmnet with OS bookworm [12:18:22] (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] "LGTM." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1003606 (owner: 10Majavah) [12:18:57] (03CR) 10Majavah: [C: 03+2] toolforge: k8s: depool_and_remove_node: Update hiera data [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1003606 (owner: 10Majavah) [12:22:17] (03Merged) 10jenkins-bot: toolforge: k8s: depool_and_remove_node: Update hiera data [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1003606 (owner: 10Majavah) [12:23:50] (DiskSpace) firing: Disk space cloudbackup1004:9100:/ 5.805% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:28:57] 10Data-Services: [toolsdb] Replica is frequently lagging behind the primary - https://phabricator.wikimedia.org/T357624 (10fnegri) [12:29:07] 10Data-Services: [toolsdb] Replica is frequently lagging behind the primary - https://phabricator.wikimedia.org/T357624 (10fnegri) p:05Triage→03Medium [12:29:23] 10Data-Services: [toolsdb] Replica is frequently lagging behind the primary - https://phabricator.wikimedia.org/T357624 (10fnegri) [12:33:17] 10Data-Services: [toolsdb] Replica is frequently lagging behind the primary - https://phabricator.wikimedia.org/T357624 (10fnegri) [12:33:26] (03PS1) 10Amire80: Fix a lego message [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003743 [12:33:44] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75 [12:34:25] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75 [12:34:42] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster [12:36:41] (ProbeDown) firing: (2) Service toolsbeta-test-k8s-haproxy-3:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:38:21] 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 10Goal: Migrate largest ToolsDB users to Trove - https://phabricator.wikimedia.org/T291782 (10fnegri) [12:40:28] (InstanceDown) firing: Project tools instance tools-k8s-worker-75 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:44:20] !log taavi@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster [12:44:20] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [12:45:28] (InstanceDown) resolved: Project tools instance tools-k8s-worker-75 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:49:03] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4), 10DC-Ops, 10SRE, 10ops-eqiad: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 (10dcaro) thanks @Jclark-ctr, unfortunately, that does not really help a lot, and does not answer any of the questio... [12:50:20] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76 [12:51:00] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76 [12:51:22] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster [13:01:11] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE, 10netops, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1031.eqiad.wmnet with OS bookworm com... [13:02:25] !log taavi@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster [13:02:25] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [13:03:02] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4 [13:03:41] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4 [13:10:28] (InstanceDown) firing: Project tools instance tools-k8s-ingress-4 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:15:28] (InstanceDown) resolved: Project tools instance tools-k8s-ingress-4 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:25:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:41:09] 10PAWS: Upgrade Jupyterlab - https://phabricator.wikimedia.org/T357027 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/376 [13:41:23] vivian-rook opened https://github.com/toolforge/paws/pull/376 [13:46:28] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE, 10netops, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10aborrero) [13:59:30] 10PAWS: Upgrade Jupyterlab - https://phabricator.wikimedia.org/T357027 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/376 [13:59:41] vivian-rook closed https://github.com/toolforge/paws/pull/376 [14:00:01] 10PAWS: Upgrade Jupyterlab - https://phabricator.wikimedia.org/T357027 (10rook) 05Open→03Resolved [14:04:19] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE, 10netops, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10aborrero) https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html [14:06:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance paws-puppetmaster-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [14:06:30] 10cloud-services-team, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10taavi) `lang=shell-session taavi@cloudcontrol1005 ~ $ sudo wmcs-openstack resource provider allocation show 240b2f85-94ce-49eb-8d9d-2559838d0738 +--------------... [14:07:19] 10cloud-services-team, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10aborrero) [14:11:59] 10cloud-services-team, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10taavi) ` root@cloudcontrol1005:~# source novaenv.sh root@cloudcontrol1005:~# nova-manage cell_v2 discover_hosts --verbose Modules with known eventlet monkey pa... [14:12:16] !log aborrero@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary [14:12:35] !log aborrero@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) [14:18:04] !log taavi@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary [14:19:18] !log taavi@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) [14:22:30] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [14:22:34] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) [14:22:42] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [14:22:48] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [15:15:59] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4), 10User-aborrero: eqiad1: fix PTR delegations for 185.15.56.0/24 - https://phabricator.wikimedia.org/T341338 (10cmooney) >>! In T341338#9542379, @taavi wrote: > What's left is updating the reverse DNS delegations for the /24 to point to ns0/1/2.wikimedia... [15:26:11] (03PS2) 10Josefanthony: Resolved footer positioning [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/991815 [15:35:22] (HAProxyBackendUnavailable) firing: (2) HAProxy service radosgw-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:40:22] (HAProxyBackendUnavailable) firing: (3) HAProxy service radosgw-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:41:41] (ProbeDown) firing: (2) Service toolsbeta-test-k8s-haproxy-3:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:45:22] (HAProxyBackendUnavailable) resolved: (3) HAProxy service radosgw-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:46:42] (CloudVPSDesignateLeaks) firing: (2) Detected 10 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:50:22] 10cloud-services-team, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10aborrero) p:05Triage→03Medium [15:53:15] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE, 10netops, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [16:02:38] 10Wikibugs: Update irc3 to 0.9.7 - https://phabricator.wikimedia.org/T153947 (10bd808) 05Open→03Resolved a:03Legoktm {3d0949fef35f134e4510c934302152a88237b59b} [16:19:51] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4), 10User-aborrero: eqiad1: fix PTR delegations for 185.15.56.0/24 - https://phabricator.wikimedia.org/T341338 (10cmooney) Change is live with RIPE, all PTR records resolving as they should (netbox generated and those from openstack) for me here at home.... [16:23:51] (DiskSpace) firing: Disk space cloudbackup1004:9100:/ 5.781% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:49:49] (PuppetConstantChange) resolved: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [16:50:18] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [16:55:06] 10Cloud-VPS (Quota-requests): request temporary quota increase for project iiab - https://phabricator.wikimedia.org/T357694 (10Peachey88) [17:03:00] vivian-rook closed https://github.com/toolforge/paws/pull/375 [17:03:42] 10PAWS: update openresty - https://phabricator.wikimedia.org/T357698 (10rook) [17:04:17] 10Tool-ducttape, 10Abstract Wikipedia team: DUCT exits with "panic: runtime error: invalid memory address or nil pointer dereference" on every run during setup-web-proxy - https://phabricator.wikimedia.org/T357354 (10Mcastro) a:03SDunlap [17:08:34] (DiskSpace) resolved: Disk space cloudbackup1004:9100:/ 5.777% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:09:01] 10PAWS: update openresty - https://phabricator.wikimedia.org/T357698 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/377 [17:09:18] vivian-rook opened https://github.com/toolforge/paws/pull/377 [17:11:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance paws-puppetmaster-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:23:28] 10cloud-services-team, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10Andrew) Typically on a reimage we don't need to remove or rediscover hosts; the pool is based on hostname so the reimaged hosts should rejoin without any issues... [17:24:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on paws-puppetmaster-2 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [17:27:10] 10cloud-services-team, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10Andrew) >>! In T357631#9547661, @Andrew wrote: > Typically on a reimage we don't need to remove or rediscover hosts; the pool is based on hostname so the reimag... [17:31:28] (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance paws-puppetmaster-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:34:33] !log fran@wmf3169 admin START - Cookbook wmcs.openstack.roll_reboot_cloudnets (T356975) [17:34:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:44:32] !log fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=0) (T356975) [18:04:28] (PuppetStaleCertificates) resolved: Found non-revoked Puppet certificates for 1 deleted instances on paws-puppetmaster-2 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [18:26:41] (ProbeDown) resolved: (2) Service toolsbeta-test-k8s-haproxy-3:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:33:41] 10PAWS: update openresty - https://phabricator.wikimedia.org/T357698 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/377 [18:33:53] vivian-rook closed https://github.com/toolforge/paws/pull/377 [18:33:59] 10PAWS: update openresty - https://phabricator.wikimedia.org/T357698 (10rook) 05Open→03Resolved [18:42:51] 10cloud-services-team: SystemdUnitDown Unit nova-fullstack.service on node cloudcontrol1006 has been down for long. - https://phabricator.wikimedia.org/T353991 (10fnegri) 05Open→03Invalid This is an old alert, the service is now running fine. [18:45:22] (HAProxyBackendUnavailable) firing: HAProxy service designate-api_backend backend cloudservices1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:49:14] 10cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T357233 (10fnegri) 05Open→03Invalid All these units are now running again. [18:49:45] 10cloud-services-team: NovafullstackSustainedFailures The automated tests were unable to create, provision and decommission a VM in the last 5h - https://phabricator.wikimedia.org/T357335 (10fnegri) 05Open→03Invalid nova-fullstack is working fine again. [18:50:22] (HAProxyBackendUnavailable) resolved: HAProxy service designate-api_backend backend cloudservices1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:51:11] 10cloud-services-team: HAProxyServiceUnavailable HAProxy service neutron-api_backend has no available backends on cloudlb1001:9900 - https://phabricator.wikimedia.org/T352541 (10fnegri) 05Open→03Invalid This is an old alert that is no longer triggering. [18:54:15] 10cloud-services-team: SystemdUnitDown Unit purge_vm_backup.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T352625 (10fnegri) 05Open→03Invalid This is an old alert that is no longer firing. [18:56:42] (CloudVPSDesignateLeaks) resolved: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:58:13] 10cloud-services-team: SystemdUnitDown Unit purge_vm_rbd_images.service on node cloudcontrol1005 has been down for long. - https://phabricator.wikimedia.org/T356473 (10fnegri) 05Open→03Invalid This alert is no longer firing. [19:00:07] 10cloud-services-team: SystemdUnitDown Unit labs-ip-alias-dump.service on node cloudservices1006 has been down for long. - https://phabricator.wikimedia.org/T357232 (10fnegri) 05Open→03Invalid This alert is no longer firing. [19:01:57] 10cloud-services-team: HAProxyServiceUnavailable HAProxy service Abuse has no available backends on cloudlb1002:9900 - https://phabricator.wikimedia.org/T357245 (10fnegri) 05Open→03Invalid This alert is no longer firing. [19:08:11] (CloudVPSDesignateLeaks) firing: (2) Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:10:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:15:28] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:18:29] 10Toolforge Build Service, 10cloud-services-team (FY2023/2024-Q3-Q4), 10Goal, 10User-Raymond_Ndibe, 10User-aborrero: [harbor] Deploy with Helm - https://phabricator.wikimedia.org/T356301 (10Raymond_Ndibe) [19:20:28] (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:21:57] (CloudVPSDesignateLeaks) resolved: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:55:38] 10Wikibugs: wikibugs having a hard time staying connected to libera.chat IRC network - https://phabricator.wikimedia.org/T357729 (10bd808) [21:58:20] 10Wikibugs: wikibugs having a hard time staying connected to libera.chat IRC network - https://phabricator.wikimedia.org/T357729 (10bd808) Adding @greg, @TheresNoTime, and @LucasWerkmeister for visibility. These folks have been kindly restarting the bot job recently. [22:05:14] 10Wikibugs: wikibugs having a hard time staying connected to libera.chat IRC network - https://phabricator.wikimedia.org/T357729 (10bd808) The storms of loss of connection to libera.chat are a bit confusing for me as I have not seen similar issues with #stashbot, #tool-bridgebot, or #jouncebot in the same period... [22:09:49] 10Wikibugs: wikibugs having a hard time staying connected to libera.chat IRC network - https://phabricator.wikimedia.org/T357729 (10AntiCompositeNumber) The SULWatchers have had some connection problems lately as well, but not to the same frequency. [22:20:28] (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources