[07:03:10] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182062 (10akosiaris) 1240 1241 1242 1243 1244 1246 1249 1250 1252 1253 1254 1257 1258 1259 1260 1262 1264 1265 1270 1271 1273 1274 1278 1279 1281 1282 1284 1286 1289 1290 1294 1296 1301... [07:53:30] 06serviceops, 06Infrastructure-Foundations, 10netops: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845 (10akosiaris) 03NEW [09:38:38] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10182322 (10JMeybohm) > Note that per past experience, changing the ippool is a arduous and dangerous process. We probably don... [09:38:45] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10182332 (10JMeybohm) [09:38:46] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10182331 (10JMeybohm) [11:01:40] 06serviceops, 10Parsoid (Tracking), 13Patch-For-Review: parsoidtest1001 implementation tracking - https://phabricator.wikimedia.org/T363402#10182577 (10akosiaris) @ssastry , @ihurbain parsoidtest1001 is up and running and should be a full replacement of scandium. Ι 've already updated https://wikitech.wikime... [11:06:27] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 07User-notice: Communication for Wikitech/Wikimedia Developer Account migration - https://phabricator.wikimedia.org/T373615#10182598 (10Ladsgroup) If we want this to be in the next week's tech news, this should be added to it now. [11:06:50] 06serviceops, 10Parsoid (Tracking), 13Patch-For-Review: parsoidtest1001 implementation tracking - https://phabricator.wikimedia.org/T363402#10182597 (10akosiaris) p:05Triage→03Medium [11:10:33] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10182615 (10akosiaris) >>! In T375845#10182322, @JMeybohm wrote: >> Note that per past experience, changing the ippool is a ar... [12:13:14] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182801 (10ops-monitoring-bot) pool host wikikube-worker1240.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:13:18] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182802 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1240.eqiad.wmnet completed: - wikikube-worker1240.eq... [12:13:21] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182803 (10ops-monitoring-bot) pool host wikikube-worker1241.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:13:22] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182804 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1241.eqiad.wmnet completed: - wikikube-worker1241.eq... [12:13:28] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182805 (10ops-monitoring-bot) pool host wikikube-worker1242.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:13:30] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182806 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1242.eqiad.wmnet completed: - wikikube-worker1242.eq... [12:13:38] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182807 (10ops-monitoring-bot) pool host wikikube-worker1243.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:13:40] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182808 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1243.eqiad.wmnet completed: - wikikube-worker1243.eq... [12:13:49] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182810 (10ops-monitoring-bot) pool host wikikube-worker1244.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:13:50] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182811 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1244.eqiad.wmnet completed: - wikikube-worker1244.eq... [12:13:54] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182812 (10ops-monitoring-bot) pool host wikikube-worker1245.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:13:56] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182813 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1245.eqiad.wmnet completed: - wikikube-worker1245.eq... [12:14:03] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182814 (10ops-monitoring-bot) pool host wikikube-worker1246.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:14:05] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182815 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1246.eqiad.wmnet completed: - wikikube-worker1246.eq... [12:14:51] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182816 (10akosiaris) 05Open→03Resolved This is done [12:14:53] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182819 (10ops-monitoring-bot) pool host wikikube-worker1247.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:14:56] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182820 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1247.eqiad.wmnet completed: - wikikube-worker124... [12:14:56] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182821 (10ops-monitoring-bot) pool host wikikube-worker1248.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:14:59] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182822 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1248.eqiad.wmnet completed: - wikikube-worker124... [12:15:00] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182823 (10ops-monitoring-bot) pool host wikikube-worker1249.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:15:02] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182824 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1249.eqiad.wmnet completed: - wikikube-worker124... [12:15:03] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182825 (10ops-monitoring-bot) pool host wikikube-worker1250.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:15:06] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182826 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1250.eqiad.wmnet completed: - wikikube-worker125... [12:15:11] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182827 (10ops-monitoring-bot) pool host wikikube-worker1251.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:15:15] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182828 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1251.eqiad.wmnet completed: - wikikube-worker125... [12:15:19] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182829 (10ops-monitoring-bot) pool host wikikube-worker1252.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:15:23] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182830 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1252.eqiad.wmnet completed: - wikikube-worker125... [12:15:27] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182831 (10ops-monitoring-bot) pool host wikikube-worker1253.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:15:31] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182832 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1253.eqiad.wmnet completed: - wikikube-worker125... [12:15:35] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182833 (10ops-monitoring-bot) pool host wikikube-worker1254.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:15:39] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182834 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1254.eqiad.wmnet completed: - wikikube-worker125... [12:15:43] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182835 (10ops-monitoring-bot) pool host wikikube-worker1255.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:15:47] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182836 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1255.eqiad.wmnet completed: - wikikube-worker125... [12:15:51] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182837 (10ops-monitoring-bot) pool host wikikube-worker1256.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:15:55] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182838 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1256.eqiad.wmnet completed: - wikikube-worker125... [12:15:59] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182839 (10ops-monitoring-bot) pool host wikikube-worker1257.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:16:03] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182840 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1257.eqiad.wmnet completed: - wikikube-worker125... [12:16:07] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182841 (10ops-monitoring-bot) pool host wikikube-worker1258.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:16:12] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182842 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1258.eqiad.wmnet completed: - wikikube-worker125... [12:16:16] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182843 (10ops-monitoring-bot) pool host wikikube-worker1259.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:16:20] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182844 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1259.eqiad.wmnet completed: - wikikube-worker125... [12:16:24] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182845 (10ops-monitoring-bot) pool host wikikube-worker1260.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:16:28] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182847 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1260.eqiad.wmnet completed: - wikikube-worker126... [12:16:32] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182848 (10ops-monitoring-bot) pool host wikikube-worker1261.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:16:36] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182849 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1261.eqiad.wmnet completed: - wikikube-worker126... [12:16:40] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182850 (10ops-monitoring-bot) pool host wikikube-worker1262.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:16:46] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182851 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1262.eqiad.wmnet completed: - wikikube-worker126... [12:16:50] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182852 (10ops-monitoring-bot) pool host wikikube-worker1263.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:16:54] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182853 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1263.eqiad.wmnet completed: - wikikube-worker126... [12:16:58] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182854 (10ops-monitoring-bot) pool host wikikube-worker1264.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:17:02] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182855 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1264.eqiad.wmnet completed: - wikikube-worker126... [12:17:06] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182856 (10ops-monitoring-bot) pool host wikikube-worker1265.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:17:10] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182857 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1265.eqiad.wmnet completed: - wikikube-worker126... [12:17:14] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182858 (10ops-monitoring-bot) pool host wikikube-worker1266.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:17:18] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182859 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1266.eqiad.wmnet completed: - wikikube-worker126... [12:17:22] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182860 (10ops-monitoring-bot) pool host wikikube-worker1267.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:17:26] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182861 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1267.eqiad.wmnet completed: - wikikube-worker126... [12:17:35] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182862 (10ops-monitoring-bot) pool host wikikube-worker1268.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:17:39] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182863 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1268.eqiad.wmnet completed: - wikikube-worker126... [12:17:43] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182864 (10ops-monitoring-bot) pool host wikikube-worker1269.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:17:47] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182865 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1269.eqiad.wmnet completed: - wikikube-worker126... [12:17:51] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182866 (10ops-monitoring-bot) pool host wikikube-worker1270.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:17:55] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182867 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1270.eqiad.wmnet completed: - wikikube-worker127... [12:18:03] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182868 (10ops-monitoring-bot) pool host wikikube-worker1271.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:18:07] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182869 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1271.eqiad.wmnet completed: - wikikube-worker127... [12:18:11] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182870 (10ops-monitoring-bot) pool host wikikube-worker1272.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:18:15] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182871 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1272.eqiad.wmnet completed: - wikikube-worker127... [12:18:19] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182872 (10ops-monitoring-bot) pool host wikikube-worker1273.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:18:23] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182873 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1273.eqiad.wmnet completed: - wikikube-worker127... [12:18:27] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182874 (10ops-monitoring-bot) pool host wikikube-worker1274.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:18:31] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182875 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1274.eqiad.wmnet completed: - wikikube-worker127... [12:18:39] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182876 (10ops-monitoring-bot) pool host wikikube-worker1275.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:18:43] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182877 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1275.eqiad.wmnet completed: - wikikube-worker127... [12:18:47] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182878 (10ops-monitoring-bot) pool host wikikube-worker1276.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:18:52] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182879 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1276.eqiad.wmnet completed: - wikikube-worker127... [12:18:56] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182880 (10ops-monitoring-bot) pool host wikikube-worker1277.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:19:04] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182881 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1277.eqiad.wmnet completed: - wikikube-worker127... [12:19:08] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182882 (10ops-monitoring-bot) pool host wikikube-worker1278.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:19:12] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182883 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1278.eqiad.wmnet completed: - wikikube-worker127... [12:19:16] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182884 (10ops-monitoring-bot) pool host wikikube-worker1279.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:19:20] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182885 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1279.eqiad.wmnet completed: - wikikube-worker127... [12:19:24] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182886 (10ops-monitoring-bot) pool host wikikube-worker1280.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:19:28] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182887 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1280.eqiad.wmnet completed: - wikikube-worker128... [12:19:36] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182888 (10ops-monitoring-bot) pool host wikikube-worker1281.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:19:40] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182889 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1281.eqiad.wmnet completed: - wikikube-worker128... [12:19:44] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182890 (10ops-monitoring-bot) pool host wikikube-worker1282.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:19:49] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182891 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1282.eqiad.wmnet completed: - wikikube-worker128... [12:19:57] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182892 (10ops-monitoring-bot) pool host wikikube-worker1283.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:20:01] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182893 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1283.eqiad.wmnet completed: - wikikube-worker128... [12:20:05] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182894 (10ops-monitoring-bot) pool host wikikube-worker1284.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:20:09] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182895 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1284.eqiad.wmnet completed: - wikikube-worker128... [12:20:13] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182896 (10ops-monitoring-bot) pool host wikikube-worker1285.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:20:17] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182897 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1285.eqiad.wmnet completed: - wikikube-worker128... [12:20:21] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182900 (10ops-monitoring-bot) pool host wikikube-worker1286.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:20:25] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182901 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1286.eqiad.wmnet completed: - wikikube-worker128... [12:20:29] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182902 (10ops-monitoring-bot) pool host wikikube-worker1287.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:20:33] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182903 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1287.eqiad.wmnet completed: - wikikube-worker128... [12:20:37] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182904 (10ops-monitoring-bot) pool host wikikube-worker1288.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:20:41] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182905 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1288.eqiad.wmnet completed: - wikikube-worker128... [12:20:45] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182906 (10ops-monitoring-bot) pool host wikikube-worker1289.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:20:49] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182907 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1289.eqiad.wmnet completed: - wikikube-worker128... [12:20:58] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182908 (10ops-monitoring-bot) pool host wikikube-worker1290.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:21:02] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182909 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1290.eqiad.wmnet completed: - wikikube-worker129... [12:21:10] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182910 (10ops-monitoring-bot) pool host wikikube-worker1291.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:21:14] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182911 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1291.eqiad.wmnet completed: - wikikube-worker129... [12:21:20] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182912 (10ops-monitoring-bot) pool host wikikube-worker1292.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:21:24] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182913 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1292.eqiad.wmnet completed: - wikikube-worker129... [12:21:28] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182914 (10ops-monitoring-bot) pool host wikikube-worker1293.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:21:32] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182915 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1293.eqiad.wmnet completed: - wikikube-worker129... [12:21:36] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182916 (10ops-monitoring-bot) pool host wikikube-worker1294.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:21:40] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182917 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1294.eqiad.wmnet completed: - wikikube-worker129... [12:21:44] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182918 (10ops-monitoring-bot) pool host wikikube-worker1295.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:21:49] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182919 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1295.eqiad.wmnet completed: - wikikube-worker129... [12:21:57] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182920 (10ops-monitoring-bot) pool host wikikube-worker1296.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:22:01] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182921 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1296.eqiad.wmnet completed: - wikikube-worker129... [12:22:05] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182924 (10ops-monitoring-bot) pool host wikikube-worker1297.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:22:09] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182925 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1297.eqiad.wmnet completed: - wikikube-worker129... [12:22:13] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182926 (10ops-monitoring-bot) pool host wikikube-worker1298.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:22:17] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182927 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1298.eqiad.wmnet completed: - wikikube-worker129... [12:22:21] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182928 (10ops-monitoring-bot) pool host wikikube-worker1299.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:22:25] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182929 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1299.eqiad.wmnet completed: - wikikube-worker129... [12:22:29] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182930 (10ops-monitoring-bot) pool host wikikube-worker1300.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:22:33] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182931 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1300.eqiad.wmnet completed: - wikikube-worker130... [12:22:37] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182932 (10ops-monitoring-bot) pool host wikikube-worker1301.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:22:41] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182933 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1301.eqiad.wmnet completed: - wikikube-worker130... [12:22:45] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182934 (10ops-monitoring-bot) pool host wikikube-worker1302.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:22:49] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182935 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1302.eqiad.wmnet completed: - wikikube-worker130... [12:22:53] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182936 (10ops-monitoring-bot) pool host wikikube-worker1303.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:22:58] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182937 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1303.eqiad.wmnet completed: - wikikube-worker130... [12:23:05] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182938 (10ops-monitoring-bot) pool host wikikube-worker1304.eqiad.wmnet by akosiaris@cumin1002 with reason: None [12:23:09] 06serviceops: wikikube-worker1240 to wikikube-worker1304 implementation tracking - https://phabricator.wikimedia.org/T369744#10182939 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by akosiaris@cumin1002 pool for host wikikube-worker1304.eqiad.wmnet completed: - wikikube-worker130... [13:06:55] 06serviceops, 10Prod-Kubernetes, 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18), and 3 others: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10183111 (10Gehel) [13:41:50] 06serviceops, 06SRE: php7.2-fpm_check_restart should be resilient to php7adm error pages - https://phabricator.wikimedia.org/T285593#10183281 (10akosiaris) 05Open→03Invalid 3 years later, we no longer have appservers, this is probably best closed as invalid [15:10:18] 06serviceops, 10MediaWiki-Uploading, 07Regression, 07Wikimedia-production-error: Large file uploads broken via Special:Upload - https://phabricator.wikimedia.org/T374436#10183599 (10Cparle) That looks more like a few thousand times a //month// on commons to me. Am I reading it wrong? [15:14:27] 06serviceops, 10MediaWiki-Uploading, 07Regression, 07Wikimedia-production-error: Large file uploads broken via Special:Upload - https://phabricator.wikimedia.org/T374436#10183626 (10hnowlan) >>! In T374436#10183599, @Cparle wrote: > That looks more like a few thousand times a //month// on commons to me. Am... [16:28:11] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10183874 (10Scott_French) Collecting remaining follow-ups so far: **kartotherian**: Given that we've seen capacity related failures during the last two swit... [17:22:48] 06serviceops, 10MW-on-K8s: Functional replacement for `sql` on Kubernetes - https://phabricator.wikimedia.org/T375910 (10RLazarus) 03NEW [17:48:05] 06serviceops, 10MediaWiki-Uploading, 07Regression, 07Wikimedia-production-error: Large file uploads broken via Special:Upload - https://phabricator.wikimedia.org/T374436#10184187 (10Quiddity) [19:32:16] 06serviceops, 10MediaWiki-Uploading, 07Regression, 07Wikimedia-production-error: Large file uploads broken via Special:Upload - https://phabricator.wikimedia.org/T374436#10184550 (10Tgr) Looking at the [[https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.09.27?i...