[00:38:38] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10Papaul) [01:05:06] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10Papaul) [01:30:18] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10Papaul) [02:09:01] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q1:rack/setup/install druid10[09-11] - https://phabricator.wikimedia.org/T314335 (10Papaul) [02:33:29] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q1:rack/setup/install druid10[09-11] - https://phabricator.wikimedia.org/T314335 (10Papaul) [02:51:34] (03PS2) 10Cicalese: Update pingback MediaWiki versions to include new values [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/879595 (https://phabricator.wikimedia.org/T326825) [09:26:11] 10Data-Engineering, 10Data-Services: Wiki replicas are not fully setup for newly created wikis - https://phabricator.wikimedia.org/T315442 (10BTullis) a:03BTullis [09:40:54] 10Data-Engineering, 10Data-Services: Wiki replicas are not fully setup for newly created wikis - https://phabricator.wikimedia.org/T315442 (10BTullis) Apologies for the delay in responding again to this. I've checked the four wikis mentioned here and they all seem to be working now. Are you able to verify this... [09:57:24] 10Data-Engineering-Planning, 10Data Pipelines, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive - https://phabricator.wikimedia.org/T311738 (10BTullis) a:03BTullis I'm very happy to work on this task and to try... [11:25:14] 10Data-Engineering-Planning, 10Epic, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) [11:25:42] 10Data-Engineering-Planning, 10Epic, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) p:05Triage→03Medium [11:32:23] 10Data-Engineering-Planning, 10Epic, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) [11:58:41] 10Data-Engineering-Planning, 10Epic, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) OK, so first things first, we have to [[https://docs.ceph.com/en/latest/releases/general/#understanding-t... [11:59:34] 10Data-Engineering-Planning, 10Epic, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) [14:03:57] 10Data-Engineering-Planning, 10Epic, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) Looking at the existing WMCS Ceph cluster, I can see that this uses packages built by croit.io ` btullis@... [14:32:01] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10Papaul) [14:34:28] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host an-coord100... [15:01:47] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host an-coord100... [15:18:50] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host an-coord1003.eq... [15:20:12] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host an-mariadb1... [15:34:30] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host an-coord1004.eq... [15:37:51] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10Papaul) [15:37:57] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): Upgrade the WDQS streaming updater to latest flink (1.16) - https://phabricator.wikimedia.org/T289836 (10dcausse) [16:07:03] dcausse, ebernhardson: Hi folks - I just realized you are keeping old small-files data on HDFS that piles up to quite many files - The folder is /wmf/data/discovery/transfer_to_es [16:07:20] Do you think this could be pruned, possibly regularly? [16:09:31] 10Data-Engineering-Planning, 10Data Pipelines, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive - https://phabricator.wikimedia.org/T311738 (10Ottomata) Presto is complete. If you can do Iceberg for Spark 3 via... [16:09:51] Details: ~9.7M files, ~350G, mostly daily dates from 2020-01-05 [16:10:18] 10Data-Engineering-Planning, 10Data Pipelines, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive - https://phabricator.wikimedia.org/T311738 (10Ottomata) > Presto is complete. If you can do Iceberg for Spark 3 via... [16:20:25] joal: yes, it's on the backlog (T323616) [16:20:26] T323616: Cleanup the /wmf/data/discovery/transfer_to_es folder in hdfs - https://phabricator.wikimedia.org/T323616 [16:20:58] moving up so that we can pick it up soon [18:00:43] 10Data-Engineering, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Create partman recipe for cephosd servers - https://phabricator.wikimedia.org/T324670 (10fnegri) @BTullis it looks like for `cloudcephosd*` we used the following partman recipes: `cloudcephosd1*) echo partman/s... [19:22:34] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, and 2 others: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host an-mariadb10... [19:41:06] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host an-mariadb1001.... [19:58:16] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host an-mariadb1002.... [19:59:34] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10Papaul) [20:13:28] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q2:rack/setup/install an-coord100[3,4] & an-mariadb100[1,2] - https://phabricator.wikimedia.org/T321119 (10Papaul) 05Open→03Resolved @BTullis this is done. [20:42:07] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q1:rack/setup/install druid10[09-11] - https://phabricator.wikimedia.org/T314335 (10Papaul) @BTullis can you please specify the exact partman recipe to use? Thanks [21:02:33] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q1:rack/setup/install druid10[09-11] - https://phabricator.wikimedia.org/T314335 (10Papaul) [22:53:10] 10Data-Engineering, 10SRE, 10Shared-Data-Infrastructure: geoip_update_main failure on puppetmaster1001 - https://phabricator.wikimedia.org/T324548 (10Dzahn) In an ideal world, once we know a new expiration date, we could add it to the "mainteance calendar", like 2 weeks before it expires. And then the clinic...