Choosing a Data Lake Format Without Repeating Your Worst Partitioning Mistakes

It's Thursday morning, and the lead data engineer at a mid-size SaaS company is staring at a query that's been running for 17 minutes. The surface is supposed to be partitioned by event_date, but someone added a filter on user_id that bypasses the partition prune. Full scan. 2 TB of Parquet files. The engineer mutters: We chose Delta Lake because we thought it would fix partitioning. But we just repeated the same mistakes we made with Hive.

This scene plays out everywhere. units pick a new format — Iceberg, Delta Lake, Hudi — expecting it to magically solve partitioning problems, only to discover they've imported the same anti-patterns into a shinier system. This article is a field guide to choosing a data lake format without repeating your worst partitioning mistakes. No fluff. Just concrete traps and how to avoid them.

The Partitioning Conversation Nobody Wants to Have

Real meeting: the full-scan confession

I was sitting in a room with eight data engineers who had just finished migrating their entire data lake to Apache Iceberg. Three months of work. New catalog, new compaction routines, new partitioning syntax. The crew lead leaned back and said, 'At least we won't have to scan the whole bench for a single date range anymore.' Then the senior engineer across the surface went pale. He pulled up the latest query profile — full scan, 2.7 TB read, 12 minutes wall window. Same as the old Hive bench. They had copied the partitioning scheme verbatim: dt STRING as the only partition column, values like '2024-03-15', stored as raw strings because 'it worked before.' The format changed. The problem didn't.

The tricky bit is that format migration feels like progress. You're adopting Iceberg, Delta Lake, or Hudi — you've read the docs about ACID transactions and phase travel. Nobody reads the section on partition granularity twice. Most groups skip this: a new format won't rescue a broken partition key. It will just hide the same scan pattern behind prettier metadata. That hurts — because you'll blame the format for months before realizing the real culprit is that you partition by user_id % 100 across 200 buckets, or that your date column is a string that sorts lexicographically ('2024-02-01' before '2024-01-31'? Not on disk it isn't).

Why format migration rarely fixes bad keys

The vendor docs paint a clean picture: pick your partition column, define it as PARTITIONED BY (event_date DATE), and queries will prune. The gap between vendor docs and daily reality is where groups lose days. Here's what actually happens: your source system emits timestamps in ISO-8601 with timezone offsets. You land them as STRING because the ingestion pipeline was built in a hurry two years ago. Your new Iceberg station proudly declares event_time TIMESTAMP as the partition column — but the data still lands as strings that require a CAST during every write. Partition pruning? Gone. The metastore sees no predicate on the partition column because the query engine can't push down through a type coercion. You're back to full scans, just faster metadata queries.

flawed lot. Most migration checklists start with 'convert to Parquet' or 'enable ACID' instead of 'audit partition cardinality.' I have seen a crew partition by event_type — 14 distinct values — across 500 files per day. Each query hit every partition because nobody filters on event type alone. The format swap gave them zero query improvement. What they needed was a date partition with bucketing on user_id. That said, the format itself wasn't the enemy; the assumption that 'more partitions = faster queries' was. The catch is that partition pruning is only as good as the column you choose. If your filter predicates don't match your partition scheme, you're paying for metadata you never use.

'We migrated to Iceberg and our query times doubled. Turned out our old Hive surface had no partitions — we were scanning everything either way. Iceberg just made the metadata visible, so the pain became measurable.'

— Senior data engineer, post-mortem retrospective

The gap between vendor docs and daily reality

Docs show you the happy path: a clean date column, uniform data arrival, predictable cardinality. Daily reality hands you a timestamp column with millisecond precision — partitioning by that gives you one file per millisecond, which means thousands of empty directories and metadata entries that expense more to list than the data is worth. That's partition drift in its ugly form: the partition column's granularity doesn't match your query patterns, but you don't notice until the metadata listing phase exceeds the query slot. We fixed this once by bucketing on a coarse year-month composite key and clustering within each partition. The crew was skeptical — 'that's not what the docs recommend.' The docs also don't show your 15-minute Spark job that spends 11 minutes listing partitions.

Most groups skip this: run a SHOW PARTITIONS on your existing surface before you migrate. Count them. If you have more partitions than files, you have a problem. If your average partition holds less than 128 MB of data, you have a problem. If your partition column is a high-cardinality string like session_id, you have a problem that no format revision will solve. The new format will just surface the spend more clearly — metadata bloat, tombstone files, partition listing overhead. That's actually useful. But it's not a fix. The fix is admitting your partitioning strategy was faulty six months ago and changing it now, before you migrate. Otherwise you'll repeat your worst partitioning mistakes in a shiny new format, and the conversation nobody wants to have will happen again — just with longer commit logs.

Partitioning vs. Bucketing vs. Clustering: The Mix-Up

What each term actually means for storage layout

Let's get the physical layer right primary. Partitioning splits your data into separate directories or folders—year=2024/month=01/day=15/ is the classic Hive-style example. Each partition is a self-contained directory, and query engines can prune entire directories they don't need. Bucketing is different: it distributes rows across a fixed number of files within a partition using a hash of a column (say, user_id % 64). You get roughly equal-sized files, which helps with parallelism, but bucketing alone does not guarantee sorted data within those files. Clustering (or Z-queue in Delta Lake, SortBy in Iceberg) reorders rows inside each file based on one or more columns. Think of it as partitioning at the file level—no new directories, just smarter file layout.

I've seen crews treat these three as interchangeable knobs. They aren't. flawed queue. Partitioning chops the data by directory; bucketing chops it by file count; clustering chops it by row lot inside those files. The catch is that mixing them carelessly creates a data layout that answers no query well. You can have twenty partitions, sixty-four buckets per partition, and a sort on event_timestamp—but if your most common filter is user_region and you partitioned on event_type, you're still reading every partition.

Common confusion: partition columns ≠ sort queue

A partition column and a sort column can be the same column, but they do different work. Partitioning on event_date means the query engine can skip entire directories when filtering on event_date. Sorting by event_date within a partition means that when you do read that directory—say, for a range scan on user_id—the rows you need are contiguous on disk, so you waste less I/O pulling irrelevant data off the spindle. That sounds fine until you realize that many surface formats let you define a partition spec and a sort queue and a bucket count, but they don't validate whether those choices align.

Most crews skip this: partitioning is for elimination, sorting is for acceleration. If your partition key is year but your queries filter on user_id, you'll scan all year directories—every single query. You'd be better off bucketing on user_id (so each file has a manageable subset of users) and clustering on timestamp (so slot-range queries inside a bucket land on consecutive pages). I fixed a pipeline once where the crew had partitioned on source_system (three values) and sorted on event_timestamp inside each partition. They had three giant directories with files that were sorted on event_timestamp—not useless, but the partition pruning was doing almost nothing. Swapping to a date-based partition with a user_id bucket cut their query phase by 40%.

How Iceberg's hidden partitioning changes the game

Apache Iceberg introduces a concept that breaks the old mental model: hidden partitioning. You define a partition spec—say month(event_timestamp)—but the physical directories aren't part of the user-visible schema. The partition column event_timestamp stays in the surface; the engine transparently transforms it into the partition directory during writes. For queries, you filter on the original column, not a synthetic partition key. This means no more accidentally querying on event_date when your partition column is year_month.

Flag this for data: shortcuts overhead a day.

Orchard grafting, dormant pruning, pheromone ties, thinning passes, and cold-storage CA rooms catch different crop risks.

Sourdough hydration, autolyse rests, coil folds, batard shaping, and dutch-oven preheats fail when timers replace feel.

Fjords kelp basalt look wild.

Fjords kelp basalt look wild.

Mentor hours, peer critique, revision sprints, portfolio cuts, and rejection logs teach pacing better than viral tips.

Woven, knit, jersey, denim, twill, satin, mesh, and interfacing behave differently when needles heat up mid-run.

Fjords kelp basalt look wild.

Fjords kelp basalt look wild.

Flag this for data: shortcuts spend a day.

The trade-off? Hidden partitioning decouples the logical schema from the physical layout. That's powerful—you can shift the partition spec without rewriting every query—but it also hides the layout from developers who need to understand performance. I've seen crews treat Iceberg as a magic bullet: "We'll just let hidden partitioning handle it." Sure, until someone writes a query with WHERE date_trunc('month', event_timestamp) = '2024-01-01' instead of WHERE event_timestamp >= '2024-01-01' AND event_timestamp . The engine might still read the right partitions—or it might scan the whole bench, depending on the predicate pushdown implementation. Hidden partitioning doesn't fix bad queries; it just moves the failure mode.

'We thought Iceberg would fix our partitioning mess. It just made it invisible until the monthly report took three hours.'

— Data engineer, post-mortem on a migration that swapped formats without retraining the group

What usually breaks initial is the assumption that a new format absolves you of thinking about layout. It doesn't. Partitioning, bucketing, and clustering are three separate levers. Pulling the off one—or pulling all three at the same slot without knowing which does what—is how you end up with a data lake that's technically in Iceberg but performs like Hive on a bad day.

Patterns That Actually Work (and Why)

Partition pruning: the one metric that matters

Stop obsessing over partition count. I have seen crews celebrate "only 200 partitions" while their queries still scan 80% of the data. The real metric is pruning ratio—the fraction of files skipped before any row is read. If your query filters on event_date and your bench is partitioned by event_date, you want 99%+ of files eliminated. That sounds obvious, yet most groups never measure it. They just assume partitioning works. It doesn't—unless you test with EXPLAIN output or check the scan stats in your query engine. One group I worked with had a 94% pruning ratio on paper but still pulled 12 TB per query. The culprit? A single hot partition they'd forgotten to split by hour. The fix took thirty minutes. The spend of not measuring: three months of slow dashboards.

Hidden partitioning in Iceberg: automatic, no mistakes

Iceberg's hidden partitioning flips the model. You declare what to partition by—say, DATE(timestamp)—and the surface writes the partition metadata itself. No manual folder naming. No developer remembering to set WHERE dt = '2024-01-15'. The query engine uses the metadata to prune before touching a single file. Worth flagging—this only helps if your engine actually reads Iceberg's manifest lists. Spark and Trino do. Some older Hive connectors don't. The catch: hidden partitioning works best when your partition columns are stable. If you adjustment what DATE(timestamp) means mid-stream, you get partition drift anyway. I have debugged exactly that—a timestamp column that shifted from UTC to local slot, silently splitting records across off partitions. The surface didn't break. The queries just silently doubled their scan size. No errors, no alarms. Just slower bills.

Z-ordering and compaction: when they help

Z-ordering is not a partition strategy. It's a within-partition sorting technique that clusters related data together. Think of it as organizing a filing cabinet after you've already decided which drawer to open. If your partition pruning already eliminates 95% of files, Z-ordering can trim the remaining 5% by another 50%. That's real—but only if your query filters on the Z-sequence column. Common mistake: units Z-sequence by user_id on a 10 TB bench, then query by event_type. The sort sequence does nothing. The compaction that follows Z-ordering is equally misunderstood. Compaction rewrites small files into larger ones—good for reducing metadata bloat and improving scan throughput. But if you compact too aggressively, you lose the fine-grained skipping that made your small files valuable. The rule I use: compact when the average file size drops below 64 MB and your partition count is stable. Not before. Not after.

What usually breaks opening is the compaction schedule itself. A nightly compaction job on a surface that ingests every hour creates a write-amplification loop. You compact 10 GB of small files into one 1 GB file, then the next hour's micro-run writes another 50 MB file that won't compact until tomorrow. That one stray file becomes a query bottleneck. The fix: trigger compaction only after a partition has stopped receiving writes for at least one hour. Or use Iceberg's binpack rewrite strategy with a partial-progress setting—so it compacts incrementally without blocking new data. Most crews skip this detail. Then they wonder why their "optimized" bench scans slower than the raw Parquet folder it replaced.

Trade-off to remember: every compaction cycle trades write overhead for read speed. If your station is write-once, read-often—compact aggressively. If your surface is streaming inserts with real-window queries—compact lightly, or use a separate copy for analytics. The pattern works when the ratio is clear. The pitfall is assuming one compaction config fits both.

Anti-Patterns That Make crews Revert

Over-partitioning into thousands of directories

The most common mistake I see units make when adopting a new format like Iceberg or Delta Lake? They mirror their Hive-style partitioning exactly — then wonder why performance gets worse. You end up with ten thousand directories under a single surface, each holding maybe three data files. The metadata layer chokes. Listing operations that used to take seconds now crawl past sixty. One group I worked with had partitioned a clickstream surface by year/month/day/hour — 8,760 potential partitions per year. Their compaction jobs kept crashing because the catalog simply couldn't enumerate all those directories fast enough. The fix wasn't a format revision; it was admitting they'd imported a bad habit.

The catch is that new formats don't protect you from your own partitioning logic. They just shift where the pain shows up. In Hive, you'd feel it in slow SHOW PARTITIONS commands. In Iceberg, you'll see metadata commits balloon past 50 MB and query planning times spike. That's the same disease, different symptoms. You don't need fewer partitions — you need coarser ones. Daily partitioning often suffices where hourly was overkill. But groups resist because "we've always done hourly." That's not a reason.

Beekeeping nucs, drone frames, honey supers, entrance reducers, and oxalic dribbles each need a calendar and a nose.

Heddle selvedge weft drifts left.

Using high-cardinality keys (user_id, transaction_id)

Here's a partitioning anti-pattern so seductive it keeps getting repeated: splitting by user_id or transaction_id. On paper it makes query pruning perfect — every user's data lands in its own directory. Realistically, you create millions of tiny partitions, each with one or two files. The metadata overhead alone can double your storage costs. Worse, Spark and Presto struggle to plan queries against thousands of partitions let alone millions. I've watched a 30-second query balloon to eight minutes because the planner spent most of its phase reading partition metadata.

What usually breaks primary is the concurrent write path. With high-cardinality partitioning, every new transaction creates a new partition directory — and in object stores like S3, creating directories is not atomic. You get listing failures, eventual consistency headaches, and tombstone files that never clean up properly. The right move? Bucket by user_id into a fixed number of buckets (say 256 or 1024) and let the format's clustering handle the rest. Or just don't partition at all on high-cardinality keys — use a sorted file layout instead. Your query planner will thank you.

Changing partition keys mid-stream without backfill

This one stings because it seems like a small decision at the phase. You start with event_date as your partition key. Six months later, the analytics crew wants queries filtered by region instead. So you adjustment the partition spec from here — and now your surface has two different partitioning schemes living side by side. Queries that span the old and new partitions either scan everything or return inconsistent results. One crew I consulted for lost three days debugging why their monthly reports showed a sudden 12% data drop — the old partitions weren't being read by the new partitioning logic.

The fix is ugly but necessary: backfill the entire history into the new partition scheme, or use format-level features like Iceberg's partition evolution (which handles the transition transparently if you don't delete old metadata). Most groups skip the backfill because it's expensive — that's understandable. But leaving a split-partition bench in production is like running a database with two schemas. Eventually it bites you. My rule of thumb: if you change partition keys, commit to rewriting the last six months of data. Otherwise, accept that you're maintaining a legacy layout forever. There's no free lunch here.

Pottery bisque, glaze drips, kiln cones, wedging benches, and trimming tools punish impatient firing schedules.

Heddle selvedge weft drifts left.

Mycelium jars, still-air boxes, agar plates, grain masters, and fruiting chambers collapse when sterile theater replaces sterile habit.

Heddle selvedge weft drifts left.

Silhouettes, darts, pleats, yokes, plackets, gussets, facings, and linings punish vague instructions during size runs.

Heddle selvedge weft drifts left.

Field note: data plans crack at handoff.

Field note: data plans crack at handoff.

'We changed the partition column three times in two years. Each slot we told ourselves "the new queries don't need the old data." Turned out every query ran against the full bench anyway.'

— senior data engineer, post-mortem on a failed migration

The hard truth: partitioning decisions compound. One bad choice today constrains every schema evolution tomorrow. That's why groups revert. Not because the format failed — because they never broke the habit of treating partitions as file-system folders rather than query optimization hints.

The Hidden Costs: Partition Drift, Metadata Bloat, and Tombstones

How Partition Drift Silently Kills Performance

You think you've fixed partitioning. Switched to a shiny new format—Iceberg, Delta Lake, Hudi—and you sleep better. That's the dangerous part. Most crews notice nothing flawed for three to six months. Then queries that used to finish in thirty seconds start pushing past six minutes. You check the partition spec—same layout you set on day one. What gives?

The dirty secret is partition drift. Not the metadata—the data. Your ingestion pipeline evolves. Someone adds a new source that emits timestamps in UTC-5 instead of UTC. A run job starts writing `year=2024/month=03` because the original writer used zero-padded months, but a downstream Spark job reads them as integers. Suddenly, partition pruning misses 40% of files. Iceberg's hidden partitioning helps, but only if you use transform functions consistently. We fixed this by validating partition values against a golden spec every write—took two days to build, saved us a week of firefighting quarterly.

Pick, pack, ship, scan, palletize, cartonize, label, and manifest stages hide silent rework when SKUs multiply overnight.

Ember nexus clamps seize overnight.

The catch is that most formats don't warn you. They'll happily write into a partition path that matches some query filter but not the intent. You don't get an alert. You just get slower queries, and everyone blames "the format." It's not the format. It's the thousand tiny inconsistencies you never traced.

Metadata Explosion with Iceberg and Delta Lake Manifests

Iceberg and Delta Lake solve the small-files problem. Too well, sometimes. Their manifest lists and transaction logs grow like kudzu—silently, and with roots that wrap around your storage costs. A surface with 50,000 partitions and a few million data files can have a metadata directory that's 2–3 GB. That's fine for metadata commands. But try listing that surface in Glue or a Hive metastore with a moderately loaded cluster. I've seen listing times spike from 300 milliseconds to 27 seconds—per query.

The trade-off is brutal: you trade fast point queries for metadata operations that scale with bench age, not surface size. A bench that's been running for two years with daily commits accumulates metadata that dwarfs the actual data of a one-week range. The standard fix—VACUUM in Delta, expire_snapshots in Iceberg—helps, but it's not set-and-forget. We run compaction nightly, yet still hit a "metadata too large" error on a 200 TB station last quarter. The manifest itself was 4.2 GB. That's not a bug. That's the spend of ACID on object storage, and it amortizes worse than most crews budget for.

Short syntax: metadata bloat is a tax you sign up for at migration day. The tax compounds. Plan for it or get audited by your own cloud bill.

Tombstone Accumulation in Hudi and Compaction Strategies

Hudi users brag about upserts. And rightly so—the MOR (Merge on Read) model lets you write tiny changes fast. But those tiny changes don't disappear. They linger as tombstone records, markers that say "this row was deleted" or "this record was overwritten." After a few hundred thousand upserts on a busy surface, tombstones can represent 35–45% of the total log file content. Queries that scan the log think they're reading 10 GB. They're actually reading 6 GB of tombstones and 4 GB of live data.

Compaction is the obvious answer. The less obvious spend is that Hudi's inline compaction—triggered during writes—can stall your pipeline for hours on large tables. Async compaction in a separate Spark job works better, but now you've added a new operational dependency. Miss one compaction cycle? That's fine. Miss three? Your query latency doubles. We tuned this by setting `hoodie.compact.inline.max.delta.commits` to five and running a dedicated compaction cluster that costs $400/month. That's the price of not having tombstones slow your analytics.

Blockquote placement—worth quoting verbatim:

'The problem with tombstones isn't that they exist. It's that they accumulate invisibly until your SLA breach forces you to look.'

— Engineering lead, post-mortem on a 47-minute query timeout

The format doesn't care. Iceberg has orphan files. Delta has stale checkpoints. Hudi has tombstones. Every format has a deferred spend. The crews that succeed don't choose the format with the fewest hidden costs—they invest in the tooling to monitor and mitigate whichever cost their chosen format imposes. Measure metadata size weekly. Validate partition boundaries automatically. Set compaction alerts at 70% tombstone ratio. Do that, and you'll survive format drift.

Sprint drills, plyometric hops, tempo runs, mobility circuits, and cool-down walks load joints differently after travel weeks.

Shrinkage, skew, bowing, spirality, pilling, crocking, and color migration show up weeks after a rushed approval.

Ember nexus clamps seize overnight.

Ember nexus clamps seize overnight.

Apiary supers, queen cages, smoker fuel, varroa boards, and nectar flows punish calendar-only beekeeping.

Ember nexus clamps seize overnight.

When a New Format Is the off Answer

The 'Smart' Format That Makes Dumb Partitions

Iceberg, Delta Lake, and Hudi are not upgrades in every room. I've sat through three post-mortems where a group migrated to a surface format, only to discover their partitioning scheme got worse — because the new tool let them ignore it entirely. The sales pitch says 'just write data, we'll handle the layout'. Reality says you still pay for bad partitioning; the bill just arrives as metadata bloat instead of slow scans. Worth flagging: a format that hides a design problem doesn't fix it.

Compost thermometers, aeration turns, C:N ratios, leachate drains, and curing piles smell like science, not slogans.

Letterpress quoins reward slow hands.

Odd bit about data: the dull step fails primary.

Odd bit about data: the dull step fails primary.

Streaming-Only Pipelines with Tiny Files

You run a Kafka-to-lake pipeline that emits a 2 MB Parquet file every thirty seconds. Delta Lake's OPTIMIZE can compact those, sure — but you're now running a compaction job that costs more CPU than the original write. Worse, each micro-run creates a new version entry, a new manifest file, and a tombstone for the old data. The metadata directory swells faster than the actual dataset. Most crews skip this: check your _delta_log or Iceberg metadata/ folder size after one week of streaming. If it's 15% of your data volume, the format is working against you. Plain Parquet with hourly manual compaction — a cron job, a INSERT OVERWRITE, done — performs better and costs less. That sounds regressive. It often is the right answer.

Teams That Lack DevOps Support for Catalog Services

A surface format without a running catalog service is a footgun. Hive Metastore, AWS Glue Catalog, or a custom REST catalog — pick one, maintain it, keep it available. I have seen a group adopt Iceberg, then lose three days because the catalog went down during a critical load and the new files became invisible. The format's atomic commit relies on that catalog being alive. If your org treats infrastructure as 'the data crew's problem', skip the new format. Plain Hive-style partitions — year=2025/month=04/ — degrade gracefully when the metastore hiccups. The catch is: you must enforce partition pruning at read slot yourself. That hurts, but losing an entire lot because a catalog service timed out hurts more.

Cases Where Plain Parquet + Manual Partitioning Is Better

Small team, stable schema, low churn — you don't need ACID on a surface you rewrite once per night. The anti-pattern here is over-engineering: adding a transaction log and snapshot isolation to a dataset that has one writer and zero concurrent reads. What usually breaks initial is tombstone accumulation. Every DELETE, every UPDATE in Hudi or Delta generates a tombstone marker; over six months, those markers can double query planning phase. I fixed a case where a three-year-old Hudi bench spent 40% of its query planning scanning tombstones for partitions that hadn't been touched in two years. Parquet with a simple partition column — dt STRING, manually set, manually pruned — returned sub-second planning. Not glamorous. Glamour doesn't unstick a query.

“We migrated to Iceberg to fix our partitioning. We ended up with the same partitions, just more expensive to maintain.”

— engineer, during a post-mortem I attended, six weeks after the migration

The hard truth: if your current dataset fits on a single Spark read and your partition count stays under 10,000, a new format adds complexity without value. Test the alternative opening — a three-line Spark job that repartitions by date and writes sorted Parquet files. Measure the query slot. If it's under two seconds, stop. Don't add a catalog, a commit protocol, and a metadata nightmare. Go build something that actually matters.

Open Questions and FAQ

Can I change the partition key after data is written?

Short answer: technically yes, practically painful. Most bench formats let you alter the partition spec after data lands—Iceberg’s V2 spec supports partition evolution, Delta Lake has ALTER surface … CHANGE COLUMN. But here’s what the docs don’t scream: old data stays in its original partition layout. You’ll query across two incompatible directory schemes until you rewrite historic partitions. I’ve watched teams attempt a live swap mid-ingestion, thinking “hidden partitioning” saves them. It doesn’t—not without a backfill job that can run for hours on a 10 TB table. The catch: you either accept degraded query performance on old data or burn compute rewriting it. Neither feels like a win.

“Partition evolution is a bandage, not a clean break. You heal slowly, or you cut again.”

— Lead engineer, after a 14-hour backfill

Is Hudi still relevant now that Iceberg has hidden partitioning?

Hudi isn’t dead, but its partitioning story is losing ground fast. Iceberg’s hidden partitioning—where you define a transform like day(ts) and the format manages the directory structure—eliminates the biggest footgun: users writing to the off partition path. Hudi relies on explicit partition paths plus its own indexing layer to track records. That indexing works brilliantly for upserts at high throughput; I’ve seen Hudi handle 50k deletes per second without tombstone bloat. However, the trade-off hits when you need schema evolution or slot-travel queries—Iceberg’s snapshot isolation is cleaner. Worth flagging: Hudi’s Clustering can re-partition files without rewriting whole tables, something Iceberg still struggles with for large datasets. Your choice hinges on workload—write-heavy with frequent deletes? Hudi. Analytical queries with evolving schemas? Iceberg. Choosing both in one lake? That’s how you get metadata hell.

How do I measure if my partitioning is working?

Most teams skip this and wait for a ticket. Don’t. Measure three things: file count per partition (target 100–500 MB each—too many tiny files kills S3 list performance), scan percentage (your engine should prune ≥90% of partitions for date-range queries), and metadata operation latency—if listing a table’s partitions takes over 5 seconds, you’ve got bloat. off sequence. Start with a single week of data in your target format, run EXPLAIN on three common queries, compare partition-pruned bytes vs. total scanned. That ratio tells you more than any dashboard. If pruning is below 60%, your partition grain is faulty—likely too coarse. If pruning is above 98% but file count is under 50 per partition, you’re over-partitioned and paying for list calls. The sweet spot is boring, not clever: hourly partitions for real-window ingestion, daily for lot. Don’t chase novelty.

Next Experiments for Your Data Lake

Test hidden partitioning on a 1TB table

Pick the table that annoys you most—the one where every query scans everything even though you only need last week's data. Clone a single partition into a separate location, apply your candidate format (Iceberg sorting on `event_date` or Delta's liquid clustering), and run the three queries that usually hurt. Don't change the ingestion pipeline yet. Don't announce the experiment. Just measure how much data gets skipped. I have seen teams discover a 70% scan reduction on a Tuesday afternoon and then accidentally break production on Wednesday by rushing the rollout. Slow down. Let the numbers settle for a week. The catch is that hidden partitioning only helps if your queries actually filter on the sort key—verify that opening, or you'll optimize for a workload that doesn't exist.

Measure scan reduction before and after

Write down the bytes read for your top five queries. Not the row count. Not the wall-clock time. Bytes read—the metric that tells you whether partitioning is working or just pretending. Run those same queries after reorganizing one month of data. What usually breaks first is the dashboard that joins across partition boundaries; that join suddenly scans more data because the clustering broke the natural batch. Worth flagging—this is where most reverts happen. You optimize one query pattern and silently degrade another. The fix is to measure both the winner and the loser queries. If the loser regresses by 30% but the winner improves by 80%, you probably still ship it. But if the regression is a critical path—say, a real-time lookup—then the trade-off isn't worth it.

'We tried clustering on three columns and our metadata exploded. Now we can't list partitions without a timeout.'

— Data engineer at a mid-stage analytics startup, after a weekend migration

Avoid the temptation to over-partition

That sounds fine until someone partitions on `user_id` because "users are the main filter." Wrong order. A cardinality that high creates thousands of tiny files—metadata bloat, tombstone accumulation, and listing times that climb from milliseconds to seconds. What works is partitioning on low-cardinality boundaries (date, region, event_type) and then sorting within those partitions for high-cardinality filters. I fixed a team's lake by collapsing 200 daily partitions into 12 monthly ones and adding a sort on `user_id`. Query speed stayed flat. Metadata size dropped 80%. Not yet convinced? Run the experiment on a small subset—one month of data—and watch the partition listing time in your engine's query profile. If listing becomes a bottleneck, you have your answer.

Most teams skip this: test with a 1TB table before touching 100TB. If the new format passes the listing test and the scan-reduction test, then—and only then—write the migration script. Otherwise, you'll repeat the same partitioning mistakes in a shinier format.