Databricks ConcurrentAppendException on disjoint partitions? It's the isolation level

Two Databricks jobs write to different partitions of the same Delta table, and one still fails: ConcurrentAppendException: Files were added to partition ... by a concurrent update. The instinct — retry — loops. The real cause is the isolation level.

Why disjoint partitions still conflict

Delta Lake's default WriteSerializable isolation detects conflicts more coarsely than the partitions suggest. It allows certain concurrent writes to be reordered (serializable between them), and its conflict check looks at files added to the table — so two jobs on genuinely disjoint partitions can still collide if Delta can't prove the disjointness from your operation's predicates. Related exceptions on concurrent MERGE/DELETE/UPDATE: ConcurrentDeleteReadException, ConcurrentDeleteDeleteException (SQLSTATE 2D521 / DELTA_CONCURRENT_APPEND).

The fix

• Add explicit partition filters to the MERGE/UPDATE condition (merge on the partition column) so Delta can prove the writes are disjoint — this is what actually resolves it.
• Serialize the writers — run concurrent merges in the same task, or queue them, if you can't partition cleanly.
• Retry with capped exponential backoff only for genuinely transient conflicts (not indefinitely), and never retry a non-idempotent MERGE without dedup.
• Databricks 14.2+ row-level concurrency reduces false conflicts; on S3, use S3DynamoDBLogStore for safe multi-cluster writes.

Snapback's data-warehouse family catches this — diagnose_infra_error returns the partition-filter/isolation fix instead of the retry loop.