feat: support v3 column default values in UpdateSchema (3/4)

[huan233usc]

What

Part 3 of 4 of Iceberg v3 column default-value support (POC #731), built on the
schema-layer support merged in #746. Independent of the read-path PRs (#792 and
the Avro follow-up).

Adds default-value handling to UpdateSchema (schema evolution).

Changes

• AddColumn / AddRequiredColumn take an optional default_value. When
  provided it is set as both the column's initial-default and write-default.
  A non-null default also lets a required column be added without
  AllowIncompatibleChanges() — rows written before the change read the default
  instead of null.
• UpdateColumnDefault(name, default) (new) sets, or clears with
  std::nullopt, a column's write-default; the initial-default is fixed when
  the column is added.
• Defaults are cast to the column type (rejecting uncastable or out-of-range
  values) and preserved across rename / doc / type-promotion updates and
  nested field-id reassignment.
• RequireColumn may now mark a column that was added with a default required.

The SchemaField constructor stores defaults verbatim (it does not coerce
them), so the cast/promotion is performed explicitly at each evolution site —
the same effect as Java, where NestedField's constructor runs castDefault.
Same-scale decimal precision widening is handled directly (the unscaled value is
unchanged), since Literal::CastTo does not cast between decimal types.

Tests

13 cases in update_schema_test.cc: add optional/required/nested column with a
default, mismatched/narrowing rejection, UpdateColumnDefault
(set / clear / cast-to-type / pre-existing column), require-after-default, and
preservation across doc updates and type promotion — including same-scale
decimal precision promotion.

Stack

1. feat(schema): represent, serialize and validate v3 column default values (1/4) #746 — schema: represent / serialize / validate (merged)
2. feat(parquet): apply column default values when reading missing fields (2/4) #792 — read path: Parquet
3. read path: Avro (follows)
4. this PR — schema evolution: addColumn / updateColumnDefault

[manuzhang]

For map/list nested adds, AddColumnInternal stores the new field in added_name_to_id_ using the canonical internal path:

  added_name_to_id_[CaseSensitivityAwareName(full_name)] = new_id;

where full_name becomes something like:

  locations.value.alt
  points.element.z

But RequireColumn receives and checks the user-facing shorthand path:

  added_name_to_id_.contains(CaseSensitivityAwareName(name))

where name is typically:

  locations.alt
  points.z

So the lookup misses, even though it refers to the same added column

[huan233usc]

Thanks for catching this -- yes it stored only the canonical full_name but callers use the short name. Fixed by also indexing the short name (matching how the schema name index handles map value / list element paths), with a regression test for requiring a defaulted nested column by short name in the same transaction.

It seems that same bug exists in Java; working on a fix apache/iceberg#17034.

[wgtmac]

This PR overall looks great. I just found a few issues while reviewing this PR and pushed fixes directly to avoid more back-and-forth:

• Removed the added TableMetadataV3Valid.json test fixture and generate v3 metadata in the unit test instead. We don't want to give the confusion that this json file is copied from other repo.
• Moved default literal casting out of update_schema.cc into SchemaField, closer to Java’s NestedField handling.
• Added coverage for default casting, decimal widening, null/range sentinel failures, and UpdateColumnDefault no-op behavior.
• Fixed UpdateColumnDefault to match Java behavior when the new write default is already set.
• Added small SchemaField::With* helpers to make schema field updates less error-prone.
• Cleaned up overly verbose comments.

Thanks @huan233usc for working on this and @manuzhang for the great review feedback!