fix: Emit stats for all partitions including empty/deleted and tables without readable_metrics

[srawat98-dev]

Summary

Three fixes to ensure partition stats (S3) are emitted reliably for all tables and partitions:

1. Emit rowCount/columnCount when readable_metrics is empty — affects both partitioned and unpartitioned tables
2. Use LEFT JOIN to retain deleted/empty partitions — ensures partitions with 0 rows after DELETE/truncate still emit stats with rowCount=0

Root Cause

In TableStatsCollectorUtil:

• Both populateStatsForPartitionedTable() and populateStatsForUnpartitionedTable() returned Collections.emptyList() when getColumnNamesFromReadableMetrics() returned empty — skipping stats emission entirely, even though rowCount (from sum(record_count)) and columnCount (from table.schema()) are always independently available.
• joinStatsWithCommitMetadata() used INNER JOIN between latest commits and data_files stats — silently dropping partitions that have commits but no current data files (e.g., after DELETE/truncate operations).

Impact

• Downstream DQ pipeline permanently had null rowCount/columnCount for affected tables (first-arrival stats are immutable once set)
• ~34% of commits for certain unpartitioned tables had no S3 entry despite valid S1/S2 events
• Partition DELETE/truncate events were invisible to downstream consumers

Changes

• Bug Fixes
• Tests

Fix 1: Empty readable_metrics (partitioned + unpartitioned)

• Removed early return in both populateStatsForPartitionedTable() and populateStatsForUnpartitionedTable() when columnNames is empty
• Handle empty columnAggExpressions in SQL query builders to avoid invalid SQL with trailing comma
• Column-level metrics (nullCount, nanCount, min, max, columnSize) will be null when readable_metrics is unavailable — correct behavior

Fix 2: LEFT JOIN for deleted partitions

• Changed joinStatsWithCommitMetadata() from INNER to LEFT join
• Partitions with commits but no current data_files now emit stats with rowCount=0
• Downstream transform already handles this: rowCount != null ? rowCount : 0L

Test: Deleted partition coverage

• Added testPartitionStatsForDeletedPartition: creates partitioned table with 2 partitions, deletes all rows from one, verifies deleted partition emits stats with rowCount=0

Testing Done

• Added new tests for the changes made.

• Updated existing tests to reflect the changes made.

• Some other form of testing like staging or soak time in production. Please explain.

• New test: testPartitionStatsForDeletedPartition — verifies LEFT JOIN retains deleted partitions with rowCount=0

• Existing integration test testPartitionStatsForUnpartitionedTable continues to pass

• Existing unit test testBuildColumnAggregationExpressions_withEmptyList confirms empty list produces empty expressions

• extractColumnMetricsFromAggregatedRow with empty columnNames returns empty metric lists (no iteration) — safe

• transformRowsToPartitionStatsFromAggregatedSQL already null-guards total_row_count with fallback to 0L

• Production trace: null stats for u_daliview.non_hive_migration_hdfs_path_to_table_phase_mapping_war on holdem root-caused to the early return path

Additional Information

No breaking changes. Fully backward compatible:

• Tables with readable_metrics: unchanged behavior
• Tables without readable_metrics: now emit stats with valid rowCount/columnCount, null column-level metrics (previously silently skipped)
• Deleted partitions: now emit stats with rowCount=0 (previously silently dropped by INNER JOIN)