英文标题：When to Align, When to Predict: A Phase Diagram for Multimodal Learning
作者：Ilay Kamai, Hugues Van Assel, Aviv Regev, Hagai B. Perets, Randall Balestriero
arXiv ID：2606.11190 | 分类：cs.LG | 发表：2026-09-08
许可：CC-BY
摘要 跨模态对齐（CA）与跨模态预测（CP）是多模态表示学习的主导范式，然而对于二者各自何时成功、何时失败，尚缺乏系统性的理解——这一空白使得实践者，尤其是在仪器异构、测量层次多样的科学领域中，无法诊断为何标准方法的表现反而不如最佳单一模态。我们在带有结构化跨模态干扰相关性的尖峰信号加噪声模型下研究这两个目标，该干扰相关性正是打破经典恢复保证的关键因素，并推导出分离比，揭示出互补的失败模式：对齐会对每个模态进行白化，当干扰在各视图间强相关时便会失败；预测通过单侧白化编码一切可跨模态预测的信息，其恢复由源模态质量决定。由此得到的相图将多模态问题划分为四个区域——两者皆可（Both）、仅CA（CA
Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds and when each fails --- a gap that leaves practitioners, especially in scientific domains with heterogeneous instruments and multiple levels of measurement, unable to diagnose why standard methods underperform the best single modality. We study both objectives under a spiked signal-plus-noise model with structured cross-modal nuisance correlation, the ingredient that breaks the classical recovery guarantees, and derive separation ratios that expose complementary failure modes: alignment whitens each modality and fails when nuisance is strongly correlated across views; prediction encodes whatever is cross-predictable through a one-sided whitening, with recovery governed by source-modality quality. The resulting phase diagram partitions multimodal problems into four regimes --- Both, CA only, CP only, and Neither --- refined by a recovery count that separates partial recovery from complete failure. We present a data-driven procedure to locate real-world datasets in this diagram using a small labeled subsample, identifying the preferred objective and prediction direction before any cross-modal training, and identifying when no objective in the CA/CP family can improve on the stronger modality alone. Experiments on synthetic data, stereo-vision benchmarks, image--caption pairs, and two real scientific domains --- astronomy and single-cell multi-omics --- validate the predictions in the nonlinear regime, including both faces of the Neither regime. Code to reproduce the results is available at https://github.com/IlayMalinyak/mm_align_vs_pred.
查看完整双语翻译 →
正在跳转到翻译阅读页… 如果没有自动跳转，请点击这里。