Approximation Capabilities of Neural ODEs and Invertible Residual Networks

Approximation Capabilities of Neural ODEs and Invertible Residual Networks

Neural ODEs and i-ResNet are recently proposed methods for enforcing invertibility of residual neural models. Having a generic technique for constructing invertible models can open new avenues for advances in learning systems, but so far the question of whether Neural ODEs and i-ResNets can model any continuous invertible function remained unresolved.Here, we show that both of these models are limited in their approximation capabilities. We then prove that any homeomorphism on a $p$-dimensional Euclidean space can be approximated by a Neural ODE operating on a $2p$-dimensional Euclidean space, and a similar result for i-ResNets. We conclude by showing that capping a Neural ODE or an i-ResNet with a single linear layer is sufficient to turn the model into a universal approximator for non-invertible continuous functions.

神经ODE和可逆残差网络的逼近能力

神经ODE和i-ResNet是最近提出的用于增强残差神经模型的可逆性的方法。拥有一种通用的构造可逆模型的技术可以为学习系统的发展开辟新的途径，但到目前为止，神经ODE和i-ResNets是否可以对任何连续的可逆函数建模的问题仍未解决。.. 在这里，我们表明这两个模型的逼近能力均受到限制。然后，我们证明对 p 欧几里德空间可以通过在 2p 维欧几里得空间，以及i-ResNets的相似结果。通过得出结论，表明用单个线性层覆盖神经ODE或i-ResNet足以将模型转换为不可逆连续函数的通用逼近器。 （阅读更多）