Enemy is Inside: Alleviating VAE's Overestimation in Unsupervised OOD Detection

Deep generative models (DGMs) aim at characterizing the distribution of the training set by maximizing the marginal likelihood of inputs in an unsupervised manner, making them a promising option for unsupervised out-of-distribution (OOD) detection. However, recent works have reported that DGMs often assign higher likelihoods to OOD data than in-distribution (ID) data, $\textit{i.e.}$, $\textbf{\textit{overestimation}}$, leading to their failures in OOD detection. Although several pioneer works have tried to analyze this phenomenon, and some VAE-based methods have also attempted to alleviate this issue by modifying their score functions for OOD detection, the root cause of the $\textit{overestimation}$ in VAE has never been revealed to our best knowledge. To fill this gap, this paper will provide a thorough theoretical analysis on the $\textit{overestimation}$ issue of VAE, and reveal that this phenomenon arises from two Inside-Enemy aspects: 1) the improper design of prior distribution; 2) the gap of dataset entropies between ID and OOD datasets. Based on these findings, we propose a novel score function to $\textbf{A}$lleviate $\textbf{V}$AE's $\textbf{O}$verestimation $\textbf{I}$n unsupervised OOD $\textbf{D}$etection, named $\textbf{``AVOID''}$, which contains two novel techniques, specifically post-hoc prior and dataset entropy calibration. Experimental results verify our analysis, demonstrating that the proposed method is effective in alleviating $\textit{overestimation}$ and improving unsupervised OOD detection performance.