The research investigates the performance of autoencoders in the compression of structured data. Autoencoders are a prominent model in many branches of machine learning and data compression. The researchers prove that the gradient descent algorithm disregards the sparse structure of the input in the case of 1-bit compression of sparse Gaussian data, performing as if compressing a Gaussian source with no sparsity. The study also explores a phase transition phenomenon in the gradient descent minimizer as a function of data sparsity. It was found that adding a denoising function to a shallow architecture reduces the loss provably, and a suitable multi-layer decoder leads to further improvement. The findings are validated on image datasets like CIFAR-10 and MNIST.
Publication date: 8 Feb 2024
Project Page: Not Provided
Paper: https://arxiv.org/pdf/2402.05013
