This paper proposes an efficient lightweight deep spatial residual autoencoder (SRAE) model to detect anomalous events in video surveillance systems. A lightweight network is essential in real-time situations where time is critical. Moreover, it could be deployed on low-resource devices like embedded systems or mobile devices. This makes it a very useful option for real-world situations where there may be a shortage of resources. The proposed network is composed of a 3-layer residual encoder-decoder architecture that is adopted to acquire the salient spatial characteristics representative of normal events in videos. Then, the reconstruction loss is used to find abnormalities, where normal frames are recreated well with a low reconstruction loss, and abnormal ones are found as the opposite. The model's efficiency is tested by two benchmark datasets, the UCSD Pedestrian2 and the CUHK Avenue, achieving AUC ≈ 95% and 81% for the two datasets, respectively. Hence, its performance proves to be comparable with the state-of-the-art models.