Abstract

Single image rain streaks removal has recently witnessed substantial progress due to the development of deep convolutional neural networks. However, existing deep learning based methods either focus on the entrance and exit of the network by decomposing the input image into high and low frequency information and employing residual learning to reduce the mapping range, or focus on the introduction of cascaded learning scheme to decompose the task of rain streaks removal into multi-stages. These methods treat the convolutional neural network as an encapsulated end-to-end mapping module without deepening into the rationality and superiority of neural network design. In this paper, we delve into an effective end-to-end neural network structure for stronger feature expression and spatial correlation learning. Specifically, we propose a non-locally enhanced encoder-decoder network framework, which consists of a pooling indices embedded encoder-decoder network to efficiently learn increasingly abstract feature representation for more accurate rain streaks modeling while perfectly preserving the image detail. The proposed encoder-decoder framework is composed of a series of non-locally enhanced dense blocks that are designed to not only fully exploit hierarchical features fromall the convolutional layers but also well capture the long-distance dependencies and structural information. Extensive experiments on synthetic and real datasets demonstrate that the proposed method can effectively remove rainstreaks on rainy image of various densities while well preserving the image details, which achieves significant improvements over the recent state-of-the-art methods.

Framework

We introduce an end-to-end convolutional neural network for single image de-raining, called non-locally enhanced encoder-decoder network (NLEDN). Our framework contains a fully convolutional encoder-decoder network which has been proven able to learn complex pixel-wise mappings from large amount of input-output image pairs.

Particularly, in order to exploit the abundant structure cues in rain streak maps and the self-similarities in rainfree nature scenes, we propose the non-locally enhanced dense block (NEDB) as the basic component in our network architecture. We carefully integrate NEDBs with both encoding and decoding layers to enable the computation of long-range spatial dependencies as well as efficient usage of the feature activation of proceeding layers.

Experiment

We compare our method against five existing methods, including DSC [1], GMM [2], DDN [3], JORDER [4], DID-MDN [5].

Quantitative Comparisons

Visual Comparisons On Synthetic Datasets

Visual Comparisons On Real Datasets

Conclusion

In this paper, we have introduced a non-locally enhanced encoder-decoder network framework for rain streaks removal from single images. It is designed as a concatenation of an encoder network followed by a corresponding decoder network, which are both composed of a series of tailored, non-locally enhanced dense blocks (NEDB). The NEDB is designed to not only fully exploit hierarchical features from densely connected convolutional layers but also well capture the long-distance dependencies and structural information by employing a non-locally weighting operation at a specific range of feature maps. Experimental results on both synthetic and real datasets have demonstrated that our proposed method can effectively remove rain-streaks on rainy image of various density while promisingly preserve the image texture similar to the rain streaks, which greatly outperforms the state-of-the-art. In our future research work, we plan to extend the proposed algorithm to a wider range of image restoration tasks, including but not limited to image denoising, image dehazing and image super-resolution.

References

[1] Yu Luo, Yong Xu, and Hui Ji. 2015. Removing Rain from a Single Image via Discriminative Sparse Coding. In IEEE International Conference on Computer Vision. 3397–3405

[2] Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, and Michael S. Brown. 2016. Rain Streak Removal Using Layer Priors. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2736–2744.

[3] Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing Rain from Single Images via a Deep Detail Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1715–1723.

[4] Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep Joint Rain Detection and Removal from a Single Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[5] He Zhang and Vishal M. Patel. 2018. Density-aware Single Image De-raining using a Multi-stream Dense Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.