WORKSHOP PAPER
SLIM: Small and Learnable Image Signal Processing Module for CMOS and Quanta Image Sensors
Stanley H. Chan1, Yiheng Chi1, Preston Rahim1
1DeepLux Technology Inc., West Lafayette, IN 47906, USA

Abstract

While multibit Quanta Image Sensors (QIS) today have demonstrated a superior sub-electron read noise characteristic, at extreme photon-limited conditions they still face the fundamental photon shot noise problem. The image signal processing (ISP) unit of today's multibit QIS is largely identical to those used for CMOS image sensors. In extreme photon-limited conditions, these physics-based ISP struggle to generate high-quality images. Deep learning methods are seen as the potential solution to overcome the low-light bottleneck, but existing neural networks are too large to fit into any camera products. In this paper, we present a learning-based ISP where key components are replaced by a lightweight neural network followed by traditional physics-based filtering steps. The proposed ISP, known as the Small and Learnable ISP Module (SLIM), allows us to jointly demosaick and denoise images at a photon level as low as 1 photon per pixel where traditional ISP fails.
Publisher: IISS (Int. Image Sensors Society)
Year: 2023
Workshop: IISW
URL: https://doi.org/10.60928/tdqh-lxyd
Download PDF

Keywords

Image Signal Processing, Quanta Image Sensors, Deep Learning,

References

1) Y. Chi, A. Gnanasambandam, V. Koltun, and S. H. Chan, "Dynamic low-light imaging with quanta image sensors", Proc. European Conf. Computer Vision, 2020. https://doi.org/10.1007/978-3-030-58589-1_8
2) O. A. Elgendy, A. Gnanasambandam, S. H. Chan, and J. Ma, "Low-light demosaicking and denoising for small pixels using learned frequency selection", IEEE Trans. Computational Imaging, 2021. https://doi.org/10.1109/tci.2021.3052694
3) P. Getreuer, I. Garcia-Dorado, J. Isidoro, S. Choi, F. Ong, and P. Milanfar, "BLADE: Filter learning for general purpose computational photography", IEEE International Conference on Computational Photography (ICCP), 2018. https://doi.org/10.1109/iccphot.2018.8368476
4) A. Gnanasambandam, O. Elgendy, J. Ma, and S. H. Chan, "Megapixel photon-counting color imaging using quanta image sensor", OSA Optics Express, 2019. https://doi.org/10.1364/oe.27.017298
5) B.-H. Kim, J. Song, J. C. Ye, and J. Baek, "Pynet-ca: enhanced pynet with channel attention for end-to-end mobile image signal processing", Proc. European Conf. Computer Vision, 2020. https://doi.org/10.1007/978-3-030-67070-2_12
6) J. Ma, D. Zhang, O. A. Elgendy, and S. Masoodian, "A 0.19e- rms read noise 16.7mpixel stacked quanta image sensor with 1.1 um-pitch backside illuminated pixels", IEEE Electron Device Letters, 2021. https://doi.org/10.1109/led.2021.3072842
7) Y. Romano, J. Isidoro, and P. Milanfar, "RAISR: Rapid and accurate image super resolution", IEEE Trans. Computational Imaging, 2016. https://doi.org/10.1109/tci.2016.2629284
8) S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, "Restormer: Efficient transformer for high-resolution image restoration", Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, 2022. https://doi.org/10.1109/cvpr52688.2022.00564
9) S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, "Learning enriched features for real image restoration and enhancement", Proc. European Conf Computer Vision, 2020. https://doi.org/10.1007/978-3-030-58595-2_30
10) K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, "Plug-and-play image restoration with deep denoiser prior", IEEE Trans. Pattern Analysis and Machine Intelligence, 2021. https://doi.org/10.1109/tpami.2021.3088914