WORKSHOP PAPER
Designing task-oriented cameras by machine learning
Abstract
Deep neural network (DNN) is a powerful tool for solving image processing and computer vision tasks such as image and video reconstructions, object recognition, and scene understanding, etc. However, DNN have been used for only digital domain in the imaging pipeline, such as the feature extractor and classifier models after an image is captured and digitized. In this research, we propose a new framework called “deep sensing.” The proposed framework also models the analog layer to the neural network model and jointly optimizes the parameters in optics and sensor designs of a camera, as well as reconstruction and classification models by the same training strategy.Keywords
Deep neural network, Computational photography, Compressive sensing,References
[1]) Y. Inagaki, Y. Kobayashi, K. Takahashi, T. Fujii, and H. Nagahara, "Learning to capture light fields through a coded aperture camera", European Conference on Computer Vision, pp. 418–434, 2018,., 2018. https://doi.org/10.1007/978-3-030-01234-2_26
[2]) M. Yoshida, A. Torii, M. Okutomi, K. Endo, Y. Sugiyama, R. Taniguchi, and H. Nagahara, "Joint optimization for compressive video sensing and reconstruction under hardware constraints", European Conference on Computer Vision, pp. 634–649, 2018., 2018. https://doi.org/10.1007/978-3-030-01249-6_39
[3]) T. Okawara, M. Yoshida, H. Nagahara, and Y. Yagi, "Action Recognition from a Single Coded Image", Proceedings of International Conference on Computational Photography, 2020., 2020. https://doi.org/10.1109/iccp48838.2020.9105176
[4]) K. Sakai, K. Takahashi, T. Fujii, H. Nagahara, "Acquiring Dynamic Light Fields through a Coded Aperture Camera", European Conference on Computer Vision, 2020., 2020. https://doi.org/10.1007/978-3-030-58529-7_22
[5]) R. Mizuno, K. Takahashi, M. Yoshida, C. Tsutake, T. Fujii, H. Nagahara, "Acquiring a Dynamic Light Field through a Single-Shot Coded Image", IEEE Conference on Computer Vision and Pattern Recognition, June, 2022., 2022. https://doi.org/10.1109/cvpr52688.2022.01921
[6]) S. Kumawat, T. Okawara, M. Yoshida, H. Nagahara, Yasushi Yagi, "Action Recognition From a Single Coded Imaging", IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-14, Aug., 2022., 2022. https://doi.org/10.1109/tpami.2022.3196350
[7]) S. Kumawat, H. Nagahara, "Privacy-Preserving Action Recognition via Motion Difference Quantization", European Conference on Computer Vision, Oct, 2022., 2022. https://doi.org/10.1007/978-3-031-19778-9_30
[8]) Shuji Habuchi, Keita Takahashi, Chihiro Tsutake, Toshiaki Fujii, Hajime Nagahara, "Time-Efficient Light-Field Acquisition Using Coded Aperture and Events", IEEE Conference on Computer Vision and Pattern Recognition, June, 2024., 2024. https://doi.org/10.1109/cvpr52733.2024.02354
[9]) Michitaka Yoshida, Daisuke Hayashi, Lioe De Xing, Keita Yasutomi, Shoji Kawahito, Keiichiro Kagawa, Hajime Nagahara, "Single-shot efficient depth imaging based on time-compressive CMOS image sensor", IEEE Sensors, Oct., 2024., 2024. https://doi.org/10.1109/sensors60989.2024.10784996