WORKSHOP PAPER
A Study on a Feature Extractable CMOS Image Sensor for On-Chip Image Classification
Abstract
In the emerging IoT era, a CMOS image sensor (CIS) that can output features required for AI recognition is expected to reduce the power consumption of image recognition systems. In this paper, we propose an on-chip signal processing pipeline to extract two channels of feature data composed of 1-bit grayscale intensity and horizontal edges for a 2.0T pixel CMOS image sensor. Additionally, a tiny neural network model for on-chip AI processing is verified to examine image classification accuracy. According to simulation results, the image classification accuracy with the feature data reached 79.8%, even though the effective data volume of the feature data is reduced to only 0.7% of that of the 8-bit RGB color image. The memory occupation flow for a 2 Mpixel 60 fps CIS is also estimated to consider the feasibility of implementing an on-chip image classifier. Around 60 kB of peak memory and weight memory are additionally required for the processing, which is only 17 times the line memory required to serialize the RGB color image.Keywords
Image classification, Artificial neural networks, CMOS image sensors, Feature extraction,References
1) C. Young, A. Omid-Zohoor, P. Lajevardi, and B. Murmann, "A data-compressive 1.5/2.75-bit log-gradient qvga image sensor with multi-scale readout for always-on object detection", IEEE Journal of Solid-State Circuits, 2019. https://doi.org/10.1109/jssc.2019.2937437
2) S. Yoneda, Y. Negoro, H. Kobayashi, K. Nei, T. Takeuchi, M. Oota, T. Kawata, T. Ikeda, and S. Yamazaki, "Image sensor capable of analog convolution for real-time image recognition system using crystalline oxide semiconductor fet"
3) M.-J. Park and H.-J. Kim, "A real-time edge-detection cmos image sensor for machine vision applications", IEEE Sensors Journal, 2023. https://doi.org/10.1109/jsen.2023.3263461
4) T. Finateu, A. Niwa, D. Matolin, K. Tsuchimoto, A. Mascheroni, E. Reynaud, P. Mostafalu, F. Brady, L. Chotard, F. LeGoff, H. Takahashi, H. Wakabayashi, Y. Oike, and C. Posch, "5.10 a 1280×720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86µm pixels, 1.066geps readout, programmable event-rate controller and compressive data-formatting pipeline", 2020 IEEE International Solid-State Circuits Conference - (ISSCC), 2020. https://doi.org/10.1109/isscc19947.2020.9063149
5) Y. Morikaku, R. Ujiie, D. Morikawa, H. Shima, K. Yoshida, and S. Okura, "On-chip data reduction and object detection for a feature-extractable cmos image sensor", Electronics, 2024. https://doi.org/10.3390/electronics13214295
6) K. Kuroda, Y. Morikaku, Y. Osuka, R. Iegaki, K. Yoshida, and S. Okura, "Lightweight object detection model for a cmos image sensor with binary feature extraction", 2024 IEEE SENSORS, 2024. https://doi.org/10.1109/sensors60989.2024.10784814
7) A. Chowdhery, P. Warden, J. Shlens, A. Howard, and R. Rhodes, "Visual wake words dataset", 2019
8) J. Yoo, D. Lee, C. Son, S. Jung, B. Yoo, C. Choi, J.-J. Han, and B. Han, "Rascanet: Learning tiny models by raster-scanning images", 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. https://doi.org/10.1109/cvpr46437.2021.01346
9) C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, "Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors", 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. https://doi.org/10.1109/cvpr52729.2023.00721