WORKSHOP PAPER
Histogram-less direct time-of-flight imaging based on a machine learning processor on FPGA
Abstract
The investigation of a novel architecture for direct time-of-flight (TOF) SPAD based imaging systems is presented. In the proposed architecture, a pulsed laser source illuminates a scene and the reflected light is captured by a SPAD, which detects photons and converts them to a digital pulse. Unlike time-correlated single-photon counting (TCSPC), where each detected photon generates a timestamp that is organized in a histogram, in this architecture, timestamps are fed to a machine-learning processor (MLP) that is trained to recognize SPAD responses in direct TOF. The MLP generates the distance to the target directly, addressing potential non-idealities in timestamp generation and processing. The architecture demonstrated practical application in scenes and reported its performance using standard LiDAR characterization methods.Keywords
Direct Time-of-Flight, SPAD, Machine Learning,References
1) A. R. Ximenes, P. Padmanabhan, M.-J. Lee, Y . Yamashita, D.-N. Yaung, and E. Charbon, "A 256 ×256 45/65nm 3d-stacked spad-based direct tof image sensor for lidar applications with optical polar modulation for up to 18.6 db interference suppression", 2018 IEEE International Solid-State Circuits Conference-(ISSCC), 2018. https://doi.org/10.1109/isscc.2018.8310201
2) P. Padmanabhan, C. Zhang, M. Cazzaniga, B. Efe, A. R. Ximenes, M.-J. Lee, and E. Charbon, "A 256 ×128 3d-stacked (45nm) spad flash lidar with 7-level coincidence detection and progressive gating for 100m range and 10klux background light", 2021 IEEE International Solid-State Circuits Conference (ISSCC), vol. 64, 2021. https://doi.org/10.1109/isscc42613.2021.9366010
3) I. Gyongy, N. A. Dutton, and R. K. Henderson, "Direct time-of-flight single-photon imaging", IEEE Transactions on Electron Devices, vol. 69, no. 6, 2021. https://doi.org/10.1109/ted.2021.3131430
4) G. Chen, C. Wiede, and R. Kokozinski, "Data processing approaches on spad-based d-tof lidar systems: A review", IEEE Sensors Journal, vol. 21, no. 5, 2020. https://doi.org/10.1109/jsen.2020.3038487
5) C. Zhang, S. Lindner, I. M. Antolovi ´c, J. M. Pavia, M. Wolf, and E. Charbon, "A 30-frames/s, 252×144 spad flash lidar with 1728 dual-clock 48.8-ps tdcs, and pixel-wise integrated histogramming", IEEE Journal of Solid-State Circuits, vol. 54, no. 4, 2018. https://doi.org/10.1109/jssc.2018.2883720
6) S. Hochreiter and J. Schmidhuber, "Long short-term memory", Neural computation, vol. 9, no. 8, 1997. https://doi.org/10.1162/neco.1997.9.8.1735
7) S. A. Mirsalari, N. Nazari, S. A. Ansarmohammadi, S. Sinaei, M. E. Salehi, and M. Daneshtalab, "Elc-ecg: Efficient lstm cell for ecg classification based on quantized architecture", 2021 IEEE International Symposium on Circuits and Systems (ISCAS), 2021. https://doi.org/10.1109/iscas51556.2021.9401261
8) D. Kadetotad, S. Yin, V. Berisha, C. Chakrabarti, and J.-s. Seo, "An 8.93 tops/w lstm recurrent neural network accelerator featuring hierarchical coarse-grain sparsity for on-device speech recognition", IEEE Journal of Solid-State Circuits, vol. 55, no. 7, 2020. https://doi.org/10.1109/jssc.2020.2992900
9) A. Aßmann, B. Stewart, and A. M. Wallace, "Deep learning for lidar waveforms with multiple returns", 2020 28th European Signal Processing Conference (EUSIPCO), 2021. https://doi.org/10.23919/eusipco47968.2020.9287545
10) J. Zhao, T. Milanese, F. Gramuglia, P. Keshavarzian, S. S. Tan, M. Tng, L. Lim, V. Dhulla, E. Quek, M.-J. Lee et al., "On analog silicon photomultipliers in standard 55-nm bcd technology for lidar applications", IEEE Journal of Selected Topics in Quantum Electronics, vol. 28, no. 5, 2022. https://doi.org/10.1109/jstqe.2022.3161089