The success of deep learning is attributed mainly to the common-used datasets for the benchmark like ImageNet and COCO. DVS simulates the human visual nervous system from a specific perspective and level of abstraction which allows spiking neural networks (SNNs) to take full advantage of the temporal information provided by such sensors. To facilitate the development of brain-inspired artificial intelligence, especially SNNs, several neuromorphic datasets has been produced using dynamic vision sensor (DVS). For example, N-MNIST, N-caltech101, and DVS-CIFAR10 are obtained by capturing images or camera movements from traditional classification datasets, avoiding the corruption of the original image information during encoding. Besides, more researchers acquire event data from natural environments, such as DVS-Gesture and N-Cars. However, the existing datasets have very low temporal correlation and could not fully reflect and exploit the temporal information processing capability of SNNs. Their characteristics are shown in Table 1. In addition to sparse coding to reduce energy consumption, rapid learning of new concepts from a few samples is one of the essential capabilities of the human brain, but this remains an open problem in spiking neural network-based machine learning.
To address these issues and bridge this gap, we propose the first neuromorphic dataset for few-shot learning: the N-Omniglot. The original Omniglot dataset is widely used in few-shot learning. The N-Omniglot consists of 1,623 handwritten characters from 50 different languages. Each character has only 20 different samples. It is usually considered a static text image and ignores the rich temporal information in the writing process.
We note that SNNs is still in its infancy in neuromorphic few-shot learning. Although we have constructed N-Omniglot, very few existing algorithms are suitable for this task. All experiments have been performed on both N-Omniglot and encoded Omniglot, to demonstrate the difference between them. As shown in the Figure below, we treat static images with poisson and constant coding. Since the DVS camera for acquiring neuromorphic data has a high temporal resolution, the excessively long time axis is a huge burden for the current clock-driven SNN algorithms. Therefore, we process the event data with OR operations and firing rate. To demonstrate the effectiveness of N-Omniglot and the potential to provide new challenges for training SNN algorithms, we conducted experiments on four SNN algorithms, including adaptations or SNN versions of two general classical pattern classification methods and two few-shot learning algorithms.
N-Omniglot dataset is more sparse in the spatial dimension, and the data is less similar in temporal dimension than a static image or poisson coding-based inputs, which brings new challenges to spiking neural network learning. It is crucial to improve the ability of spiking neural network to extract more important spatio-temporal features. Therefore, N-Omniglot can be considered an effective, robust, and challenging neuromorphic dataset for developing future spiking neural networks.
Prof. Yi Zeng is the corresponding author of this article, Yang Li, Yiting Dong and Dr. Dongcheng Zhao are the co-first authors of this article.
For more details: https://doi.org/10.1038/s41597-022-01851-z