My thinking about promoting the AI Development

Hi There! This is Haoran, an undergraduate at Drexel University studying Data Science and doing research in the area of AI. More specifically, I do research under NLP/Computational Linguistics & Machine Learning.

Recently, I’ve had some insights and thinking/thoughts about current situation of AI and its future development. AI stands for Artificial Intelligence; its ultimate goal is to let artificial systems achieve real human-level intelligence. In this sense, only scaling the models’ size and enlarge the training dataset size will be really hard to reach there. So, it brings me to think about how to promote the AI area to the higher and more advanced stage.

The term “AI” was first coined by John McCarthy who is a computer scientist and cognitive scientist in the mid-1950s. As an interdisciplinary field, it embraces multi-disciplines into this area. However, as I’ve observed, so far computer science has been the dominant subject studying AI. The vast majority of researchers come into this field with a CS background. From my perspective, to promote and accelerate the AI revolution, we need to embrace other fields, such as cognitive science, psychology & neuroscience more into the AI field and cultivate new researchers (including myself) with interdisciplinary backgrounds than ever before.

Looking back over the last decades, after the end of the AI winter, the development of AI ushered in its spring. Tons of new ideas have come out by awesome researchers across the world. With computing power, big and high-quality dataset, and efficient algorithms all become feasible and applicable, seminal research papers and findings have been continuously emerging. From ImageNet (Deng et al. 2009) to AlexNet (Alex et al. 2012) to ResNet (He et al. 2016), , Computer Vision has achieved or overperformed some human-level intelligence and started to integrate into our daily application and lives. From CBOM & Skip-Gram (Mikolov et al. 2013) to Attention Mechanism (Bahdanau et al. 2015) to Transformer (Vaswani et al. 2017) to BERT (Devlin et al. 2018), Natural Language Processing also has successfully got the breakthrough to reach human-level intelligence in some tasks and been becoming an essential part of our daily techniques realm.

The path of AI development looks smooth and meets no obstacles in the past decade. However, it looks like the continuous development of AI/NLP has been kind of stagnant recently, especially since Google invented Transformer and BERT. (Because of my NLP background, I will talk more towards the NLP side. Please bear me and point out if I said something wrong and glad to hear your voice). In the past five years, a series of Transformer-based models has been created and relevant works have been done. Pre-trained large language models with few-shot prompting becomes the new paradigm for tackling a broad range of NLP-related tasks. This is amazing and really useful for NLP applications. But no significant improvement of model architecture (algorithm side) has been done; everything is still transformer-based. And computing power has become a more and more obvious constraint/obstacle with model-scale becoming larger and larger to achieve higher performances. Of course, the effort and work get put in is tremendous and there are many amazing ideas that come out along the way, but the actual effect to push the field forward is limited mostly. In contrast, seminal and original work of developing new and more efficient algorithms is indeed the hand that can actually push the field forward in my perspective.

So, it brings me to think about what this new generation of researchers (current students) can do to push this AI gigantic boat forward. And it confirms my thinking after reading the paper “Towards Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution” (Zador et al. 2022). They mention a new term “NeuroAI” in this paper which I got really excited about. I saw on twitter about the discussion about this paper. Even though some researchers doubt the real role neuroscience plays in the development of AI, I always firmly believe it’s a big factor in accelerating AI development.

I remembered last winter, back then, I just came to the States for about half a year and didn’t even know what my real research interest was, specifically with only a vague idea of NLP being the broad area. One night, I got an immature idea that I was wondering why don’t people dedicate their research time to try to figure out how the brain actually processes language using techniques such as fMRI and leverage the findings to develop the next-generation language model’s architecture. The day after, I looked up relevant information on the Internet and it turned out not many works in related fields have been done. It brought me the next question: why? With some conversations with professors, I found some clues. It seems the tools to study neuroscience are not so advanced to detect the interaction between neurons when humans perform some certain tasks. I don’t know if it’s true since I had no neuroscience background yet but I will. Nonetheless, it brought me some deeper thoughts towards AI’s future, specifically more about NLP.

In my opinion, study about human cognition and neuroscience will help AI practitioners at macro level and micro level better understand how a real human processes input and generates output to react to the surrounding, respectively. Then researchers could use computational tools to encode discoveries and findings in a way that can build better neural network structures.

About NLP: I think as a NLP researcher, neurolinguistics and psycholinguistics are the two areas that we need to learn more about. Understanding how languages and our cognition interact together and at a micro level trying to detect how some certain neurons connect and interact with each other will bring a new angle to do NLP tasks and promote its further development. Bringing Linguistics, Cognitive Science, Psychology, Neuroscience, NLP, Computer Visioin together to flourish NLP research and let computers achieve better performance on language-related tasks.

Towards further research about AI and to let artificial systems reach a level of intelligence closer to humans, bringing more disciplines together and reaching a brighter future of AI.

References

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Third International Conference on Learning Representations, 2015.

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks.In Advances in neural information processing systems, 2012. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, 2013b.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 2009.

Anthony Zador, Blake Richards, Bence Ölveczky, Sean Escola, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, James DiCarlo, Surya Ganguli, Jeff Hawkins, Konrad Koerding, Alexei Koulakov, Yann LeCun, Timothy Lillicrap, Adam Marblestone, Bruno Olshausen, Alexandre Pouget, Cristina Savin, Terrence Sejnowski, Eero Simoncelli, Sara Solla, David Sussillo, Andreas S. Tolias, Doris Tsao. Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution. arXiv preprint arXiv:2210.08340, 2022.