Publications

2024

1. 

   How Language Models Prioritize Contextual Grammatical Cues?

   Hamidreza Amirzadeh, Afra Alishahi , and Hosein Mohebbi

   In The 7th BlackboxNLP Workshop at EMNLP , 2024

   Abs Bib PDF

   Transformer-based language models have shown an excellent ability to effectively capture and utilize contextual information. Although various analysis techniques have been used to quantify and trace the contribution of single contextual cues to a target task such as subject-verb agreement or coreference resolution, scenarios in which multiple relevant cues are available in the context remain underexplored. In this paper, we investigate how language models handle gender agreement when multiple gender cue words are present, each capable of independently disambiguating a target gender pronoun. We analyze two widely used Transformer-based models: BERT, an encoder-based, and GPT-2, a decoder-based model. Our analysis employs two complementary approaches: context mixing analysis, which tracks information flow within the model, and a variant of activation patching, which measures the impact of cues on the model’s prediction. We find that BERT tends to prioritize the first cue in the context to form both the target word representations and the model’s prediction, while GPT-2 relies more on the final cue. Our findings reveal striking differences in how encoder-based and decoder-based models prioritize and use contextual information for their predictions.

   @inproceedings{amirzadeh2024how,
     title = {How Language Models Prioritize Contextual Grammatical Cues?},
     author = {Amirzadeh, Hamidreza and Alishahi, Afra and Mohebbi, Hosein},
     booktitle = {The 7th BlackboxNLP Workshop at EMNLP},
     year = {2024},
     url = {https://openreview.net/forum?id=XDxNnRHGRp},
   }

2. 

   data2lang2vec: Data Driven Typological Features Completion

   Hamidreza Amirzadeh, Sadegh Jafari , Anika Harju , and Rob Goot

   2024

   Abs Bib PDF

   Language typology databases enhance multi-lingual Natural Language Processing (NLP) by improving model adaptability to diverse linguistic structures. The widely-used lang2vec toolkit integrates several such databases, but its coverage remains limited at 28.9%. Previous work on automatically increasing coverage predicts missing values based on features from other languages or focuses on single features, we propose to use textual data for better-informed feature prediction. To this end, we introduce a multi-lingual Part-of-Speech (POS) tagger, achieving over 70% accuracy across 1,749 languages, and experiment with external statistical features and a variety of machine learning algorithms. We also introduce a more realistic evaluation setup, focusing on likely to be missing typology features, and show that our approach outperforms previous work in both setups.

   @misc{amirzadeh2024data2lang2vecdatadriventypological,
     title = {data2lang2vec: Data Driven Typological Features Completion},
     author = {Amirzadeh, Hamidreza and Jafari, Sadegh and Harju, Anika and van der Goot, Rob},
     year = {2024},
     journal = {arXiv preprint arXiv:2409.17373},
     eprint = {2409.17373},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
     url = {https://arxiv.org/abs/2409.17373},
   }

3. 

   HalluSafe at SemEval-2024 Task 6: An NLI-based Approach to Make LLMs Safer by Better Detecting Hallucinations and Overgeneration Mistakes

   Zahra Rahimi , Hamidreza Amirzadeh, Alireza Sohrabi , Zeinab Taghavi , and Hossein Sameti

   In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024) , Jun 2024

   Abs Bib PDF

   The advancement of large language models (LLMs), their ability to produce eloquent and fluent content, and their vast knowledge have resulted in their usage in various tasks and applications. Despite generating fluent content, this content can contain fabricated or false information. This problem is known as hallucination and has reduced the confidence in the output of LLMs. In this work, we have used Natural Language Inference to train classifiers for hallucination detection to tackle SemEval-2024 Task 6-SHROOM (Mickus et al., 2024) which is defined in three sub-tasks: Paraphrase Generation, Machine Translation, and Definition Modeling. We have also conducted experiments on LLMs to evaluate their ability to detect hallucinated outputs. We have achieved 75.93% and 78.33% accuracy for the modelaware and model-agnostic tracks, respectively.

   @inproceedings{rahimi-etal-2024-hallusafe,
     title = {{H}allu{S}afe at {S}em{E}val-2024 Task 6: An {NLI}-based Approach to Make {LLM}s Safer by Better Detecting Hallucinations and Overgeneration Mistakes},
     author = {Rahimi, Zahra and Amirzadeh, Hamidreza and Sohrabi, Alireza and Taghavi, Zeinab and Sameti, Hossein},
     editor = {Ojha, Atul Kr. and Do{\u{g}}ru{\"o}z, A. Seza and Tayyar Madabushi, Harish and Da San Martino, Giovanni and Rosenthal, Sara and Ros{\'a}, Aiala},
     booktitle = {Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)},
     month = jun,
     year = {2024},
     address = {Mexico City, Mexico},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.semeval-1.22},
     doi = {10.18653/v1/2024.semeval-1.22},
     pages = {139--147},
   }

2023

1. 

   Imaginations of WALL-E : Reconstructing Experiences with an Imagination-Inspired Module for Advanced AI Systems

   Zeinab Sadat Taghavi , Soroush Gooran , Seyed Arshan Dalili , Hamidreza Amirzadeh, Mohammad Jalal Nematbakhsh , and Hossein Sameti

   arXiv preprint arXiv:2308.10354, Jun 2023

   Abs Bib PDF

   In this paper, we introduce a novel Artificial Intelligence (AI) system inspired by the philosophical and psychoanalytical concept of imagination as a “Re-construction of Experiences". Our AI system is equipped with an imagination-inspired module that bridges the gap between textual inputs and other modalities, enriching the derived information based on previously learned experiences. A unique feature of our system is its ability to formulate independent perceptions of inputs. This leads to unique interpretations of a concept that may differ from human interpretations but are equally valid, a phenomenon we term as “Interpretable Misunderstanding". We employ large-scale models, specifically a Multimodal Large Language Model (MLLM), enabling our proposed system to extract meaningful information across modalities while primarily remaining unimodal. We evaluated our system against other large language models across multiple tasks, including emotion recognition and question-answering, using a zero-shot methodology to ensure an unbiased scenario that may happen by fine-tuning. Significantly, our system outperformed the best Large Language Models (LLM) on the MELD, IEMOCAP, and CoQA datasets, achieving Weighted F1 (WF1) scores of 46.74%, 25.23%, and Overall F1 (OF1) score of 17%, respectively, compared to 22.89%, 12.28%, and 7% from the well-performing LLM. The goal is to go beyond the statistical view of language processing and tie it to human concepts such as philosophy and psychoanalysis. This work represents a significant advancement in the development of imagination-inspired AI systems, opening new possibilities for AI to generate deep and interpretable information across modalities, thereby enhancing human-AI interaction.

   @article{taghavi2023imaginations,
     title = {Imaginations of WALL-E : Reconstructing Experiences with an Imagination-Inspired Module for Advanced AI Systems},
     author = {Taghavi, Zeinab Sadat and Gooran, Soroush and Dalili, Seyed Arshan and Amirzadeh, Hamidreza and Nematbakhsh, Mohammad Jalal and Sameti, Hossein},
     year = {2023},
     journal = {arXiv preprint arXiv:2308.10354},
     eprint = {2308.10354},
     archiveprefix = {arXiv},
     primaryclass = {cs.AI},
   }