When Do Neural Networks Think Like Humans? Comparing Shallow and Deep CNN Representations on the Odd-One-Out Task
Using THINGS, a dataset of 4.7 million odd-one-out judgments from 12,340 participants, we compared layer-wise representations from a shallow CNN (AlexNet) and a deep CNN (ResNet-50) with human object representations, using odd-one-out accuracy on 50,000 triplets and RSA against human SPoSE embeddings. We expected the deeper ResNet-50 to align more closely with humans, and alignment to peak in mid-to-late layers. The second prediction held, but the shallower AlexNet outperformed ResNet-50 on both measures.
Layer-wise alignment with human judgments
Using representational similarity analysis (RSA), we correlated each layer's similarity matrix with a human similarity matrix built from SPoSE embeddings of the 1,854 THINGS concepts. Alignment generally rose in later layers of both models, peaking at AlexNet fc8 (r = .54) and ResNet-50 res5c (r = .42).
Odd-one-out accuracy
Agreement with human choices generally improved in later layers of both models, but stayed only moderately above chance (33.3%). AlexNet's final layer, fc8, scored highest (50.11%), ahead of ResNet-50's res5c (47.25%).
Category-level results
Performance also varied by object concept. Some concepts aligned best in late layers, such as coyote (0.911 in AlexNet fc8), while others reached high accuracy early, such as spinach (0.723 in AlexNet pool1).
Conclusion
This study examined how well AlexNet and ResNet-50 align with human object representations using the odd-one-out task. Human-like representational structure emerged most strongly in the mid-to-late layers of both networks, supporting the idea that deeper processing stages capture increasingly semantic object information. However, contrary to our original hypothesis, AlexNet had stronger alignment with human judgments than ResNet-50, achieving both higher accuracy and stronger RSA correlations in its final layers. These findings suggest that greater architectural complexity does not necessarily lead to representations that more closely resemble human conceptual organization. Our category-level analyses varied across object concepts, showing that representations emerge differently depending on the visual complexity and the layer of the model.
Overall, this work contributes to the growing body of research investigating the relationship between artificial neural networks and human cognition. Although both models captured meaningful aspects of human similarity judgments, their performance remained only moderately above chance, highlighting important gaps between current vision models and human perception. Future work could explore fine-tuning models on behavioral data to more closely reflect individual differences in human perception.