A relative indifference to the human voice is a characteristic of Autism Spectrum Disorder (ASD). Yet, studies of voice perception in ASD provided contradictory results: one study described an absence of preferential response to voices in ASD while another reported a larger activation to vocal sounds than environmental sounds, as seen in typically developed (TD) adults. In children with ASD, an absence of preferential response to vocal sounds was attributed to an atypical response to environmental sounds. To have a better understanding of these contradictions, we re-analyzed the data from sixteen children with ASD and sixteen age-matched TD children to evaluate both inter- and intra-subject variability. Intra-subject variability was estimated with a single-trial analysis of electroencephalographic data, through a measure of inter-trial consistency, which is the proportion of trials showing a positive activity in response to vocal and non-vocal sounds. Results demonstrate a larger inter-subject variability in response to non-vocal sounds, driven by a subset of children with ASD (7/16) who do not show the expected negative Tb peak in response to non-vocal sounds around 200 ms after the start of the stimulation due to a reduced inter-trial consistency. A logistic regression model with age and clinical parameters allowed demonstrating that not a single parameter discriminated the subgroups of ASD participants. Yet, the electrophysiologically-based groups differed on a linear combination of parameters. Children with ASD showing a reduced inter-trial consistency were younger and characterized by lower verbal developmental quotient and less attempt to communicate by voice. This data suggests that a lack of specialization for processing social signal may stem from an atypical processing of environmental sounds, linked to the development of general communication abilities. Discrepancy reported in the literature may arise from that heterogeneity and it may be inadequate to divide children with ASD based only on intellectual quotient or language abilities. This analysis could be a useful tool in providing complementary information for the functional diagnostic of ASD and evaluating verbal communication impairment.