Ventriloquism works when people trust their eyes rather than their ears (see Ventriloquism entry). But this isn’t true for humans of all ages.
In the womb, babies can hear but not see. Thus when born, infants show a marked preference for sounds rather than images. This appears to extend at least until a child is 4 or 5 years old. It has been suggested that it may be this preference which could drive the child to try to understand the language spoken around it – and thus learn to speak.
To my knowledge, no-one has tested for ventriloquism effects in infants – but my guess is that they would show a reverse ventriloquist effect.
Robinson, C.W. and Sloutsky, V.M. (2004) Auditory Dominance and Its Change in the Course of Development. Child Development 75 (5), 1387-1401.
Sloutsky, V. M, & Napolitano, A.C. (2003). Is a picture worth a thousand words? Preference for auditory modality in young children. Child Development, 74 (3), 822-833.