"Baby" was the name the Eleventh Doctor gave to the language supposedly spoken by human infants. It was spoken by the flesh duplicate of Melody Pond and Alfie Owens. It appeared to be a one-way language, spoken by babies and heard by the auditors. The listener — in these cases, the Doctor — responded in standard English, which the babies seemed to understand.
The TARDIS translation matrix, however, did not translate Baby for other listeners. Only the Doctor appeared able to communicate with them in this way. Judging solely by the Doctor's responses, "Baby" was a complex language with a complete grammar, a large vocabulary and many rich colloquialisms. Given the eccentricities of the Eleventh Doctor, of course, it is entirely possible that he was just joking, and that no such language exists. (DW: A Good Man Goes to War, Closing Time)
The Fourth Doctor also indicated an ability to speak Baby. When he and Sarah were in Pescaton-terrorised London, they came across an apparently abandoned baby. Unable to get any information from the child, the Doctor said, "He won't talk." Unaware that the Doctor may have have spoken literally, Sarah responded, "He can't talk." (AG: Doctor Who and the Pescatons)