"Baby" was the name the Eleventh Doctor gave to the supposed language spoken by human infants. It was spoken by Melody Pond and Alfie Owens. It appeared to be a one-way language, spoken by babies and heard by the auditors. The listener — in these cases, the Doctor — responded in standard English, which the babies seemed to understand.
The TARDIS translation matrix, however, appeared unable to decipher Baby for other listeners. Only the Doctor appeared able to communicate with them in this way. Judging solely by the Doctor's responses, "Baby" was a complex language with a complete grammar, a large vocabulary, and many rich colloquialisms. Given the eccentricities of the Eleventh Doctor, of course, it is entirely possible that he was just joking, and that no such language exists. (DW: A Good Man Goes to War, Closing Time)
The Fourth Doctor also exhibited an apparent ability to speak Baby. When he and Sarah were in Pescaton-terrorised London, they came across a seemingly abandoned baby. Unable to get any information from the child, the Doctor said, "He won't talk." Unaware that the Doctor may have been speaking literally, Sarah then responded, "He can't talk." (AG: Doctor Who and the Pescatons)