Actually, technically John Bailey doesn't have any scenes with Tom Baker in The Horns of Nimon. (Similar situations with Walter Randall and Cyril Shaps among others).
John Levene's the obvious one, having played Benton alongside three Doctors (four if watching a playback of William Hartnell counts). A couple of other Dalek operators managed it: Cy Town appeared with five Doctors, Murphy Grumbar with three and there's a couple who did all the 80s stories. Terry Molloy played Davros opposite three Doctors on television and one in the audios.
Dallas Cavell appeared with four Doctors. Others who've managed three include Donald Pickering, Wanda Ventham, Christopher Benjamin (Pertwee, Tom Baker and Tennant), Bernard Horsfall, Martin Jarvis, Ric Felgate, Michael Kilgarriff, John Abineri, Kevin Stoney, Philip Madoc counting Big Finish, Tony Caunter, Peter Miles if you count Big Finish, Norman Jones, Alan Rowe, James Bree...
Actors with a large amount of uncredited appearances, often in monster costume and/or missing episodes, are hard to be sure of: Pat Gorman appeared with up to six Doctors, Terry Walsh four, Stuart Fell three.