Olympic-sized pools were venues used in the Olympic Games, and thus were places where one could attempt to break Olympic swimming records. Miranda, for instance, was a swimmer who tried to break an Olympic record, even though her friends, like Dinah, were merely attempting to be competitive at the county level. (PROSE: Father Time)
Of course, not everyone who used an Olympic-sized pool was trying to break any records whatsoever. On at least one occasion, Ace swam in the TARDIS' Olympic pool, noting to herself that it had "all the trappings of a council house bathroom", and that the Seventh Doctor called it a "bathroom". (PROSE: Loving the Alien)