Howling:Rewinding time stream...

The Howling → Rewinding time stream...

There be spoilers about un-released stories here.
Run back to the forums if you're scared.

Sorry if this has been brought up elsewhere. In the finale the impression seems to be that The Doctor is rewinding along his own time stream. However, visiting a young Amy Pond sees him in a time and place that wasn't part of his time stream. He didn't return that night; he showed up years later by mistake. Taking that into consideration, it seems The Doctor is rewinding along Amy's timestream (she's the common denominator), not his own.

Is this just an oversight by the writers or is there an explanation I just didn't catch?