Victor's wife was his childhood sweetheart. They were very happy together, especially when their daughter was born. When a vampire came to the village, it was this passion for his family that was so desirable to her so she set about making him undead. After this, the woman took Victor's wife and child away from him.
Victor always carried a locket with an image of his maker to remind him of his family. The place in the church graveyard where Olive Hawthorne discovered Victor feasting on a deer in the 1960s was the grave of his wife and child and his memories of her death still visibly shook him when he spoke of it. (HOMEVID: White Witch of Devil's End)