What would you do stay away from him or go for it?

I met a man who treated me wonderful and i was really starting to have feelings for him but , he has a common law wife, who he was fixing to leave because of me , which I was single and have two children he also had children a little older than mine and his daughter came to me and told me they were having problems and she did not want to lose her dad , so I broke it off , now the kids are gone, he is still with her but they have not been intimate in a while ,lately i have started to see him again and I am feeling those feelings again no man has ever treated me or respected me the way he has, but I feel he is more weary of me now because i think i hurt him . my heart is tearing and I don't know if I can take this again? will there always be this ?I told him I was infatuated with him and he told me go slow is this a brush off or trying to keep me on the sly?? what do you think ? be honest I have been through some very rough times and I can take it