Does he have a right to see this child although he doesnt support it?

i am 7 months pregnant and me and the baby's dad are seperated. he walked out of his job while i was 2 months gone and is still unemployed. he does not drive, or have use of a vehicle, stays at his parents house out in the countryside (which is filthy and has no central heating), he jas no transport and is 4 miles from the nearest town. one minute he is begging for me back but when i say no, he turns very very nasty. he has not helped out with getting things bought for the baby arriving. i have spent alot of money (with help from my family) but he and his family have done nothing but give me hle. but he is now saying he is going to take me to court because he has rights as he is the father. is this right? how can someone who has done nothing to support his child financially have rights to see this baby when ever he wants? if i have to let him see this baby, does he have to take the baby to his partents house? it really is the filthiest place i have ever seen and with no heating, my baby will freeze. what should i do?