I think it depends on what KIND of fertility problems that your family has had.
For example, my brother, sisters and I all had trouble conceiving. My parents did not. My elder sister and I were eventually able to conceive, my brother was not, my younger sister ended up with a complete hysterectomy at the age of 27.
My father-in-law and his brother had the mumps when they were 16 and 13 years old, so they were never able to sire children. However, their other brother who had a very mild case (at age 14) had no trouble, and their younger sister (who was only 10) who never caught the disease was only able to have one child and was infertile afterwards.
Without knowing the causes of their infertility, it would be really hard to guess about your odds of having trouble.