sounds like the issue is not about your time at adoration, but about family use of time in general, including balancing work, baby, household, friendship, church and other commitments. Sounds like some good solid discussion, with professional guidance if necessary, is indicated.
when we retired and moved down here it was after long discussion, and for the purpose, or so DH said and I agreed, of spending more time growing spiritually, finding an apostolate etc. Things turned out quite differently and when I got tired of waiting for DH to golf, play softball, run social activities in our retirement part, watch TV, work, and other commitments, I found this job at the parish. he finally looked up, realized I was no longer hanging around, got a little whiny, and now is even more active in the parish than I am. who knew. I think he was just burned out on church stuff from his hard work at our old parish.
my point is that through all this time I tried mightily to preserve our Friday night date, and our Sundays (almost impossible with my job but I did it) but he is the one who started the Friday bridge game and the one who is glued to the TV watching sports all day and evening Sunday. I really think it was at least a year before he even missed me. we had to sit down and really thrash out the issue before we found workable solutions. yep at some point we really do need to talk.