How far in advance do you ask? Many priests can’t do “hey, want to come to dinner tomorrow?”. Call and ask when he will be available for a dinner.
You say he is your godfather, so, he likely knows your family well. He may feel more comfortable with your family and be able to say now without worrying he will offend. Priests are “ON” so much, maybe invite him for something that is more of a leisure thing. “Hey, we have family game night on Thursday” or “We are going fishing on Friday next month” or just ask him to come with your family for pizza, beer and a movie. Priests are human beings with the same need for down time to relax as the rest of us.