It's called a "priest-confidante". And yes it happens. When i was in college, i had heard of research saying married women with a priest-confidant usually have happier marriages.
Priests and nuns who left their vows to get married, however, had an incredibly high divorce rate.
Since I haven't seen a pic of the luncheon, I have to go along with the OP's description of the situation. What size town do you live in? If NYC, there won't be quite as much danger of scandal as, say, a town of 5,000 (like where I grew up) .
Is it possible to take something to him at the lunches where hubby can't attend, since the priest is from another country? Would make it look like it was a somewhat casual encounter.
As long as hubby knows about it, and can answer the questions from the curious, there shouldn't be a problem.
If two ppl are comfortable in their vocation, there is such a thing as a 'spiritual friendship'. Look at St Francis and "Brother Jacoba". Or Boniface and Lioba. In that last situation, St Boniface wanted Lioba to be buried in their cemetary, but the brothers said no way.
St Aelred of Riveleux (sp?) Wrote about spiritual friendships. There has been a book in publication in the last 20 years about the subject.
The priest does need to find male friends, though. Your family may just be a springboard for him in the community, and this wont be an issue this time next year.
A bit of humor. Every man a particular woman invited over for supper became a priest. True story.