This might be terrible of me, and I’m not sure if I can even express it clearly, but here goes.
I think there is a huge difference in all of this, depending on the situation of the person. I don’t have a lot of problem with, say, the admiring looks of an unmarried man, particularly if he’s young, “checking out” a young woman who might, ultimately, be a marriage prospect. Physical attraction is part of that whole complex whereby people choose mates. It’s not the only thing, for sure, but it’s part of it, and I suspect God’s okay with it, even realizing sometimes the admiration might go a touch “over the line”. Let’s face it (and without going into excessive detail), to a significant degree, physical attractiveness is related to health. Health is related to fecundity and the strength that one might need in raising a family. I think the young man or woman in question is doing what he/she ought to be doing at that point in life, and much can be forgiven in the process.
Now, for a married guy, or somebody whose whole purpose is just plain old lust, I would say it wouldn’t take much ogling to be sinful.