Pilgrim's Progress, by John Bunyan. Yes, it's anti-Catholic, but only a few sentences. The rest of the novel is quite Catholic.
Dracula by Bram Stoker
I've read it dozens of times. Never cease to find something new.
The Adventures of Huckleberry Finn, by Mark Twain. If you haven't read this and the prequel, The Adventures of Tom Sawyer, these books should be at the top of your bucket list.
The March family trilogy by Louisa May Alcott. * Little Women, Little Men, Jo's Boys.* If you are a man, you will probably have a hard time getting excited about these novels. Well, read them anyway. Plenty of us women had to suffer through The Red Badge of Courage or *Call of the Wild * in school. I hated both of those novels, but I appreciate their influence on American culture.
Uncle Tom's Cabin by Harriett Beecher Stowe. This novel has immense historical significance. It shows us how powerful the written word can be--Ms. Stowe essentially roused the abolitionists and got things stirred up in the U.S. with her novel.
I think you need to be careful about condemning novels as time-wasters. There are many novels that fit your definition of "having a lasting impact," and these novels are not necessarily famous or have literary status.
Also, the "lasting impact" will vary depending on the person. E.g., I have a children's novel called Black and Blue Magic, which I've read and re-read hundreds of times. It's just a simple story that has never been promoted or famous or made into a movie, but it has impacted much of my life because of the plotline about a father who isn't there.
Another example of a novel with a huge impact (too huge--at one point in history, it was banned) is Bridge to Terabithia, a children's novel by Katherine Patterson. The ending is a shocker.
I have read all of the women's novels written by Victoria Holt (a pen name). Time wasters? I don't think so. These novels are about strong women who face and overcome obstacles. There is no sex in the novels, but lots of romance. I feel that these novels have impacted me to look for adventure and to live every day with grace. I also think that these novels have helped me to be more appreciative of my husband.
Finally, any novel that helps us to relax and gain a new perspective has merit and is worthwhile. A lot of people enjoy reading Grisham's novels (I don't) or Siddon's novels (I do), not because they are high literature, but because they are R and R, something that is hard to find cheap in this technological and fast-moving culture.