We cannot avoid reading secular literature (in the broadest sense). It’s always difficult to judge a book by its cover. We need to preserve our culture in which religious freedom is enshrined, for example. Our evangelization efforts need to start often times with where people are right now.
I’m reading GK Chesterton’s “orthodoxy” with some difficulty due to the mismatch of his talent and my inability to absorb it. But, the point is, he is familiar with this secular literature that perhaps you are referring to, and it is his expressed goal to unpack the errors in it.
In modern times, it’s like we have to read the DaVinci Code to carefully unfold its misrepresentation of Catholic history, for example. We don’t all have to read it, but some will and should do so to review it for the rest of us.
At a higher level of confrontation are the works of malignant atheists which require a lot of sophistication to refute.
Another genre of secular literature are detective and romance novels, where there’s usually some immoral activity in large doses.
So, yes, avoidance of all kinds of secular literature is warranted, and it a higher level of maturity to develop sound judgment about what secular literature to read and which to avoid.
This also depends on where you draw the line between secular and non-secular literature. You may consider a Jewish commentary on scripture as secular literature, that is, non-Catholic. But, the Church explicitly approves of studying such, under caution for the different points of view contained therein.