By attending a baptism, marriage or other religious service, you are participating in the action. There are ways of non-participation, especially if you are say a police officer and a fellow officer is killed and you attend his funeral along with the other officers. Then it's like a duty and it might be scandalous if you didn't attend, people might think you were being cold.
But you cannot attend a wedding you know to be invalid (such as a remarriage after divorce) or non-Catholic (certain passive attendance excepted, as above).
Nor can you attend a baptism of someone for the same reason. Remember, there is no Lutheran baptism or Presbyterian baptism or Calvinist baptism -- it's all just Catholic baptism, done outside of the Church. Thus there are two problems. The first is that any Sacrament which is celebrated outside of the Church is intrinsically a sacrilege. The second is that baptism creates an obligation to explicit Faith. We baptize adults after we catechize them, we baptize babies before we catechize them, but catechesis is important either way. By being baptized, that child now has a right to be educated in the Catholic Faith. This is why the priest will (or rather, should) not baptize a baby if the parents are not licitly married, practicing Catholics (or, in a mixed marriage, that they promise to raise the child Catholic). He doesn't just go through the neonatal unit at the hospital baptizing random children, it would seem that this is a good thing (since if they die before they commit their first mortal sin, they will go straight to Heaven rather than to Limbo), but it is actually wrong to do so because it is a violation of the rights of the parent about whether or not to baptize their child and it is also setting up the child to have his or or her rights violated if they are not raised in the Catholic Faith -- it creates a debt that may not be fulfilled.
[quote=Summa, IIIa, q.69, a.6]As Augustine says (Serm. clxxvi): "Mother Church lends other feet to the little children that they may come; another heart that they may believe; another tongue that they may confess." So that children believe, not by their own act, but by the faith of the Church, which is applied to them: by the power of which faith, grace and virtues are bestowed on them.
[quote=Summa, IIIa, q.70, a.1]Baptism is called the Sacrament of Faith; in so far, to wit, as in Baptism man makes a profession of faith, and by Baptism is aggregated to the congregation of the faithful.
[quote=Summa, IIIa, q.71, a.1]Just as Mother Church, as stated above (69, 6, ad 3), lends children another's feet that they may come, and another's heart that they may believe, so, too, she lends them another's ears, that they may hear, and another's mind, that through others they may be taught. And therefore, as they are to be baptized, on the same grounds they are to be instructed.
He who answers in the child's stead: "I do believe," does not foretell that the child will believe when it comes to the right age, else he would say: "He will believe"; but in the child's stead he professes the Church's faith which is communicated to that child, the sacrament of which faith is bestowed on it, and to which faith he is bound by another. For there is nothing unfitting in a person being bound by another in things necessary for salvation. In like manner the sponsor, in answering for the child, promises to use his endeavors that the child may believe. This, however, would not be sufficient in the case of adults having the use of reason.
When Baptism begins, the priest asks the godparents, "What do you ask of the Church of God?" and they reply "Faith." There is an obligation created for them to teach the child the true Faith, if they teach him false things (as in Lutheranism or other non-Catholic religions), they are not guilty of sinning against him, by violating his right of baptism to be educated in the true Faith.
Thus, by attending a non-Catholic baptism, you are condoning this sin of sacrilege and the sin which will come if they do not raise the child in the Catholic Faith. In short, you should never attend non-Catholic worship, especially actively participating (that is, apart from a merely material presence) at a Sacrament, such as marriage or baptism (or the mockery of the Holy Sacrifice of the Mass, which is Protestant communion). There are rare instances when a merely material presence at non-Catholic worship can be tolerated but this is probably not one of those instances (depending on what kind of message and chaos in your family not attending.
My sister is getting "married" again (she just got divorced and her fiancee and father to her most recent child is also divorced) and I have no intention of attending the wedding. I love my sister, I watch her children, I love the new baby for his own sake, but I know it would be offensive to God to attend her new "wedding" and so I cannot go (and yes, that applies to the reception, wedding presents, etc.). We shall see what happens when I do not go, our relationship with God is more important than our relationship with our family. Christ said, "He that loveth father or mother more than me, is not worthy of me; and he that loveth son or daughter more than me, is not worthy of me." (Mt 10:37).
But you can show your brother that you love him by praying and making sacrifices for him and his family, that they may come to/back to the Catholic Faith. This is a greater love than confirming them in their sin by attending the baptism. And you should explain to them why you are morally unable to attend, because they are not Catholic, rather than making some excuse why you can't go.