Are Prayers in a State of Mortal Sin Effective?

The short answer is no. The reasoning that St. Thomas Aquinas gives is that just as an act cannot be meritorious for the doer of the act if he is in a state of mortal sin, neither can it be meritorious for others (Summa Theologiae, Suppl. 71:3).

However, someone who is in mortal sin can still benefit the souls in purgatory by offering a Mass for them. The good work of the Mass produces the effect ex opera operato (Latin, “by the work worked”). It is not your action that produces the effect but the Church’s action and the action of Christ.