Bhagavad-gita says that a devotee doesn't differentiate between good and bad—both are equal to him. But if a person is harming you, how can one stay calm thinking that there shouldn't be dualities?
Perhaps you're speaking about Bhagavad Gita 6:7 "For one who has conquered the mind, the Supersoul is already reached, for he has attained tranquility. To such a man happiness and distress, heat and cold, honor and dishonor are all the same."
In this verse, as well as others like it, what is meant is not that these differences aren't perceived, but that they don't affect a person's actions.
In the Mahabharata, Arjuna was prepared to fight those who had wronged his wife, his family, his kingdom, and even exiled himself and his family to the forest. He saw them as enemies.
Then, when he got on the battlefield, he didn't want to fight. He didn't want to kill his grandfather, his guru, and others on the Kuru side because he saw them as friends. But once he heard the Bhagavad-gita from Krishna, Arjuna agreed to fight. But now his agreement to fight was not because he saw his teachers and relatives as enemies or even as friends, but rather because he realized that the fight was Krishna's will. He realized that by doing what Krishna wanted all requirements would be met. This is the qualification of one fixed in transcendence—to be fixed in doing what Lord Krishna wants and not anything else for any other reason.
He may feel happiness or distress, he may understand good or bad, but that judgment is made differently; things that make Krishna happy cause "happiness," and not being able to serve Him causes "distress." "Good" is what brings one closer to self-realization, and "bad" is what takes one away from that.
Of course if it's one's job to protect people—as a policeman, for example—then he must see "bad people" as those who infringe on others' property or rights, and protect others from them. That's his responsibility, and he does it out of duty. He doesn't neglect to distinguish between good and bad. However, that vision is useful and reasonable because an administrator's duty is to see those differences.
As a devotee, he may see everyone as soul. He may have no personal enemies; but in terms of karma and duty he understands that "this person must be punished" if he's breaking the laws of God and harming others.
I hope this makes things more clear.