In this paper, an innovative agent-based security scheme is proposed for identifying potential impacts of cyber attacks in smart grid protection systems. Smart grids are critical cyber-physical energy systems (CPESs), where potential attacks can manipulate the normal operation of digital relays by injecting malicious control signals. This event could cause the relays to trip at undesired time which in turn results unintentional trippings, even if there are no faults. The proposed scheme identifies the malicious cyber attacks by utilizing cyber and physical properties of power systems. Moreover, the agents further distinguishes attacks from physical faults and their potential is validated through computer simulation using a benchmark power system.