Psychological novels deal with some disturbed aspect of the human mind, whether insanity or an altered perception of reality or simply some inner struggle with an element of control over the human mind. Writing a good psychological novel takes skill and an understanding of the effect you want to achieve. Here's how to write a psychological novel that will take readers inside the minds of your protagonist and antagonist and have them turning pages to see what happens next.
Create strong characters. Both the protagonist and antagonist should have strong emotional issues that stand in the way of accomplishing their goals. The antagonist in a psychological novel may not even be a person but could very well be the inner demon of your protagonist, working just as hard as any living antagonist to prevent your hero from achieving his goals.
Write a strong backstory. If there is some emotional torment keeping your protagonist from accomplishing what he needs to, there should be a solid history to back it up. Write a complete history for all your characters before you begin, even if you use only enough to give the audience insight into your hero's psyche.
Plot your story around the emotional aspect of the story. A psychological story is internal, which means a good deal of it revolves around the turmoil in your lead character's head. The emphasis is not on physical action or external elements but on the struggle within your character.
Set up the internal struggle early in your story. Maybe your character has a fear he will need to overcome later in the story to reach his goal. By setting up his fear early, your readers wait for a payoff you finally deliver when your character overcomes his fear. Never cheat your readers out of the payoff, though.
Following these basic rules and applying them will allow you to write a psychological story in any subgenre, including horror, thriller, mystery or suspense.