This effect can be startling if done correctly. One problem that you face is giving the wall a sense of "weight". People will not be scared by a moving piece of plywood, unfortunately.
Reinforce the idea of a heavy, dangerous wall with pictures, lamps, wall hangings... anything that hangs, this gives the illusion of it bearing a load. Also, sound effects can be key! Scraping, sliding sounds can really sell the effect. Finally, don't allow any light to escape from around the edges! Rshorts was spot on with the weather stripping. Good luck!
I became insane, with long intervals of horrible sanity.