Mainstream, popular theatre in our culture today mainly perscribes to a naturalistic style - aiming to refelct real life back to the audience. An audience may expect to enter a theatre space which is very clearly defined, the audience sit and face the proscenium arch stage and view a play, a story presented in such a way that they are transported to the world of the play, and engrossed by the actors' performance, empathising with the characters. In such a naturalistic performance, the job of the set is to aid this transportation - to recreate real life on the stage. Say the scene is set in a kitchen - the stage lights are set to recreate the natural light of the day time, the back ground has a realistic image of a kitchen, the set may be made up of real kitchen worktops and apliances and the audience watch as if they are looking through the "fourth wall".
In non-naturalistic set design, the aim is just about the complete opposite. The theatre practitioner Bertolt Brecht, for example, used non-naturalistic set design in his Epic Theatre productions to wake up the audience, to never let them forget that they were watching a theatrical production, and to throw a literal spotlight on the issues of the day, so that the audience would be insipred to discuss and develop their own oppinions. Non-naturalistic techniques may also be used to compliment (or contrast!) with the action in the scene - for example during a one of Hamlet's monolouges, using blue coloured gels on lights that steadily increase in the depth of colour may be seen to create a symbol for water, and how Hamlet is drowning in his thoughts and sinking deeper into his depression. Non-naturalistic set is important in creating symbol and metaphor that may be then interpreted by the audience - though it is important to note that what the audience intends, and the audience interpretation, hardly coincides. But in this lies an intrige of its own!
When you think of non-naturalistic set design you may also think about the placement of the audience. Working outwith the boundaries of naturalism allows a directer to place the audience where they like - or even leave it up to the audience to decide. The audience/actor relationship is thrown into turmoil and this can be incredibly valuble and influential to a performance. The audience are no longer just passive viewers within a non-naturalistic set design - they could be a pool of possible performers, active commentators, or even a part of the set its self!