So much depends on your town and your neighbors. Every single situation is different. If you will do a search on here, you will find some of the problems people have had over the years.
Here is an extreme example: http://www.cnn.com/video/#/video/liv...ted.house.knxv
I say extreme because these guys wanted to charge admission and hold it in in a gated community. That is about as big a no-no as you can get.