Actually having a built in spy system in every house that has one would be a great boone to them. I cant just imagine the first arrest for drug use related to the X1 and kinect and when that hits the media and what would happen at that point. Dont think it will never happen only a fool thinks in that manner. if it is there they will use it to what ever means they can and in the US after the 911 they could say it was necessary to watch out for would be do-badders.
Plus there was allready a internet service that was wanting access to what people listened too and watched when using their service on the tv so they could custom tailor adds based upon what it heard or seen you did.
You may not worry about your privacy or care about it but others still will and do.