Xbox 360 problems on netflix?

I can start watching a show but I'll get 5 minutes in and it will say my connection is slow and is adjusting then it won't adjust and say can't start playback. But then I can play the episode from where it left off and it e fine and do the same thing 2-3 times an episode. My xbox is connected to the motem dirrectly. Its not wireless it's connected. Any help?