I am trying to transfer my itunes videos to videos I can use on my Nook HD. But it seems impossible because every time I tried, it failed. What am I doing wrong?

You purchased these videos from iTunes, right? They are DRM protected, in other words, you can only watch them on local computer. If you wanna watch on portable devices like Nook HD, stripping of this protection is needed. Use Pavtube ChewTune, I successfully used it moved couple iTunes movies to my Kindle Fire.

blu-ray-dvd-converter.com/chewtune