The way I've found restoring chat history is to add the backup data into the new phone before you start using WhatsApp. Once you start using WhatsApp and then try adding the previous data, it won't recognise it as the backup data will conflictv with what's already on the phone. I have a note 4 and use WhatsApp but the app wasn't pre installed on the phone so bit confused as to why it was on yours.
The way around it is to make WhatsApp think you're starting from scratch by deleting the app and reinstalling. In your case, the only thing I can think of is to reset to factory settings and then make sure the sub-folders from your old WhatsApp folder (database, media and profile picture folders) have been moved over to the note 4 main WhatsApp folder before you start using the app I.e before going through the setup process.
You're phone settings and apps should have been backed up by Google and Samsung already so resetting the phone shouldn't be too much. Just to make sure, go into accounts section in the main phone settings menu and check that the Samsung account has been synched and also the Google accoubt you use for the play store has synched the play store. This is done from the accounts section as well
Hope this helps. I do this all the time when I move between old and new phone or where I have to reset my phone and works all the time. Just a note, if you ever have to reset your note 4 for any reason after the above, say in a couple of months or whatever, just make sure you go into WhatsApp and do an up to date backup of the chat history (in settings and then chat history) as backups are usually done in the middle of the nigh. One you've done this, copy the entire WhatsApp folder to your laptop or PC before resetting your phone. That way you don't lose any chat history and photos etc!