MediaCodec Buffer unavailable.

I am attempting to record an mp4 through the use of OpenGL SurfaceView and using a MediaCodec encoder to encode the frames of the video in one thread and in another thread I am using AudioRecord to record audio samples and using a seperate encoder to encode the audio data and finally a MediaMuxer to do the actualy muxing.

On some devices (mostly higher end devices) the method that I am using works completely fine and the output video is to be expected with no error or anything like that. However, on some other devices that isnt the case.

What happens is on some devices, (ie Moto Droid Turbo 2) the audio encoder (which is running in its own thread) will process a few audio samples then return MediaCodec.INFO_BUFFER_UNAVAILABLE when attempting to dequeue an input buffer for the encoder, this start happening after about 5 samples that are successfully encoded and the video encoder runs completely fine. But on another device (ie the Samsung Galaxy Alpha) just opposite happens where the video encoder begins to return the MediaCodec.INFO_BUFFER_UNAVAILABLE status when dequeue an input buffer for the video encoder and the audio encoder runs fine.

I guess my question is, and I have looked all over for a completely clear explanation of this, is what is the cause of a buffer being unavailable? Other then not releasing the buffer before next use, what can cause this? I am 98% certain that I am releasing the buffers when the need to be released because the exact same code will work on most devices that I have tested on. (Nexus 5x, Nexus 6P, Samsung S6, Samsung Edge/Edge+, Moto X (2014), Moto X (2015), HTC One M9, HTC One M8, etc).

Does anyone have any ideas? I didnt think posting code examples was necessary because I know it works on most devices I have tested but my question is what can cause MediaCodec.INFO_BUFFER_UNAVAILABLE to be returned aside from not release the buffer, if anything?