The exact layout of the ByteBuffer is determined by the codec for the input format you've chosen. Not all devices support all possible input formats (e.g. some AVC encoders require planar 420 YUV, others require semi-planar). Older versions of Android (<= API 17) didn't really provide a portable way to software-generate video frames for MediaCodec.
In Android 4.3 (API 18), you have two options. First, MediaCodec now accepts input from a Surface, which means anything you can draw with OpenGL ES can be recorded as a movie. See, for example, the EncodeAndMuxTest sampleEncodeAndMuxTest sample.
Second, you still have the option of using software-generated YUV 420 buffers, but now they're more likely to work because there are CTS tests that exercise them. You still have to do runtime detection of planar or semi-planar, but there's really only two layouts. See the buffer-to-buffer variantvariants of the EncodeDecodeTest (linked from same page)EncodeDecodeTest for an example.