Sorry if this goes a bit long. The topic is not a simple one.
Just a couple of questions:
How are you ripping from VHS? Are you using a transcription deck (that records directly from VHS to DVD)? I am using an older Panasonic DVD Recorder
You're also using raw .mov footage from a camera. Is this what you are converting with Handbrake? (Doesn't seem like it.) No. I dub with the DVD Recorder from VHS to DVD, rip with Handbrake, and edit with avidemux. The movies from the camera I download onto computer and edit with avidemux.
When I look at the video options in handbrake, under video encoder, the default is the H.264, but I can also choose mpeg-4 (ffmpeg) or mpeg-2 (ffmpeg). Are those better alternatives?
That "Panasonic DVD Recorder" is a transcription deck. Tape goes in one side, blank DVD-R goes in the other, press a few keys, and a player-ready DVD-R is the final result. Not fancy, (e.g., no menus or control over chapter divisions) but it gets the job done.
As to the "better alternatives", that can turn into a religious war in some circles. Me, I tend to be more pragmatic: if your conversions are playing back correctly in the desired environment, you are being successful to that point.
Where it seems you are having trouble is during playback after editing with avidemux. I don't really use avidemux that much myself, so I would hesitate to give app-specific advise there. You might consider consulting the avidemux forums
for something more authoritative.
Getting back to the compression format: this quote from Wikipedia may help clarify for you why h.264 is not necessarily a bad choice:
H.264 is perhaps best known as being one of the codec standards for Blu-ray Discs; all Blu-ray Disc players must be able to decode H.264. It is also widely used by streaming internet sources, such as videos from Vimeo, YouTube, and the iTunes Store, web software such as the Adobe Flash Player and Microsoft Silverlight, and also various HDTV broadcasts over terrestrial (ATSC, ISDB-T, DVB-T or DVB-T2), cable (DVB-C) and satellite (DVB-S and DVB-S2).
So, you can see h.264 is quite versatile and popular. If you can make it work for you, your finished product will be capable of reaching a wide audience on a variety of media outlets. If that's a consideration, you may want to stick with it. If you're not so concerned with this aspect, other formats may serve your needs as well as or better than h.264. I would hazard to guess that MP4 (ffmpeg) would be worth a try on a small sample, just to see.
Another item of interest is whether you're using MP4 or MKV output from Handbrake. These two container formats
have different strengths and weaknesses. Here's a comparison from the Handbrake Wiki
I favor MKV outputs for a lot of what I do, but I have different requirements than you may have (ability to add foreign language subtitles, alternate audio tracks, etc.), and for some things I find MP4 output is simpler (transcoding for tablets like the Nook Tablet).
What would tempt you to consider another option? I'd probably consider finding another way to edit the h.264 stuff you have now, but that's me.
Hope some of this helps you sort it out.