what you are seeing does not make sense
the drive should not care what file system you are putting on it,it should work with all or with none.
It could be that the device is going bad and you are seeing old fragments of vfat when you reput vfat.
try to first wipe it out and then try to add ext4,when it fail,add vfat again and check
wipe the device with:
dd if=/dev/zero of=/dev/sdb1
then add ext4 fs
them check with blkid to see if the fs sticked.If it still does not show then add vfat and then run blkid to see if it now show