One downside is that you will have to pay for private mortgage insurance (PMI) if your down payment is less than 20%. Additionally, FHA loans typically have higher interest rates than conventional mortgages.