Not always, but it can also be caused by excessive gas intake into the stomach. You should be careful not to leave the bottle too flat. It is not advisable to continue breastfeeding when you have hiccups. You can pick your baby up and give him/her some warm water.