As a quick guide, I advise self-publishing authors to budget $5,000 for editorial, cover design, interior design, and ebook coding. Even audiobook. And if they have any money left over, they should spend that on marketing and more. I explain the breakdown in my book Cover to Cover: What First-Time Authors Need to Know about Editing. Your experience with agents and traditional publishing is not unusual but certainly avoidable.