Try starting with flavours you or your family likes. Taking recipes off the internet is really kind of lame because it has none of your personality in it.
Like peaches? Add peaches. Like mangos? Add mangos. Start with a base of flavour (like any kind of cooking), make sure that is correct...
I have never heard of seeds adding bitterness unless it makes up some crazy percentage of the bottle like anything over 1% overall volume. But at that point its more of a seed sauce than a hot sauce HAHA