I dedicated last 10 years of my life to a single product: Targetprocess. It's an agile project management software. Company had its highs and lows, product evolved, goals changed. The hardest problem to me as a Product Owner was features prioritization.
We have thousands of ideas: small and large, good and great. Many of them came from customers, many of them were invented inside a company. I always struggled and rarely felt a deep satisfaction about features order in a backlog: "This feature has so many votes, maybe we should do it first?" and then "But we have a cool vision and people don't see it so far, we should focus and deliver a set of features that support our vision". These voices took endless battles inside my head and I was quite unhappy. I just couldn't decide what argument is the best. Now I know the answer — both.
I think there are two major phases in product development: Innovate and Refine. And you, as a Product Owner, should use different mechanisms to prioritize features in these phases.
Innovate phase is fully dedicated to your vision. You do listen to customers and pay attention to their feedback, but in the end you invent a solution that supports your vision and solves many problems in a new, unexpected way that almost nobody can see so far. Let's say, you want to mix two domains like Project Management and Information Visualization. Nobody asks for that. When you share your ideas you get positive feedback, but many people still ask about a faster horse. You ignore these requests and move forward with your vision.
In this phase you rely on your intuition more.
Refine phase is completely different. Now you silence your ideas and focus on customers requests. Customers are extremely good at small and moderate improvements. They spot all major problems and give your an honest feedback. Definitely, there is no need to implement every single idea — that is a direct road to hell. If you have a service like UserVoice where customers can vote for ideas, most likely you can safely implement top 5% of requests.
In this case you refine the product and polish it based on customers feedback.
It is clear that these phases demand different prioritization schemes. What work for Innovate phase will not work for Refine phase. That is why I had cognitive dissonance, and it took me 10 years to nail the problem and find a solution.
The solution is to mix these phases, like that
You switch focus every 3 months and it makes prioritization a breeze. Next 3 months we are going to innovate? OK, let's take these two features that will deepen our vision and differentiate us from competitors even more. Refine phase? OK, let's take these top ten requests from customers, find solutions, get feedback and implement these solutions.
There is no constant battle between innovation and refinement anymore. You cease to compare oranges with apples. You finally feels confident and calm. And everybody wins.