Why Products Leaders Should Invest in Hard AI Skills
The AI industry has been shipping at breakneck speed. Let's discuss how product managers can adapt to a new reality.
Product Managers (PMs) and product leaders are being challenged by the high pace of AI development. In this post, I'll dissect why I think they struggle, shed light on the critical skills that are overlooked, and make the case for early investment in what I call "hard AI product skills."
The Surface Talk
I’ve noticed that discussions often center around buzzwords like semantic search, "agents," or the integration of chat interfaces. But do these conversations truly translate into a profound understanding within product teams?
While teams can identify customer problems, propose solutions, and collaborate seamlessly with design and development, I think the current challenge lies in their ability to draft a comprehensive and future-proof Product Requirements Document (PRD) for an "AI-powered" feature.
Non-deterministic PRDs
From what I’ve seen, it’s not unusual for 9 out of 10 PMs to find themselves floundering when tasked with the challenge of articulating a PRD for an "AI-powered" feature.
They struggle because generative AI systems introduce a layer of non-determinism, demanding a consideration of additional requirements beyond the surface-level demo applications.
Beyond Surface-Level Understanding
When confronted with generative AI concepts, there is one overarching question that has to be solved: "How can we ensure a consistently high-quality user experience across a spectrum of use cases?"
When talking to PM’s, their typical response, deflecting responsibility to engineers and trusting them to navigate the intricacies, falls short. This approach disregards the fundamental truth that, similar to any other product feature, PMs must articulate details encompassing goals, outcomes, and user experience.
Develop an understanding of ‘hard skills’
So how do product managers proceed? Well, I think that they should develop a very good understanding of what it takes to build an AI product or feature. Let’s dissect the essential components that are the foundation for AI applications, extending beyond the constraints of conventional PRD templates:
Prompt Strategy, Writing, and Testing:
This is about figuring out how to talk to the AI. What words or questions will make it understand and respond the way we want? It's like teaching the AI the right language to get the best results. Writing good prompts is crucial, and testing them ensures the AI understands and gives the desired output.
Scoping Agent Tasks:
Here, we decide what jobs the AI should do. It's like telling the AI what's on its to-do list. We need to be clear about the tasks we want it to handle, setting the boundaries so it doesn't get confused or go off track. This ensures the AI focuses on what it's meant to do.
Retrieval Strategies:
The ability to efficiently fetch pertinent information distinguishes a proficient AI system. Understanding and implementing effective retrieval strategies ensure that the AI doesn't merely respond but does so with a depth of knowledge, enhancing the overall user experience.
Context Management:
This is like helping the AI remember what it talked about earlier. It's about keeping track of the conversation so that the AI doesn't forget important details. Managing context ensures a smoother and more coherent interaction with the AI over time.
Feedback Strategies:
Feedback serves as the compass for AI improvement. Establishing effective feedback loops is not just about receiving input; it's about structuring mechanisms that enable continuous learning. Whether from user interactions or system evaluations, strategic feedback strategies are the conduits for refining AI performance, fostering a cycle of iterative enhancement.
Unit Economics:
In the domain of generative AI, unit economics extend beyond traditional business metrics. It involves evaluating the cost and value associated with each AI interaction. Understanding the resource allocation, computational expenses, and the value derived from AI outputs is essential. Unit economics in AI is a perspective that product leaders must adopt to ensure efficiency and sustainability.
Security & Prompt Injection Hardening:
This is about making sure the AI is safe and protected. It's like putting a strong lock on the door to prevent any unwanted access. Ensuring security and prompt injection hardening safeguards the AI from potential risks, making sure it operates in a secure environment.
Model Selection:
Choosing the right AI model is like picking the best tool for a job. Different models have different strengths. Model selection is about understanding which one suits our needs the most, ensuring we get the best performance from the AI.
Identifying and Structuring Sources for Fine Tuning:
Fine-tuning is the sculpting of AI proficiency. It involves identifying and structuring diverse data sources to enhance the model's capabilities. Product leaders must develop an eye for relevant data, understanding how to structure it to refine AI behaviour. Fine-tuning is an ongoing process, a continuous refinement that separates a proficient AI system from the ordinary.
Chains & Agent Orchestration:
In the symphony of AI interactions, chains and agent orchestration play a pivotal role. It's the art of choreographing the sequence of actions, ensuring a harmonious flow of responses. Product leaders must master the orchestration of AI agents, structuring their interactions in a seamless manner. Chains and orchestration are the keys to orchestrating a symphony of AI responses that resonate with user expectations.
Analytics & Quality Management:
Analytics is like keeping track of how well the AI is doing its job. It's about measuring its performance and identifying areas for improvement. Quality management ensures that the AI consistently delivers high-quality results, maintaining a standard that meets our expectations.
Hands-On Learning
To rapidly acquire these essential skills, a hands-on approach is essential. It's not just about building things; it's about thinking how to guide your team through the development process. What are the core requirements? How do you quantify success for your team? How do you communicate specifications for an experience that is inherently non-deterministic? How do you design for trust?
Looking Ahead
As I wrap up, I find myself leaning towards a personalised perspective. What we should do as product leaders becomes clear-embrace the challenge, develop a deep understanding of generative AI, and personally equip yourself with the hard skills that I think will shape the future of product development for years to come.