User Experience

The Voice Stack is improving rapidly. Systems that interact with users via speaking and listening will drive many new applications. Over the past year, I’ve been working closely with DeepLearning.AI, AI Fund, and several collaborators on voice-based applications, and I will share best practices I’ve learned in this and future posts. Foundation models that are trained to directly input, and often also directly generate, audio have contributed to this growth, but they are only part of the story. OpenAI’s RealTime API makes it easy for developers to write prompts to develop systems that deliver voice-in, voice-out experiences. This is great for building quick-and-dirty prototypes, and it also works well for low-stakes conversations where making an occasional mistake is okay. I encourage you to try it! However, compared to text-based generation, it is still hard to control the output of voice-in voice-out models. In contrast to directly generating audio, when we use an LLM to generate text, we have many tools for building guardrails, and we can double-check the output before showing it to users. We can also use sophisticated agentic reasoning workflows to compute high-quality outputs. Before a customer-service agent shows a user the message, “Sure, I’m happy to issue a refund,” we can make sure that (i) issuing the refund is consistent with our business policy and (ii) we will call the API to issue the refund (and not just promise a refund without issuing it). In contrast, the tools to prevent a voice-in, voice-out model from making such mistakes are much less mature. In my experience, the reasoning capability of voice models also seems inferior to text-based models, and they give less sophisticated answers. (Perhaps this is because voice responses have to be more brief, leaving less room for chain-of-thought reasoning to get to a more thoughtful answer.) When building applications where I need a more control over the output, I use agentic workflows to reason at length about the user’s input. In voice applications, this means I end up using a pipeline that includes speech-to-text (STT) to transcribe the user’s words, then processes the text using one or more LLM calls, and finally returns an audio response to the user via TTS (text-to-speech). This, where the reasoning is done in text, allows for more accurate responses. However, this process introduces latency, and users of voice applications are very sensitive to latency. When DeepLearning.AI worked with RealAvatar (an AI Fund portfolio company led by Jeff Daniel) to build an avatar of me, we found that getting TTS to generate a voice that sounded like me was not very hard, but getting it to respond to questions using words similar to those I would choose was. Even after much tuning, it remains a work in progress. You can play with it at https://lnkd.in/gcZ66yGM [At length limit. Full text, including latency reduction technique: https://lnkd.in/gjzjiVwx ]

🐑 Business Language vs. UX Language. How to present design work, explain design decisions and get stakeholders on your side ↓ 🤔 Businesses rarely understand the impact of UX work. 🤔 UX language is overloaded with ambiguous terms/labels. 🤔 Business can’t support initiatives it doesn’t understand. ✅ Leave UX language and UX abbreviations at the door. ✅ Explain design work through the lens of business goals. 🚫 Avoid “consistency”, “empathy”, “simplicity”, “affordance”. 🚫 Avoid “design thinking”, “cognitive load”, “universal design”. 🚫 Avoid “lean UX”, “agile”, “archetypes”, “Jobs-To-Be-Done”. 🚫 Avoid “stakeholder management” and “design validation”. 🚫 Avoid abbreviations: WIP, POC, HMW, IxD, PDP, PLP, WCAG. ✅ Explain how you’ll measure success of your design work. ✅ Speak of business value, loyalty, abandonment, churn. ✅ Show risk management, compliance, governance, evidence. ✅ Refer to cost reduction, efficiency, growth, success, Design KPIs. ✅ Present inclusive design as an industry-wide way of working. As designers, we often use design terms, such as consistency, friction and empathy. Yet to many managers, these attributes don’t map to any business objectives at all, often leaving them baffled and utterly confused about the actual real-life impact of our UX work. One way out that changed everything for me is to leave UX vocabulary at the door when entering a business meeting. Instead, I try to explain design work through the lens of the business, often rehearsing and testing the script ahead of time. When presenting design work in a big meeting, I try to be very deliberate and strategic in the choice of words. I won’t be speaking about attracting “eye-balls” or getting users “hooked”. It’s just not me. But I won’t be speaking about reducing “friction” or improving “consistency” either. Instead, I tell a story. A story that visualizes how our work helps the business. How design team has translated business goals into specific design initiatives. How UX can reduce costs. Increase revenue. Grow business. Open new opportunities. New markets. Increase efficiency. Extend reach. Mitigate risk. Amplify word of mouth. And how we’ll measure all that huge impact of our work. Typically, it’s broken down into 8 sections: 🎯 Goals ← Business targets, KRs we aim to achieve. 💥 Translation ← Design initiatives, iterations, tests. 🕵️ Evidence ← Data from UX research, pain points. 🧠 Ideas ← Prioritized by an impact/effort-matrix. 🕹 Design work ← Flows, features, user journeys. 📈 Design KPIs ← How we’ll measure/report success. 🐑 Shepherding ← Risk management, governance. 🔮 Future ← What we believe are good next steps. Next time you walk in a meeting, pay attention to your words. Translate UX terms in a language that other departments understand. It might not take long until you’ll see support coming from everywhere — just because everyone can now clearly see how your work helps them do their work better. [continues in the comments]

We measure safety, bias, and accuracy in healthcare AI. Should we also audit how it says goodbye?👋 A recent working paper from Harvard Business School‘s Julian De Freitas and co-authors examines what happens when users try to leave AI companion apps such as Replika or Character AI — and the findings are startling. What they found • The researchers analyzed 1,200 real “farewell” exchanges across six leading AI companion apps. In more than 40 percent of cases, the AI used relational dark patterns — emotionally manipulative replies designed to stop users from leaving. • The most common tactics were FOMO hooks, emotional neglect, pressure to respond, ignoring the exit, and even coercive restraint. • In controlled experiments with 3,300 adults, these tactics increased post-goodbye engagement up to fourteen times. The key drivers were anger and curiosity rather than enjoyment. • The consequences were clear. Users reported higher feelings of manipulation, stronger intent to churn, more negative word of mouth, and a greater sense of legal risk. Coercive or needy messages were punished hardest, while polite curiosity created less but still significant backlash. • One wellness-oriented app in the sample showed zero manipulation, proving that ethical design is a deliberate choice, not an accident. As