How to handle voice messages in WhatsApp bots with AI
Learn how to handle voice messages in WhatsApp bots using AI transcription and image analysis — and how CodeWords supports audio and visual inputs natively.
Rebecca Pearson5 min read
If your WhatsApp bot only handles text, you're already missing a significant chunk of your customers' messages. On mobile — which is how most people use WhatsApp — voice notes are faster and more natural than typing. And photos are how customers document problems, show products, and share information.
A WhatsApp bot that handles voice and images is dramatically more useful than one that only processes text. This guide explains how the voice and image processing pipeline works, and how CodeWords supports it.
TL;DR
- Voice messages are common on WhatsApp — especially from mobile users who find typing slower than speaking.
- The pipeline is: download audio → transcribe → process as text → reply. AI transcription via OpenAI Whisper is fast and accurate.
- Image analysis lets customers send photos of problems, products, or documents — and get an intelligent AI response.
Why voice messages matter
WhatsApp voice notes aren't a niche feature. For many users — particularly those on mobile, in service industries, or in markets where voice is the natural communication mode — voice notes are the default.
A tradesperson describing a plumbing fault sends a voice note. A customer trying to explain a product issue sends a voice note. A patient describing a dental symptom sends a voice note. If your bot can't understand these messages, it will respond with something like "I'm sorry, I can only understand text messages" — which is a poor experience and means you've lost the lead.
The good news is that AI transcription has become fast and accurate enough to handle voice notes in real time, making it practical to include in a production WhatsApp bot.
The voice processing flow
Here's how voice message handling works step by step:
Step 1: the customer sends a voice note. WhatsApp compresses and uploads the audio to Meta's servers and sends a webhook event to your bot. The webhook payload contains a media ID — not the audio file itself.
Step 2: your bot downloads the audio. Using the media ID, your bot makes an authenticated request to the WhatsApp API to download the actual audio file. The format is typically OGG (Opus codec).
Step 3: the audio is transcribed. The audio file is passed to a speech-to-text model — OpenAI Whisper is the standard choice. Whisper is multilingual, fast, and handles background noise well. The model returns a text transcript.
Step 4: the transcript is processed as text. From this point, the pipeline is identical to a normal text message. The transcript is passed to the AI model with the conversation history, and the model generates a reply.
Step 5: the bot replies. The reply is sent back as a text message. Optionally, you can also send the reply as an audio message using text-to-speech.
Sending audio replies
If you want the bot to reply with a voice note rather than (or in addition to) text, you can convert the text reply to audio using text-to-speech. OpenAI's TTS API produces natural-sounding speech in multiple voices.
The flow is: generate text reply → convert to MP3 via TTS API → upload to WhatsApp → send as audio message.
Whether to send audio replies depends on your use case. For most business bots, text replies are clearer and easier for customers to refer back to. But for personal assistant bots, wellness applications, or markets where audio is strongly preferred, audio replies can feel more natural.
Image analysis
WhatsApp image handling follows a similar pattern to voice: the bot downloads the image file using the media ID, passes it to an AI vision model, and uses the analysis in its response.
The AI vision models in CodeWords — including GPT-5 and Claude Sonnet 4.6 — can analyse images and describe what they see in detail. This opens up several practical use cases:
Appliance repair: a customer sends a photo of their broken washing machine. The bot analyses the image, identifies the appliance model from visible branding, describes what looks wrong, and asks clarifying questions. It can then suggest whether the issue is likely a simple fix or needs an engineer, and offer to book a visit.
Auto repair: a customer sends a photo of warning lights on their dashboard, a dent, or a flat tyre. The bot identifies what it's seeing and provides relevant guidance or books an assessment.
Ecommerce support: a customer sends a photo of a damaged product they received. The bot confirms what it sees, logs the issue with the image reference, and initiates a returns or replacement flow.
Document processing: a customer sends a photo of an invoice, form, or document. The bot extracts key information (amounts, dates, reference numbers) and uses it in the conversation.
Use cases where voice and image handling makes the biggest difference
Appliance repair businesses: most customers send a photo of the broken appliance and a voice note describing the problem. Without voice and image support, the bot gets an audio file and a photo and can't respond meaningfully. With it, the bot can gather all the diagnostic information it needs in a single exchange and book the right engineer.
Auto repair and MOT: drivers often describe issues verbally while looking at the car, making voice notes the natural input. And sending a photo of the problem is faster than describing it in text.
Dental and aesthetics: patients describing symptoms or sending photos of areas of concern. The bot can acknowledge the input, provide relevant information, and collect enough context to prepare the practitioner before the appointment.
Construction and trades: site workers and clients documenting progress, issues, or specifications with photos and voice notes is standard practice. A bot that can receive and respond to both is a natural fit.
How CodeWords handles this
CodeWords supports voice and image inputs as part of the standard agent setup. When you describe your agent to Cody (the AI automation assistant), you can tell it to handle voice notes and images — and it will configure the transcription and vision processing pipeline automatically.
You don't write transcription code. You don't manage media downloads from the WhatsApp API. You don't set up TTS if you want audio replies. Cody handles the configuration based on your description of what you need.
To set up an agent that handles voice and images, your description to Cody might include: "When customers send voice notes, transcribe them and respond based on the content. When they send photos, analyse the image and use it to help diagnose their problem or answer their question."
For industry-specific setups, see our guides for appliance repair businesses and auto repair and MOT workshops, or explore the WhatsApp agents hub.
And if you're new to building WhatsApp agents generally, start with how to build WhatsApp AI agents with CodeWords for the full walkthrough.
Build a voice and image-capable WhatsApp agent on CodeWords — describe what you need to Cody and it'll handle the setup.