How to monitor and debug your WhatsApp AI agent in production
Learn how to monitor and debug your WhatsApp AI agent — key metrics, common failure modes, logging best practices, and how to test safely in production.
Rebecca Pearson7 min read
Getting a WhatsApp AI agent live is one challenge. Keeping it reliable over time is another. Once your bot is handling real conversations, you need visibility into what it's doing, what's going wrong, and why. This guide covers the key metrics to track, the most common failure modes, and how to debug your WhatsApp AI agent in production without breaking anything.
TL;DR
- Six metrics matter most — response time, completion rate, handover rate, message volume, error rate, and cost per conversation.
- Most bugs are structural, not AI-related — the bot never replies? Check the webhook. Replies twice? You're missing deduplication.
- Log enough to debug, not enough to cause problems — avoid storing personally identifiable information; log message IDs and timestamps instead.
Key metrics to track
Response time
How long does it take for your bot to reply after a message arrives? A response time under three seconds feels instant. Over ten seconds starts to feel broken. Over 30 seconds and customers will send a follow-up message — which can cause duplicate processing bugs.
Track response time as a histogram, not just an average. A median of two seconds with a 95th percentile of 25 seconds tells you something very different from a flat two-second response across the board.
Completion rate
What percentage of conversations reach a successful outcome — an answered question, a booked appointment, a captured lead? If your completion rate is dropping, your bot is either failing to understand messages or hitting a dead end in the flow.
Compare completion rate by message type if you can. FAQ conversations might complete at 90%. Complex order flows might complete at 60%. The gaps tell you where to invest in improvement.
Handover rate
How often does the bot trigger a human handover? A high handover rate (above 20–30%) might mean your bot's knowledge base is too thin or your trigger thresholds are too aggressive. A very low handover rate might mean the bot is attempting things it shouldn't.
Track handover rate alongside completion rate. A bot with a 5% handover rate and 40% completion rate has a big problem — it's not escalating when it should.
Message volume
Track incoming messages by day and by hour. Volume patterns tell you when your bot is busiest, help you spot anomalies (a sudden spike might mean your bot went viral or got a spam flood), and inform capacity planning.
Error rate
How often does the bot fail entirely — no reply at all, an error message sent to the customer, or a crash? An error rate above 1% in production needs immediate attention. Even 0.1% matters if you're processing thousands of conversations a day.
Cost per conversation
If you're using an AI model with usage-based pricing, track how much each conversation costs. A support bot that costs £0.02 per conversation is excellent. One that costs £2 per conversation because it's sending 50-message threads to the model on every request needs to be optimised.
Common failure modes and fixes
Bot never replies
Symptoms: messages come in, nothing goes out. No error in the customer-facing chat.
Likely cause: the webhook isn't connected, the webhook URL has changed, or the WhatsApp API token has expired.
Fix: check that your webhook endpoint is receiving POST requests. Use a tool like ngrok or a logging proxy to verify. Rotate your API token if it's expired. In CodeWords, the connection status is visible on the dashboard — a disconnected indicator is your first check.
Bot replies twice
Symptoms: every message gets two identical (or near-identical) responses.
Likely cause: no deduplication. The webhook is being called twice — either because you have two webhook subscribers registered, or because WhatsApp retries delivery and your system processes both.
Fix: add a message ID check. When a message arrives, check if you've seen that message ID before (store IDs in Redis or a database for 60 seconds). If you have, skip processing. CodeWords handles this automatically.
Bot replies to old messages
Symptoms: when you reconnect the bot or restart it, it starts processing messages from hours or days ago.
Likely cause: no staleness check. The bot is processing every message in the queue regardless of when it was sent.
Fix: add a timestamp filter. If a message is more than 60 seconds old (or five minutes, depending on your tolerance), skip it. In your flow logic: if (message.timestamp < now - 300) { return; }.
Bot replies in the wrong chat
Symptoms: replies are going to different conversations than the ones that triggered them.
Likely cause: a scope filter is missing. The bot is using a stored phone number or chat ID from a previous conversation instead of the current one.
Fix: always derive the reply-to address from the incoming message, not from a stored variable. Each message payload includes the sender's phone number — use that directly.
Bot responds to its own messages or your manual replies
Symptoms: an infinite loop of bot messages, or the bot replies when you reply manually from your phone.
Likely cause: no direction gate. The bot is processing all messages, including outbound ones.
Fix: filter on message direction. Only process inbound messages (messages where the sender is not your own number). This is sometimes called an "is not from me" check, and it's essential for Personal Device connections in particular.
How to add a staleness check
The staleness check is one of the most important — and most overlooked — safeguards in a WhatsApp bot.
Here's the logic in plain English: when a message arrives, compare its timestamp to the current time. If the difference is more than five minutes (300 seconds), the message is stale — it was probably queued during a downtime and you don't want to process it now.
Tell Cody in CodeWords: "Ignore any message that was sent more than five minutes ago." Cody builds the staleness check into the flow automatically.
Logging best practices
Log enough to debug
At minimum, log:
- Message ID (for deduplication checks)
- Timestamp received
- Message direction (inbound/outbound)
- Flow outcome (completed, escalated, errored)
- Response time
These fields let you reconstruct what happened in any conversation without storing the actual message content.
Avoid PII in logs
Don't log phone numbers in plain text, and don't log message content unless you have a specific reason and appropriate data handling controls. A log file containing customer messages is a data liability.
If you need to debug a specific conversation, use the conversation ID or message ID as a lookup key — retrieve the message from WhatsApp's storage rather than keeping your own copy.
Set log retention limits
Logs older than 30 days are rarely useful for debugging. Set a retention policy and stick to it.
Testing in production safely
The safest way to test in production is to use an allowlist. Build a version of your bot that only responds to specific phone numbers — yours, your team's — and deploy that first. Once you're satisfied it's working correctly, open it up to all numbers.
For incremental changes — updating the system prompt, adding a new integration — consider a gradual rollout: run the new version for 10% of conversations, monitor the metrics, and expand if they hold.
Never test by sending yourself a message and hoping for the best. Build a test checklist: the specific messages you send, the responses you expect, and the edge cases you know have caused problems before.
How to monitor and debug your WhatsApp AI agent with CodeWords
When you build with CodeWords, many of these safeguards — deduplication, staleness checks, direction gates — are handled by Cody automatically when you describe your agent. The monitoring dashboard gives you visibility into message volume and errors without needing to build your own logging infrastructure.
For more complex monitoring needs, Cody can connect your agent to external logging services via the 3,000+ integrations available through Composio.
Related reading