Automating the takeaway telephone order
We're increasingly seeing AI applied to all sorts of interesting areas. Today I can share something truly groundbreaking... ordering a succulent Chinese meal. This is AI manifest.
The moving parts are Twilio for the telephony, ElevenLabs for the voice agent, Firecrawl for scraping the menu, and MCP Go to tie it all together.
Setting Up Twilio
First we need a phone number from Twilio. A trial account is fine for testing, though it will only call verified numbers, so to ring anyone else you'll need to upgrade.
Ideally select a phone number suited to your local area.
UK numbers need some basic identity verification for regulatory reasons. Nothing gets exposed publicly, it's just bureaucratic box-ticking. Once your regulatory bundle is approved, pick a number and buy it.
You'll also want the call routed through a Twilio location close to wherever the number is actually being used, they offer a few, as it helps cut down on latency. That's all the Twilio setup we need for now.
Creating the ElevenLabs Agent
ElevenLabs handles the agent's behaviour and its voice. You can train a custom voice (as I did) or use one of their presets, ideally something conversational given where it's going to be used. The free credits disappear quickly during testing, so I subscribed for more.
Once signed up, select "Conversational AI" from the dropdown, go to "Agents" and create a new blank agent. Name it whatever you like, pick your voice, then set the system prompt. This is the one I've been using with some success:
## Personality
You are a customer calling a restaurant to place a takeaway order.
You are polite, patient, and clear about your order.
You speak in a friendly, natural way and use polite language like "please" and "thank you."
You are enthusiastic about food and show appreciation when staff are helpful.
## Environment
You are calling the restaurant over the phone.
This is a voice conversation, so speak clearly and avoid visual references.
Restaurant staff may be busy, so be concise but complete.
Background noise is normal - be prepared to repeat information if needed.
The current time is {{system__time_utc}}.
## Tone
- **Friendly and upbeat:** "Hi there!" "That sounds great!" "Perfect!"
- **Patient and understanding:** "No worries" "That's fine" "I understand"
- **Clear and organized:** Speak one item at a time, pause between items
- **Naturally conversational:** Use contractions, casual but polite language
- **Appreciative:** "Thank you so much" "I appreciate it" "That's wonderful"
## Goal
Your primary objective is to place this specific order for collection: {{order_items}}
Follow these steps in order:
1. Wait for the restaurant greeting, then respond with a friendly greeting
2. State clearly: "I'd like to place an order for collection, please"
3. Read through your order items one by one, speaking clearly
4. Listen for staff confirmation and respond appropriately
5. Provide your name when asked: "The name is {{customer_name}}"
6. Ask about pickup time if not provided: "What time would be good for pickup?"
8. Thank the staff politely and end the call
## Guardrails
- Stay focused on placing your specific order
- Do not ask for information unrelated to your order
- Avoid inappropriate or offensive requests
- If the restaurant is closed or cannot fulfill the order, politely end the call
- Do not make up menu items or prices
- Keep the conversation professional but friendly
- If asked for your phone number, politely decline.
## Dynamic Variables
- **Customer Name:** {{customer_name}}
- **Order Items:** {{order_items}}
**Remember:** You are a real customer with a specific order to place. Be natural, friendly, and focused on completing this transaction successfully.
The {{order_items}} and {{customer_name}} variables get filled in by our MCP server with the actual order details.
I've got the agent running on Gemini 2.0 Flash with ElevenLabs' Flash voice model, both chosen for speed over raw capability, and the temperature down at 0.1, as I'd rather it stuck to the instructions than got creative with my order.
The prompt is very much a work in progress, and it's the sort of thing you'd naturally keep iterating on (with evals, for example) to catch different behaviours or to suit a particular model. As a starting point though, it's good enough.
The Karl-Bot 3000
Next, connect the agent to Twilio: select "Phone Number" from the menu and import it using the account SID, auth token and phone number from your Twilio account.
Building the Order MCP
The MCP server looks after the whole flow, from finding the menu, to building the order, to kicking off the call. There are three parts to it.
First, I used Firecrawl to pull my local Chinese restaurant's menu off their website. Through Firecrawl's playground, I scraped the site and had it hand back structured JSON, then a quick pass through an LLM tidied the data into a consistent format.
That menu then gets embedded directly in the MCP server, so both the user interface and the agent have the full picture of what's on offer. Which matters when the agent has to handle a substitution or a change mid-call.
Dry noodles, always
The flow works like this: you query the menu through the MCP interface, build up your order, then trigger the ElevenLabs API to start the phone call. The agent gets the order details through those template variables and from there is largely left to get on with it - handling questions, clarifications, and even suggesting alternatives when something isn't available - which it can do because it has the menu to hand.
The Lag Factor
You'll notice some latency in the demo call. ElevenLabs Flash turns text into speech in around 75ms, which is quick, so the lag comes from everything else in the chain: the call itself, the LLM thinking, and the fact that all of this is spread across infrastructure in different places.
The problem is geography. Our humble takeaway order bounces between Twilio's call routing, ElevenLabs' mostly US-based infrastructure, and Google's Gemini servers scattered across various data centres. In the demo, Gemini was taking about a second to respond, which is reasonable for an LLM, though painfully slow for a conversation.
I'd expect the infrastructure to consolidate as this stuff matures. You could get rid of much of the latency today by building a custom agent and running everything from the same region, or even more by running the LLMs locally. For proving the concept works though, the current setup does the job.
Beyond Takeaway
This experiment probably isn't practical for most people in its current form, but it touches on something I've been exploring as Software 3.0: the move towards more intentional, context-aware systems that understand the outcome you're after as well as the command you typed.
Imagine an AI that knows every local menu, so ordering food stops being a research project. Say "I'm craving crispy duck tonight" and it already knows what you like, finds the highest-rated place nearby and takes your dietary requirements into account. Everything the model knows gets pointed at a purely transactional job, one where the human touch adds very little, and you're left with fewer decisions to make.
Of course, a phone call isn't exactly an efficient way to move data about, and we've long since built better protocols for this (Deliveroo, say). What a voice agent gives you though is a universal translator, an adapter between our digital intentions and a world that is, quite often, still analogue.
Today it's a playful experiment, but I can see where it goes. The voice ordering my noodles could place a call for someone with speech difficulties. The model that got through a conversation with a restaurant could handle any of the routine interactions that break up a day of focused work. It's early days, but there's a flexibility to these agents that suggests a lot of everyday friction going away, without losing the human contact we'd actually miss. I'd say that's the most useful kind of innovation too, the kind that makes life slightly less annoying.
Anyway, that's all for now. Time for dinner.