I could feel it during calls. You stop talking, there’s a short pause, and then the bot responds. It felt like maybe a second. Not bad - definitely fast enough to have a conversation. But “it feels like about a second” isn’t a measurement. And if I wanted to make it faster, I needed to know where that time was actually going.

Is it the speech recognition? The LLM thinking? The text-to-speech? The network round-trips between all these services? Without numbers, I’d just be guessing.

Turns out, the answer was hiding in plain sight. Pipecat had been tracking this data all along - I just wasn’t listening.

The Data Was Already There

Here’s something that surprised me. In my bot code, I’d had these two flags set from the very beginning:

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
)

See enable_metrics=True and enable_usage_metrics=True? Those tell Pipecat to measure the performance of every service in the pipeline - how long Deepgram takes to return the first transcription, how long Groq takes to start generating tokens, how long the TTS takes to produce audio. Every time audio flows through the pipeline, Pipecat wraps these measurements into something called a MetricsFrame and sends it downstream.

But here’s the thing: nobody was listening. The metrics frames were flowing through the pipeline and getting silently discarded at the end. It’s like having a speedometer in your car that works perfectly - you just never look at it.

What I needed was an observer.

Pipecat’s Observer Pattern

Pipecat has a clean concept for this: observers. An observer is a side-channel listener that sees every frame flowing between processors in the pipeline, but it’s not in the pipeline. It can’t block anything or slow things down. It just watches.

Think of it like a security camera in a factory. The assembly line keeps running exactly the same whether the cameras are on or off. But when they’re on, you can see exactly what’s happening at every step.

Pipecat ships with several built-in observers, and two of them were exactly what I needed:

• MetricsLogObserver - Logs every metrics frame to the console: TTFB (time to first byte) for each service, token usage, TTS character counts, Smart Turn decision metrics

• UserBotLatencyLogObserver - Measures the wall-clock time from when the user stops speaking to when the bot starts speaking, and gives you a summary at the end of the call

Adding them took three lines of code.

Three Lines to Instant Visibility

Here’s the change. I added the observer imports and passed them to the PipelineTask:

from pipecat.observers.loggers.metrics_log_observer import MetricsLogObserver
from pipecat.observers.loggers.user_bot_latency_log_observer import UserBotLatencyLogObserver

# ... existing pipeline setup ...

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
    observers=[
        MetricsLogObserver(),
        UserBotLatencyLogObserver(),
    ],
)

That’s it. No other changes. The pipeline runs exactly the same as before - the observers are purely additive.

Now when I made a test call, the logs lit up with data I’d never seen before:

DEBUG | 📊 [DeepgramSTTService#0] TTFB (nova-3): 0.200s at 14.231s
DEBUG | 📊 [LocalSmartTurnAnalyzerV3#0] SMART TURN: COMPLETE (probability: 95.10%, inference: 11.2ms, server: 14.8ms, e2e: 91.3ms) at 14.322s
DEBUG | 📊 [GroqLLMService#0] TTFB (llama-3.3-70b-versatile): 0.268s at 14.590s
DEBUG | 📊 [GroqLLMService#0] LLM TOKEN USAGE (llama-3.3-70b-versatile): prompt: 290, completion: 17, total: 307 at 14.95s
DEBUG | 📊 [DeepgramTTSService#0] TTFB (aura-2-theia-en): 0.044s at 14.634s
DEBUG | 📊 [DeepgramTTSService#0] TTS USAGE (aura-2-theia-en): 32 characters at 14.95s
DEBUG | ⏱️ LATENCY FROM USER STOPPED SPEAKING TO BOT STARTED SPEAKING: 0.976s

Let me break down what each line means:

• STT TTFB: 0.200s - Deepgram took 200 milliseconds to return the first word of the transcription after receiving audio

• Smart Turn: 91.3ms e2e - The turn detection model took 91ms to decide (with 95.1% confidence) that I was actually done speaking, not just pausing

• LLM TTFB: 0.268s - Groq took 268 milliseconds to start generating the first token of the response

• LLM Token Usage: 290 prompt / 17 completion - The context size and response length for this turn

• TTS TTFB: 0.044s - Deepgram’s TTS took 44 milliseconds to start producing audio from the first text it received

• Total latency: 0.976s - The wall-clock time from when I stopped talking to when I heard the bot start responding

At the end of the call, UserBotLatencyLogObserver printed a summary:

INFO | ⏱️ LATENCY FROM USER STOPPED SPEAKING TO BOT STARTED SPEAKING - Avg: 1.048s, Min: 0.976s, Max: 1.159s

This was already revealing. But I wanted more detail - specifically, I wanted to see the breakdown per turn in a single table, so I could spot patterns across a conversation.

Building a Custom Observer

The built-in observers are great for a quick look, but their log lines aren’t correlated per turn. You can’t easily see “for turn 3, what was the STT latency vs. the LLM latency?” To get that, I built a custom observer called LatencyBreakdownObserver.

Building a custom observer is straightforward. You subclass BaseObserver and implement on_push_frame, which gets called every time a frame passes between two processors in the pipeline:

from pipecat.observers.base_observer import BaseObserver, FramePushed

class LatencyBreakdownObserver(BaseObserver):
    async def on_push_frame(self, data: FramePushed) -> None:
        # data.frame is the frame being passed
        # data.source is the processor sending it
        # data.destination is the processor receiving it
        # data.direction is DOWNSTREAM or UPSTREAM
        ...

The key insight is that certain frame types mark the boundaries of a conversation turn:

• VADUserStoppedSpeakingFrame - The user finished talking. Start the clock.

• MetricsFrame - Contains TTFB measurements from each service. Capture them.

• BotStartedSpeakingFrame - The bot started talking. Stop the clock.

• EndFrame / CancelFrame - The call is over. Print the summary.

I created a simple dataclass to hold per-turn data:

@dataclass
class TurnLatency:
    turn_number: int
    stt_ttfb: Optional[float] = None          # seconds
    smart_turn_e2e_ms: Optional[float] = None  # milliseconds
    llm_ttfb: Optional[float] = None           # seconds
    tts_ttfb: Optional[float] = None           # seconds
    total_wall_clock: Optional[float] = None   # seconds
    llm_prompt_tokens: Optional[int] = None
    llm_completion_tokens: Optional[int] = None
    tts_characters: Optional[int] = None

The observer collects one TurnLatency per conversation turn. When it sees a VADUserStoppedSpeakingFrame, it starts a new turn and records the wall-clock time. As MetricsFrame objects arrive, it routes each measurement to the right field by checking the processor name (e.g., "DeepgramSTTService" for STT, "GroqLLMService" for the LLM). When BotStartedSpeakingFrame arrives, it stops the clock and saves the turn.

The full implementation is in observers.py - it’s about 100 lines of focused code.

I added it alongside the built-in observers:

from observers import LatencyBreakdownObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
    observers=[
        MetricsLogObserver(),
        UserBotLatencyLogObserver(),
        LatencyBreakdownObserver(),
    ],
)

The Numbers: Where Does 1 Second Go?

Here’s what the summary table looks like after a real call. This is the payoff - the table prints to the logs when the call ends:

=== LATENCY BREAKDOWN (4 turns) ===
Turn | Total  | STT TTFB | Smart Turn | LLM TTFB | TTS TTFB | LLM Tokens | TTS Chars
-----+--------+----------+------------+----------+----------+------------+----------
   2 | 0.976s |   0.200s |       91ms |   0.268s |   0.044s |     290/17 |       32
   3 | 1.001s |   0.266s |       82ms |   0.268s |   0.029s |     323/33 |      111
   4 | 1.057s |   0.155s |       84ms |   0.268s |   0.013s |     388/84 |      228
   6 | 1.159s |   0.157s |       81ms |   0.271s |   0.006s |     504/33 |       13
-----+--------+----------+------------+----------+----------+------------+----------
 Avg | 1.048s |   0.195s |       84ms |   0.268s |   0.023s |            |

A few things jump out immediately.

The LLM is the slowest component. At 268ms average TTFB, the LLM takes more time than any other service. This is still a 70B parameter model running on Groq, but it’s notably slower than the sub-100ms performance you might expect. This could be due to API queuing, model load, or the specific context size.

The AI services combined take about 570ms. Add up the averages: STT (195ms) + Smart Turn (84ms) + LLM (268ms) + TTS (23ms) = roughly 570ms. That’s the time these services spend doing their thing.

So where’s the other 480ms? The total averages 1.048 seconds, but only ~570ms is accounted for by service TTFB. The rest is pipeline overhead: network round-trips between your server and each external API (Twilio to Modal, Modal to Deepgram, Modal to Groq, Modal to Deepgram again), audio buffering, frame serialisation, and the VAD’s own silence detection window before VADUserStoppedSpeakingFrame even fires.

Turn 4 generated the longest response. With 84 completion tokens and 228 TTS characters, it had the most content to generate. Interestingly, the LLM TTFB stayed constant at 268ms regardless of response length - time to first token doesn’t depend on how many tokens will eventually be generated.

Context grows, but TTFB stays remarkably flat. Notice how the prompt token count grows from 290 to 504 across the conversation (each turn adds to the context), but LLM TTFB barely changes (268ms to 271ms). Despite doubling the context size, inference latency remains essentially constant.

What Would Actually Make It Faster?

Now that we have numbers, we can think about optimisation rationally instead of guessing.

The LLM is the biggest opportunity. At 268ms average TTFB, the LLM accounts for over 25% of the total latency. This could potentially be improved by using a smaller/faster model, ensuring optimal Groq API configuration, or exploring self-hosted LLM inference with a model optimized for low latency. The question is whether faster inference would compromise response quality.

Network round-trips and pipeline overhead matter. About 480ms (nearly half the total latency) is spent on overhead: network round-trips between your server and each external API (Twilio to Modal, Modal to Deepgram, Modal to Groq, Modal to Deepgram again), audio buffering, frame serialisation, and the VAD’s own silence detection window before VADUserStoppedSpeakingFrame even fires. Self-hosting these services on the same infrastructure could eliminate many of these round-trips. The location of each component, for example the region where these components are deployed to will also have a bearing on the latency.

Smart Turn is surprisingly fast. At just 84ms average, the turn detection is much faster than expected and not a bottleneck. The VAD’s stop_secs parameter (likely set to around 0.3-0.5s) adds some additional latency, but further reduction would risk more false positives.

STT and TTS are well-optimized. Deepgram STT at 195ms and TTS at 23ms are both performing well. There’s not much to gain here without sacrificing quality.

Try It Yourself

All the code for this - the observers, the bot changes, everything - is in the latency/ directory of the GitHub repo.

It’s a self-contained version of the inbound bot with the observers wired up. Deploy it with:

cd twilio-chatbot/latency
cp env.example .env
# Fill in your API keys in .env
modal serve modal_app.py

Make a call, and you’ll see the metrics flowing in your logs. The observers work identically with the outbound bot too - just add the same three imports and the observers parameter to the PipelineTask.

What I find most interesting about these numbers is what they suggest about the next step. The API services are fast, but the network overhead is significant. What if you could run your own STT, LLM, and TTS on the same GPU, eliminating the network hops entirely? Pipecat already supports self-hosted alternatives like Faster-Whisper for STT, vLLM or Ollama for the LLM, and XTTS for TTS. The question is: would self-hosting actually be faster - and at what cost?

That’s exactly what I’m going to find out next.

But the moment it worked, a different question started bugging me: what if the bot could call someone else?

Think about it. An inbound bot waits for people to call. That’s useful - it’s basically an AI receptionist. But an outbound bot? That opens up a whole different set of possibilities. Appointment reminders that actually talk to you. Customer check-in calls. Survey bots. Automated outreach. Even something as simple as a daily wake-up call from a friendly AI voice.

The idea seemed harder at first. With inbound, the caller starts things - Twilio just hands the call to your bot. With outbound, you have to start the call. Your code has to tell Twilio “dial this number,” wait for the person to pick up, and then connect the AI. It’s a different dance.

But it turns out the core of it - the AI pipeline that listens, thinks, and talks back - is exactly the same. The only difference is the plumbing around it. And once you understand that plumbing, it clicks.

Here’s a recording of the outbound bot in action - it dials a number, the person picks up, and the conversation flows naturally:

In this article, I’ll walk through how I built the outbound bot. If you haven’t read the first article on the inbound bot, I’d recommend starting there - I cover the basics of the tech stack (Pipecat, Groq, Deepgram, and Modal) in more detail. But this article should still make sense on its own.

Inbound vs. Outbound: What Actually Changes?

Here’s the good news: most of the code is the same. The AI pipeline - the part that listens to speech, converts it to text, generates a response, and speaks it back - is identical. Same Pipecat pipeline. Same Groq LLM. Same Deepgram for speech recognition and text-to-speech. Same noise filtering. Same smart turn detection.

What changes is who starts the call and how the connection gets set up.

With the inbound bot:

1. Someone calls your Twilio number

2. Twilio asks your server for instructions on what to do

3. Your server returns TwiML that tells Twilio to “open a WebSocket and stream the audio”

4. The pipeline handles the conversation

With the outbound bot:

1. You send a request to your own server: “call this number”

2. Your server tells Twilio: “dial +1234567890 and when they pick up, ask me what to do next”

3. The person answers their phone

4. Twilio asks your server for instructions on what to do

5. Your server returns TwiML that tells Twilio to “open a WebSocket and stream the audio”

6. The pipeline handles the conversation

See the difference? There’s one extra step at the beginning - you kick things off instead of waiting. After the person picks up, everything is the same.

But that one extra step means our server needs an additional endpoint (a URL on your server that accepts requests) to handle it. Which brings us to the interesting part.

The Three-Endpoint Dance

The inbound bot needed just two endpoints:

• POST / - Twilio hits this when someone calls. We respond with instructions.

• WebSocket /ws - Where the real-time audio streams back and forth.

The outbound bot needs three:

1. POST /dialout - “Start a call”

This is where it all begins. You send a request to this endpoint with two pieces of information: the phone number you want to call and the Twilio phone number to call from.

POST /dialout
{
  "to_number": "+15551234567",
  "from_number": "+15559876543"
}

When this endpoint receives your request, it uses the Twilio REST API to start a phone call. It tells Twilio: “Dial this number, and when the person picks up, go to my /twiml endpoint to get instructions.”

Twilio starts ringing the phone. You get back a response with a call SID so you can track it.

2. POST /twiml - “What to do when they answer”

When the person picks up, Twilio needs to know what to do with the call. So it reaches out to this endpoint. Our server responds with TwiML (Twilio’s XML-based instruction language - think of it as a short instruction sheet that tells Twilio how to handle a call) that says: “Open a WebSocket connection to /ws and start streaming audio.”

There’s one clever detail here: we pass the phone numbers (who’s calling, who’s being called) as parameters in the WebSocket stream. This way, the bot knows who it’s talking to and can personalise the conversation.

3. WebSocket /ws - “The actual conversation”

This is the same WebSocket endpoint from the inbound bot. Once it’s open, audio flows in both directions through Twilio Media Streams and the Pipecat pipeline does its thing - listen, think, respond.

Here’s the whole flow as a diagram:

The key insight: the three endpoints form a chain. You trigger the first one, and the rest happen automatically. From the person receiving the call, it just feels like their phone rang and someone started talking to them.

The Pipeline: Same Trusted Stack

The AI pipeline is nearly identical to the inbound version. Same building blocks:

• Deepgram nova-3 for speech-to-text (turning what the person says into text)

• Groq with Llama 3.3 70B for generating responses

• Deepgram Amalthea voice for text-to-speech (turning the response back into natural-sounding audio with a Filipino English accent)

• RNNoise for cleaning up background noise

• Silero VAD + Smart Turn v3 for knowing when the person has finished talking

The main difference is the system prompt. Since the bot is the one making the call, it needs to introduce itself and explain why it’s calling - you wouldn’t want it to just sit there waiting for the other person to speak first. The prompt tells the bot:

“You are a friendly assistant making an outbound phone call. Your responses will be read aloud, so keep them concise and conversational. Avoid special characters or formatting. Begin by politely greeting the person and explaining why you’re calling.”

And just like the inbound bot, if the person goes quiet for more than 5 seconds, the bot picks up the conversation and keeps things moving.

Setting Up and Making Your First Call

The setup is almost the same as the inbound bot. You need the same accounts (Twilio, Groq, Deepgram, Modal), plus one difference: the outbound bot also uses the twilio Python library to make API calls for starting calls. All of these services have free tiers - a handful of test calls won’t come close to using up the free credits.

Deploy to Modal

cd twilio-chatbot/outbound
cp env.example .env
# Fill in your API keys in .env
modal serve modal_app.py

This gives you a URL like:

https://your-workspace--twilio-outbound-bot-serve.modal.run

Make a Call

Now the fun part. To make the bot call someone, you just send a request:

curl -X POST https://your-workspace--twilio-outbound-bot-serve.modal.run/dialout \
  -H “Content-Type: application/json” \
  -d ‘{”to_number”: “+15551234567”, “from_number”: “+15559876543”}’

Replace to_number with the number you want to call and from_number with your Twilio phone number.

The repo also includes a handy test script that reads the numbers from your .env file:

cd scripts
uv run python test_call.py https://your-workspace--twilio-outbound-bot-serve.modal.run

A few seconds after you run this, the phone will ring. When you pick up, the bot will greet you and start a conversation. Here’s what to expect: you’ll hear a brief pause (Modal needs to spin up a worker), then the bot will introduce itself. Say something back - you’re having a conversation with your own outbound voice AI.

Notice something? With the inbound bot, you had to configure a webhook in the Twilio Console. With outbound, you don’t. Your code is the one calling Twilio - not the other way around. Twilio only calls your /twiml endpoint because your code tells it to (in the /dialout handler). No console configuration needed.

Recordings

Every call is automatically recorded using Twilio’s built-in call recording. When the WebSocket connects, the bot starts a dual-channel recording via the Twilio REST API. Recordings are stored in your Twilio account and accessible through the Twilio Console or API - no application-level storage needed.

This keeps things simple: no recording files to manage, no volumes to mount, and Twilio handles the audio encoding and storage for you.

Running Locally with Docker

Modal is great for quick deployments, but sometimes you want to run things on your own machine - maybe for faster iteration, or to see the logs in real time, or just because you want to understand what’s happening without a cloud layer in between.

The outbound bot supports this through Docker.

There’s one catch, though: Twilio needs to reach your server from the internet to fetch TwiML and connect the WebSocket. Your laptop is usually behind a firewall (meaning the outside world can’t reach it directly), so Twilio can’t talk to it.

The solution is a tunnel - a tool that gives your local machine a temporary public URL. VS Code has this built in (they call it “dev tunnels“), and it works well:

1. In VS Code, open the Ports panel

2. Forward port 7860 and set it to Public (Twilio can’t use authenticated tunnels)

3. Copy the forwarded address - it’ll look like

https://something-7860-region.devtunnels.ms

Now run the bot:

cd outbound
export PROXY_HOST=something-7860-region.devtunnels.ms
docker compose up --build

The PROXY_HOST variable is important - it tells the bot what public URL to put in the TwiML response so Twilio knows where to open the WebSocket. Without it, Twilio would try to connect to localhost, which obviously won’t work.

Once the container is running, you can trigger calls the same way - just point the curl command at your tunnel URL instead of the Modal URL.

Recordings in Docker mode get saved to a ./recordings/ folder on your machine (mounted as a Docker volume), so you can listen to them right away. See the repo’s README for the full Docker setup details.

Wrapping Up

Over these two articles, we’ve built two complete voice AI applications:

1. Inbound: An AI that answers phone calls and has conversations with callers

2. Outbound: An AI that dials phone numbers and starts conversations with people

The core is the same - a Pipecat pipeline that connects speech recognition, an LLM, and text-to-speech into a real-time conversation engine. What changes between inbound and outbound is just the setup: who starts the call and how the connection gets wired up.

The beauty of this pipeline approach is that every piece is swappable. Want a different LLM? Swap out Groq for OpenAI or Anthropic. Want a different voice? Change the Deepgram TTS model. Want the bot to behave differently? Edit the system prompt. The pipeline structure stays the same.

A few ideas for where you could take this next:

• Connect to a CRM - Look up the caller’s info and personalise the conversation

• Appointment reminders - Schedule outbound calls with specific messages

• Surveys and feedback - Have the bot ask questions and record the answers

• Multi-language support - Swap in STT and TTS models for different languages

The full code for both bots is in the GitHub repo. If you build something with it, I’d love to hear about it.

My first thought was: “That’s cool, but it’s probably built by a team of 20 engineers at some well-funded startup.”

My second thought was: “...but what if it’s not?”

So I went down the rabbit hole. I’ve been a full-stack developer for years - I’m comfortable with APIs, backends, cloud infrastructure, all of that. But telephony and audio processing? That was completely new territory for me. I wanted to see if I could build a voice AI that answers real phone calls. Not a toy demo. A real phone number you can call, have a conversation with, and hang up.

Turns out, the tools have gotten really good. After a weekend of tinkering, I had a working voice bot named Miss Harper - an elementary school teacher who answers phone calls and chats with you in a friendly Australian accent. You call the number, she picks up, says hello, and you can talk to her like a normal person. She listens, thinks, and talks back - all in real time.

Don’t take my word for it - here’s a recording of an actual call with Miss Harper:

Hear Miss Harper tell me a story about a little seed named Sammy and how he became a great big tree! Miss Harper can even do basic mathematics!

In this article, I’ll walk you through exactly how I built it, step by step. By the end, you’ll have your own AI-powered phone number that picks up calls and has conversations. No prior experience with voice, audio, or telephony needed.

Let’s get into it.

What We’re Building

Here’s the end result: you’ll have a phone number. When someone calls it, an AI bot picks up, introduces itself, and has a natural voice conversation with the caller.

Under the hood, the flow looks like this:

You call a phone number
       ↓
   Twilio (handles the phone call)
       ↓
   WebSocket (streams audio back and forth)
       ↓
   Pipecat (processes the audio through AI services)
       ↓
   You hear the bot respond

The key idea is simple: it’s a pipeline. Audio goes in one end, gets turned into text, the AI generates a reply, that reply gets turned back into speech, and the speech goes back out. Everything happens in real time - under a second from when you stop talking to when you hear the response.

Here’s what the Pipecat pipeline looks like inside - each module feeds into the next, like an assembly line:

This cascading design is what makes Pipecat powerful. Each piece does one job and passes its output to the next. You can swap any piece out - use a different LLM, a different voice, a different speech recognition model - without touching the rest of the pipeline.

The Building Blocks

Before we start building, let me introduce the tools we’ll use and explain why each one matters.

Twilio is our phone company. It gives us a real phone number and handles all the complicated stuff about connecting phone calls to the internet. When someone calls your Twilio number, Twilio asks your server what to do with the call. Think of Twilio as the bridge between the old-school phone network and our modern AI code.

Pipecat is the glue that holds everything together. It’s an open-source framework that lets you build real-time audio AI applications by connecting different services into a pipeline. Instead of writing all the plumbing code yourself - handling audio streams, managing conversation turns, coordinating between services - Pipecat handles all of that. You just tell it what services to use and in what order.

Groq is the brain. It runs the large language model (LLM) - basically the same kind of AI that powers ChatGPT, but running on Groq’s super-fast hardware. We use a model called Llama 3.3 70B. The speed matters a lot here: in a phone conversation, even a half-second delay feels awkward. Groq is one of the fastest options available.

Deepgram pulls double duty - it’s both the ears and the mouth. Deepgram’s speech-to-text (STT) service listens to what the caller says and converts it to text so the AI can understand it. Then Deepgram’s text-to-speech (TTS) service takes the AI’s text response and converts it back to natural-sounding speech. We use their “Theia” voice, which has a warm Australian accent.

Silero is the social awareness layer. It handles something called voice activity detection (VAD), which is just a fancy way of saying “it knows when you’re talking and when you’ve stopped.” On top of that, we use something called Smart Turn v3, which is smarter than just listening for silence. It uses machine learning to figure out when you’ve actually finished your thought - not just paused to take a breath. This is what makes the conversation feel natural instead of robotic.

Modal is where our code runs. It’s a cloud platform that lets you deploy Python apps without setting up servers. You write your code, run one command, and it’s live on the internet with a public URL. It also auto-scales - if nobody’s calling, it costs nothing. If ten people call at once, it spins up more workers.

How It Actually Works

Let me walk you through what happens when someone calls your bot, step by step.

Step 1: The phone rings. Someone dials your Twilio phone number. This is a normal phone call - they could be calling from a mobile phone, a landline, whatever.

Step 2: Twilio asks your server what to do. The moment the call connects, Twilio sends a message to your server (an HTTP POST request) saying, “Hey, someone’s calling. What should I do?” Your server responds with a small piece of XML called TwiML (Twilio Markup Language). Think of TwiML as a short instruction sheet for Twilio - in our case it just says: “Open a WebSocket connection back to me and start streaming the audio.”

A WebSocket is just a persistent two-way connection - think of it like opening a phone line between Twilio and your server where data (in this case, audio) can flow in both directions at the same time.

Step 3: Audio starts flowing. Once the WebSocket is open, Twilio starts streaming the caller’s audio to your server in real time via Media Streams - small chunks of sound, one after another. Your server can also send audio back through the same connection, and Twilio plays it to the caller.

Step 4: The Pipecat pipeline kicks in. This is where the magic happens. The audio goes through a series of steps, one after another:

1. Noise filter - An RNNoise filter cleans up background noise from the caller’s audio. This makes the speech recognition more accurate.

2. Voice activity detection - Silero detects that someone is actually talking (as opposed to silence or background noise).

3. Speech-to-text - Deepgram’s nova-3 model converts the spoken words into text in real time.

4. AI thinking - The text goes to Groq’s Llama 3.3 model, which generates a response based on the conversation history and the bot’s personality (defined by a system prompt - more on that later).

5. Text-to-speech - Deepgram’s Theia voice converts the AI’s text response into spoken audio.

6. Audio out - The audio streams back through the WebSocket to Twilio, and Twilio plays it to the caller.

All of this happens fast - typically under a second from when you finish talking to when you hear the response. One reason it feels so quick: the LLM doesn’t wait until it’s finished writing the whole reply before handing it off. It sends words to the text-to-speech service as it generates them, so the caller starts hearing the response while the AI is still thinking of the rest. It works like an assembly line, not a batch job.

Voice-to-voice latency is the time from when the user finishes speaking to when they hear the bot’s first word of response.

It encompasses the entire chain:

1. Audio travels from the phone to Twilio to your server

2. STT converts the speech to text

3. Turn detection confirms the user is done speaking

4. LLM generates the first tokens of a response

5. TTS converts those tokens to audio

6. Audio travels back through Twilio to the caller’s phone

So in the above example, the voice to voice latency for the inbound caller application we are building is in the vicinity of around 600 to 1000ms. Not bad at all, considering we are running everything on CPU, and just using free-tier API models from Deepgram and Groq!

Step 5: Turn detection makes it natural. Here’s a subtle but important piece: how does the bot know when you’ve finished talking? Simple silence detection would be too aggressive - it would cut you off every time you pause to think. Smart Turn v3 uses machine learning to detect natural endpoints in speech. It understands the difference between a pause in the middle of a sentence and actually being done talking.

Step 6: The conversation continues. The bot keeps the full conversation history in memory for the duration of the call, so it remembers what you said earlier. (Each call starts fresh - it doesn’t remember previous calls.) If you go quiet for too long (5 seconds), the bot will take the initiative and continue the conversation on its own.

Step 7: Recording. The bot kicks off a Twilio-side recording via the REST API as soon as the call connects. Twilio records in dual-channel (one channel per participant), so you get clean, separated audio without any memory overhead in your application. Recordings are stored in your Twilio account and accessible from the Twilio Console or via the API.

Setting Up Your Own

Ready to build it? Here’s what you need.

Accounts to Create (All Free Tiers Available)

You’ll need accounts with four services. All of them have free tiers, and a handful of test calls won’t come close to using up the free credits.

1. Twilio - Sign up at twilio.com. You’ll need a phone number that supports voice calls. Twilio’s trial account gives you one for free.

2. Groq - Get an API key at console.groq.com. The free tier is generous enough for testing.

3. Deepgram - Get an API key at deepgram.com. Free credits are included when you sign up.

4. Modal - Sign up at modal.com. The free tier gives you enough compute for plenty of test calls.

Deploy the Bot

First, install the Modal CLI and set it up:

pip install modal
modal setup

Clone the repo and set up your keys:

git clone https://github.com/jaeyow/twilio-chatbot.git
cd twilio-chatbot/inbound
cp env.example .env

Open the .env file and fill in your API keys:

GROQ_API_KEY=your_groq_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
TWILIO_ACCOUNT_SID=your_twilio_account_sid
TWILIO_AUTH_TOKEN=your_twilio_auth_token

Now deploy to Modal:

modal serve modal_app.py

This gives you a public URL that looks something like:

https://your-workspace--twilio-inbound-bot-serve.modal.run

Point Twilio to Your Bot

The last step is telling Twilio to send incoming calls to your bot:

1. Go to your Twilio Console

2. Navigate to Phone Numbers → Manage → Active numbers

3. Click on your phone number

4. Under Voice Configuration, set “A call comes in” to:

   • Webhook

   • URL: paste your Modal URL

   • Method: POST

5. Click Save configuration

That’s it. Call your Twilio number. It might take a second or two to connect the first time (Modal needs to spin up a worker), but then Miss Harper will pick up and greet you. Say something back - you’re having a conversation with your own voice AI.

Running Locally with Docker (Alternative)

If you’d rather run the bot on your own machine instead of Modal, you can use Docker Compose. The main thing you’ll need is a way for Twilio to reach your laptop from the internet, since Twilio needs a public URL to send audio to. VS Code has built-in dev tunnels that work well for this.

The steps are: forward port 7860 in VS Code’s Ports panel (set it to Public), grab the tunnel URL, then run:

cd inbound
export PROXY_HOST=your-tunnel-hostname
docker compose up --build

Point your Twilio webhook to the tunnel URL instead of the Modal URL, and everything works the same way. See the repo’s README for the full Docker setup details.

Giving Your Bot a Personality

The bot’s personality comes from one thing: the system prompt. This is the instruction you give the AI that tells it who it is and how to behave. Here’s the one we use:

“You are Miss Harper, an elementary school teacher in an audio call. Your output will be converted to audio so don’t include special characters in your answers. Respond to what the student said in a short sentence.”

You can change this to anything you want. Make it a customer service agent, a language tutor, a trivia host - whatever fits your use case. Just remember to tell it that responses will be read aloud, so it should avoid things like bullet points, URLs, or special characters.

You can also change the voice. We use Deepgram’s “aura-2-theia-en” voice (a warm Australian accent), but they have other voices to choose from.

One more thing worth mentioning: if the caller goes quiet for more than 5 seconds, the bot doesn’t just sit there in awkward silence. It takes the lead and continues the conversation. In our case, Miss Harper keeps teaching. You can adjust this timeout or change what the bot says when it kicks in.

What’s Next

So there you have it - a voice AI that answers real phone calls, built with open-source tools and free-tier cloud services. The whole thing is about 270 lines of Python spread across two files.

But here’s what kept nagging me after I got this working: the bot only talks when I call it. What if I want the bot to pick up the phone and call someone else?

That’s exactly what I built next - an outbound voice bot that dials people and starts conversations. I’ll walk through that in the next article.

If you want to see the full code, it’s all in the GitHub repo. Questions or ideas? I’d love to hear from you.

But how do we stop guessing and start fixing? It all starts with Traces.

If an LLM application is an aeroplane, a Trace is the Flight Data Recorder (the “Black Box”). When the plane lands safely or crashes, you don’t just look at the wreckage; you look at the logs to see exactly what the sensors were reporting at every altitude.

What are we building again?

To keep things simple, I’ve been using a Streamlit-based interface for JO’s Bike Shop. It looks like a standard chat window, but behind the scenes, it’s powered by a state machine that tracks whether the user is just “browsing” or trying to “book” a repair.

Step One: Trace Collection

You cannot evaluate what you haven’t recorded. Before you can build fancy “AI Evaluators” or automated tests, you need a library of real (or high-quality synthetic) conversations. Go synthetic is especially critical when you have a brand new application, where you need “real-world” traces, but don’t have real users to collect real traces from.

I use Apache Burr for this. Burr is a state-management library that essentially takes a “snapshot” of the chatbot’s brain at every turn. It records:

• The Input: What the user said.

• The State: What the chatbot “thought” was happening (e.g., “User is asking about Saturday hours”).

• The Result: The actual text sent back.

While I’m using Burr because it fits my workflow, the tool itself matters less than the habit. Whether you use LangGraph or say Pydantic AI, these libraries allow you to save traces just like Burr, although you’ll have to confirm by yourself if they allow you to direct them to a local OTel collectors or even to local files. Or even just save your own JSON logs to a database; tracing is the fundamental foundation of all evaluation work.

What does a Trace actually look like?

It’s not just a chat log. It’s a data object. Here is a simplified snapshot of a trace from a conversation turn of our chatbot:

{
  "turn_number": 3,
  "messages": [
    {
      "role": "user",
      "content": "Do I need to book ahead or can I just walk in?",
    },
    {
      "role": "assistant",
      "content": "Hey there! At JO's Bike Shop, it's best to book ahead for our services to ensure we can give your bike the attention it deserves. Give us a buzz at (503) 555-BIKE or drop by and chat with our team to schedule your tune-up or maintenance. Can't wait to get you back on the road! \ud83d\udeb2",
    }
  ]
},
{
  "turn_number": 4,
  "messages": [
    {
      "role": "user",
      "content": "Cool. I\u2019d like to drop my bike in tomorrow morning 10am then. Is that usually a quiet time?",
    },
    {
      "role": "assistant",
      "content": "Perfect! Let me confirm your appointment:\n\n\u2022 Service: Brake and Gear Cable Replacement\n\u2022 Date: tomorrow\n\u2022 Time: 10am\n\u2022 Issues: squeaky brakes\n\nDoes this look correct? I can book this for you now!",
    }
  ]
}

Looking at this JSON, in Turn 3 the conversation could easily have turned into a hallucination, since the shop requires advance booking for brake and cable tune-ups, which this customer needs for their bike. There are more trace details in the actual JSON file here, including the state of the chat application at each turn.

Without this trace, it would be hard to determine, when running your evaluations, whether your bot responds with hallucinations or provides details grounded in facts. This is the foundation for building evaluations for our chatbot.

Synthetic Data

When you’re building your application, especially when you haven’t released it yet, there will be a shortage of real-world users from whom you can collect traces. One technique is to generate synthetic traces that we can use for error analysis. There are a couple of strategies for this.

Single Turn Conversations

Building single-turn conversations is simpler. First, we generate the scenarios, review the notebook where we generate unique dimension tuples, and use an LLM to generate the scenarios. Once the tuple scenarios have been created, we then use an LLM again to run these synthetic questions and allow our chatbot to respond.

Multi-turn Conversations

Multiple-turn conversations are a bit more complicated to build synthetic conversations from. However, they are not much more difficult with the help of some smart people in the community. One library, called Deep Eval, has a feature called Conversation Simulator that allows us to simulate full conversations between a synthetic user and our chatbot.

Step Two. Open Coding - Naming Failure Patterns

Imagine you’ve run 50 test conversations through JO’s Bike Shop Chatbot. You have 50 JSON conversation traces. If you just glance at them, you’ll think, “Yeah, it seems a bit buggy.” But “a bit buggy” is not an engineering requirement. You can’t fix “buggy.”

Open Coding is the process of reading those traces and assigning a short descriptive label to them. It has nothing to do with source code at all. This process is not something I have invented. In fact, it is a term adapted from Grounded Theory, a research method in which you build a theory directly from the data collected. Those customer or user survey feedback forms that many providers ask you to fill in? Yes, understanding and finding insights from them are most definitely achieved by the same methods we use here. 

How I did it with the Bike Shop Traces

I sat down with a croissant and a cup of coffee and opened both single and multi-turn traces. Every time the bot said something that didn’t align with the shop’s “Source of Truth” (my fictional handbook), I assigned it a label.

Here is what that looked like in practice:

• The Trace: User asks about e-bikes. Bot says, “We do motor diagnostics!”

• My Label: User is asking a general question about e-bikes, but the bot is replying with a very specific answer about e-bike motor diagnostics.
  

• The Trace: User asks about Saturday hours. Bot says, “We close at 5 PM.”

• My Label: The AI bot answers with a “common sense” closing time of 5PM; however, Saturday (or weekend) hours are different.
  

• The Trace: User says, “I’ll come in at 9:30 AM.” Bot says, “See you then!”

• My Label: The AI Bot is too agreeable and always says yes by default, without taking into account the actual opening time on that day.

The “Aha!” Moment

The magic of Open Coding is that you don’t start with a list of errors. You let the emerging error labels tell you what they are. In my first 10 traces, I noticed the bot kept telling people they didn’t need an appointment. I labelled them with similar descriptions. By the time I reached trace #15, I realised I had used that same label 6 times!

Suddenly, I wasn’t just “vibing” that the bot was wrong. I had data. I knew 40% of my failures stemmed from a failure mode in our booking policy. Fixing this particular failure mode can improve my chatbot by this much.

Why this is a “Mindset Shift”

For most of us, our instinct is to jump straight into the code and change the prompt. We see one error and we think, “Oh, I’ll just tell the LLM ‘Don’t say we do motor diagnostics’ in the system prompt.”

Stop. That is “Whack-a-Mole” engineering. Open Coding forces you to slow down. It turns a messy pile of conversations into a structured list of Failure Modes as we move into Axial Coding, which is coming next. 

Do Open Coding yourself!

If you’re doing this for the first time, don’t overthink the descriptive labels. Use plain English. I know it might be tempting to hand this activity off to a 3rd party or rely on a good LLM, which can certainly perform it in a blink of an eye.

However, the most value-adding activity you can do for your project is this part. It includes collecting traces, open coding (or adding descriptive labels), and finally axial coding, which we will discuss in the next section. Knowing how your application fails your users is the best way to improve your application!

Step Three: Axial Coding - Tidying the Notes into Categories

If Open Coding is about creating a pile of sticky notes, Axial Coding is simply the act of grouping those notes into categories.

Think of it like sorting your laundry. You have a messy pile of clothes (your descriptive labels), and you start putting them into baskets: “Whites,” “Darks,” and “Delicates.” In our case, the baskets are our Master Categories.

When I looked at my “sticky notes” for the Bike Shop, the categorisation became obvious:

• The “Bad Data” Category: I put every note about incorrect closing times, prices, or service lists here.

• The “Policy Confusion” Category: I grouped all the notes where the bot gave incorrect advice about walk-ins vs. appointments, for example.

• The “Over-Eager AI” Category: I put the notes where the bot promised motor diagnostics or firmware updates, even though that was wrong.

Why is this good

This three-step process - Collect, Describe, then Categorise - is the secret to moving beyond “vibes.”

Instead of telling your team, “The bot feels a bit hallucinatory,” you can show them a chart. You can say: “We have 15 instances of ‘Bad Data’ and 3 instances of ‘Over-Eager’ behaviour. Our priority should be updating our Saturday hours in the knowledge base, as that’s where most of our ‘Bad Data’ notes are coming from.”

You’ve turned a messy conversation into a concrete and actionable to-do list of tickets to improve our application.

Closing the Analysis Activity

You’ve now mastered the Analysis phase of the AMI (Analyse, Measure, Improve) cycle. You know how to capture the “Black Box” recordings (Traces), describe the failures (Open Coding), and group them into actionable categories (Axial Coding).

But to be honest, doing this manually for every conversation isn’t scalable.

In Part 3, we’ll tackle the Measure phase. I’ll show you how to take the categories we just created and teach another LLM to recognise them. We’re going to build Automated Evaluators so your chatbot can be audited 24/7 without you needing a croissant or even a single cup of coffee!

“Does it answer the question?” 

However, good chatbots rarely involve a one-question, one-answer exchange.

Failures often don’t show up in a single turn (though they can still happen); instead, these errors tend to appear gradually, for example, in the 3rd, 4th, or 5th turn or later. 

For this article, I have prepared a multi-turn chatbot called the JO’s Bike Shop Chatbot. Instead of the usual approach of putting everything into the prompt, I’ve opted for a graph-based chatbot using state machines, a common strategy for building chatbots that can be easier to adhere to your plans.

Using graphs or state machines to build your LLM applications allows you to create applications that you can fully customise and control, rather than relying on the “prompt and pray' approach, where you put everything in the prompt and hope the LLM model will follow all your instructions.

Let’s go through a few examples of single-turn exchanges as well as how multiple-turn conversations can lead to subtle errors as the discussion progresses.

Single-turn Examples: Subtle Failures

Typical single-turn conversations can contain subtle failures, and none of these initially seem wrong. Sometimes, the AI nearly gets these failures right. In the future, such issues could compound in multi-turn conversations.

Example 1: Over-generalisation of Services

User:
Do you service e-bikes?

LLM (subtly wrong):
Yes, JO’s Bike Shop services most e-bikes, including motor diagnostics, battery checks, and firmware updates.

Why this is a subtle failure

• Sounds helpful and confident

• Many bike shops do service e-bikes

• But your fictional shop only:

  • does basic mechanical work on e-bikes

  • does not perform motor diagnostics or firmware updates

Example 2: Invented Policy (Hallucinated Detail)

User:
Do you charge for bike safety inspections?

LLM (almost right):
Safety inspections at Jo’s Bike Shop are free if you’re getting another service done; otherwise, there’s a small inspection fee.

Why is this subtle

• This is a very common real-world policy

• The answer hedges with “small fee”

• But the shop’s actual policy is:

  • free safety inspections, always

Example 3: Shop Hours Assumption Error

User:
What time do you close on Saturdays?

LLM (plausible but wrong):
Jo’s Bike Shop is usually open until 5pm on Saturdays.

Why is this subtle

• Uses “usually” to hedge

• 5 pm is a standard retail closing time

• But the actual hours are:

  • Saturday: 9am–3pm

Multi-turn (5 turns) Example: Subtle Failures

Scenario

A customer wants a tune-up and brake adjustment, but isn’t sure if they need to book ahead.

The chatbot starts off helpful… and then slowly drifts into misinformation about store policy and service availability.

Multi-Turn Conversation + Hidden Failure Commentary

Turn 1

User:
Hi, do you guys do regular bike tune-ups?

LLM (correct):
Absolutely! Jo’s Bike Shop offers a full standard tune-up, including gear indexing, brake alignment, wheel truing, safety checks, and general lubrication. I can help you figure out which service fits your bike.

Notes:
✅ Accurate. This matches the shop’s fictional service list.

Turn 2

User:
Nice. Mine also has squeaky brakes. Do I need a separate service for that?

LLM (still okay):
Squeaky brakes are usually handled as part of a standard tune-up, though sometimes we may recommend a brake pad replacement if they’re worn. It depends on the inspection, but you typically don’t need a separate booking for squeaks alone.

Notes:
✅ Still good. Everything here is plausible and consistent.

Turn 3

User:
Do I need to book ahead or can I just walk in?

LLM (first subtle issue):
For most tune-ups, walk-ins are totally fine. If the store is busy, you might need to leave the bike for a few hours, but generally you don’t need an appointment.

Hidden problem:
⛔ Your actual shop policy (say, from your shop documents or website) says tune-ups must be booked because mechanics only take walk-ins for minor fixes.
But the model has confidently stated the opposite.

This is something beginners won’t catch because it sounds correct.

Turn 4

User:
Cool. I’d like to drop my bike in tomorrow morning then. Is that usually a quiet time?

LLM (second subtle drift):
Mornings are usually quieter, yes. Most people come in after work. If you bring the bike in around 9 or 10am, someone can take a look right away.

Hidden problem:
⛔ Your fictional shop opens at 10am.
⛔ And mornings are actually your busiest intake period for our fictitiuos bike shop.

The model sounds confident (always does), but it has now invented:

• the wrong opening time

• the wrong traffic pattern

• incorrect service protocol

All plausible. But all wrong.

Turn 5

User:
Sweet. So I’ll be there tomorrow at 9:30am for a walk-in tune-up and brake inspection.

LLM (final failure):
Perfect! Just come by anytime between 9am–12pm and we’ll get you sorted. No need to book anything - we handle walk-ins right away.

Culmination of subtle failures:

• The model has fully committed to false store hours

• It continues to reinforce the incorrect walk-in policy

• It reassures the user that they’ll be seen immediately

Nothing is obviously broken. Every answer sounds reasonable. But the chatbot has led the user to a poor customer experience, entirely due to minor inaccuracies that compounded across turns.

Failure Analysis Primer

In this section, we’ll introduce the concept of failure analysis to identify what went wrong in the LLM application. Most developers would stop at vibe checks or gut feelings. However, intuition alone does not scale. The more you have - more conversations, different versions of models, and teams, intuition becomes subjective, so each person or entity develops their own. It’s important to understand that LLM systems require explicit failure analysis, not just gut feel and intuition.

A course I recently attended was 'AI Evals For Engineers & PMs', which first introduced me to the AMI Lifecycle for building Evaluation Systems. AMI stands for Analyse, Measure and Improve. 

Analysis involves collecting representative samples, such as traces, to enable failure mode categorisation. Once the failure modes are classified and categorised, Measure follows, generating quantifiable metrics, estimating the propensity for errors, and guiding the developer in prioritising fixes.

Finally, we transition directly to the Improve cycle, where we can begin enhancing the system by updating prompts, improving pipeline components, and refining retrieval and/or fine-tuning models.

Analysis → Gather Traces, Open Coding and Axial Coding.

Measure → build evaluators for our LLM application

Improve → update and improve our LLM application

The AMI life cycle essentially creates a feedback loop that allows us to constantly enhance our LLM application.

Step One. Collecting Conversation Traces

A complete trace consists of a full single- or multi-turn conversation that includes inputs, outputs, tool calls, and metadata (such as prompts). An important distinction is that it includes the very first user input, through to the final response.

It can originate from real user interactions or be synthetically generated for testing. Synthetic traces are especially useful when real user data isn't yet available, enabling testing without requiring actual users to have used the applications.

They help cover edge cases, replicate failures, and eliminate user variability from analysis. These traces can be linked to specific examples, whether single-turn or multi-turn conversations.

Step Two. Open Coding - Naming Failure Patterns

The process of systematically reading and labelling these collected traces is called open coding, a term adapted from Grounded Theory.

Grounded theory is a systematic qualitative research method in which you build a theory directly from data, rather than starting with a hypothesis. Instead of starting with a fixed list of imagined failure modes, we learn directly from the errors exhibited by the system and allow the failure mode patterns to emerge from the data. This enables one to focus on the actual problem at hand rather than the imagined problems we think the system has.

This represents a significant mindset shift for beginners.

Step Three: Axial Coding - Connecting the Dots

Axial coding is the second stage in Grounded Theory, where researchers systematically link initial codes into broader categories, exploring relationships around a central phenomenon (the “axis”) to build a theory. This is essentially a more formal term for categorisation or grouping, and completing this will enable the team to finally understand where the error distributions in the problem space occur. 

Introducing JO’s Bike Shop Chatbot

To illustrate the steps of the Analysis stage in the AMI feedback cycle, I have created a GitHub repo for JO’s Bike Shop Chatbot, an Apache Burr-based chatbot application that we can use for single-turn and multiple-turn evaluation exercises.

This is the best place to pause and regroup. In the next article, we will continue and demonstrate how we performed open coding and axial coding for single and multiple-turn conversations. We continue with the Analysis phase, where we start to build our AI application evaluations and incrementally improve our chatbot.

This article is part of a blog series about building effective evaluation systems for your AI application. Go beyond "vibe checks" and use a more systematic way to ensure your applications work as you designed them.

Part 1 (this article): Evaluations and your AI Application- This article introduces the concept of evaluations and explains how to approach this strategy for improving your applications.

Hi friends,

This article will kick-start a brand new series, and this time, we will explore the world of generative AI. With my background in software engineering, I want to explore how to build robust and predictable AI systems. Let's see what value evaluations bring to our applications.

Building AI Applications from First Principles

To many software developers, it is second nature to write unit tests, and your mileage may vary here. Some of us are so hung up on code coverage, but for me, pragmatism is key, where we build tests that add value and make sense.

Building AI applications is not significantly different from creating any other software application. While there is still value in writing unit and integration tests, developing applications based on these probabilistic models requires much more than that. Where we would have a definitive right or wrong outcome for traditional software projects, LLM-based applications necessitate a different approach.

In-Context Learning

The two main ways of developing LLM applications these days are fine-tuning AI models, which can help a lot in some specific instances; however, many in the know will first reach out to in-context learning and prompt engineering because they are much cheaper and just work.

We all know what prompt engineering involves, where we write instructions or context for the LLM to help it carry out its tasks. ICL, on the other hand, is a part of prompt engineering so that we are giving more detailed information, or overloading the prompt with other related data such as guidelines, background, examples and similar topics, such as what we do in Retrieval Augmented Generation (RAG) systems.

AI Applications and the Levers You Can Pull

Building AI applications is an iterative process, like building any software system. For example, when writing software, we typically build it in stages and layer functionality on top of features as we go. We also generally make sure that the functionality is sound first before optimising it.

With AI applications, which in this case are systems built on top of LLMs, in-context learning and prompt engineering, improving means we have to reach into and strengthen our ICL and prompt engineering techniques and the choice of language and embedding models it uses.

Vibe checking can only go so far.

Vibe checks are a method commonly used to build such systems. We want to answer the question, “Does the system’s responses feel right?“And yes, I use them all the time, and you should, too, as the primary way of ensuring your application works as you have designed it.

An obvious way to improve your application is to swap your LLM with a better one. These days, everyone knows that there’s a new and better model every week, and it feels like every other day. But even though the LLM developer claims that these are “better“ than previous models and many competitors’ models, this does not mean anything to you and your application until you test it in your application.

Yes, we reach out to vibe checks, and you should continue to do this, because people are the main users of your application. But when your requirements grow and your application becomes more complex, you will have to do more and more of these vibe checks.

For example, you may start with a handful of five checks, which you do every time your system changes. What do you do when you have to expand this to ten, twenty, fifty? How about a hundred items in your vibe checks checklist? Yes, this is not sustainable at all.

Evaluation systems are the answer.

This series of articles will discuss what evaluation systems are, why we need them, and, more importantly, how to approach building evaluations for your application.

Last month, I participated in a program to enhance my AI Engineering skills. With my background in software engineering, I encountered significant challenges when building LLM applications: specifically, the difficulty in creating robust and trustworthy systems. This is mainly due to the inherently uncertain and non-deterministic nature of the LLMs themselves.

The program, Building LLM Applications for Data Scientists and Software Engineers, led by Hugo Bowne-Anderson (one of my favorite online educators) and Stefan Krawczyk (creator of the open-source libraries Hamilton and Burr), proved to be incredibly valuable and focused.

See you in the following article!

Part 1. So how do you build a vector embedding model? - Introduces vector embedding models and the intuition behind the technologies we can use to build one ourselves.

Part 2. Let's build our image embedding model - Shows a couple of ways to build embedding models - first by using a pre-trained model, and next by fine-tuning a pre-trained model. We use PyTorch to build our feature extractor.

Part 3. Modelling with Metaflow and MLFlow - Here we are using Metaflow to build our model training workflow, where we introduce the concept of checkpointing, and MLFlow for experiment tracking.

Part 4. From Training to Deployment: A Simple Approach to Serving Embedding Models -Packaging your ML model in a Docker container opens it up to a multitude of model serving options.

Part 5. Putting Our Bird Embedding Model to Work: Introducing the Web Frontend - (this article) -For our embedding model to prove useful to others, we have created a modern frontend to serve the similarity inference to our users.

Hi friends,

In this final installment (hopefully!) of our project, we've reached the critical point where theory meets application. Only when our app users can directly interact with our embedding-model-powered similarity system will they fully appreciate its value.

To recap, in our first article, we introduced embedding models and then explored how to build them using PyTorch. We subsequently utilised tools like Metaflow and MLflow to streamline our MLOps process. Following that, we discussed how to containerise the model for efficient serving.

However, the true impact of these steps becomes evident only through visual, side-by-side comparisons. The challenge of replicating human-like visual comparison, identifying similarities and differences between bird images, is addressed through machine learning, as demonstrated in our previous articles.

Therefore, this article will shift its focus from technical details to a practical demonstration of the front-end interface, showcasing the embedding model's capabilities through bird image similarity searches.

The Bird Similarity Search System: An Overview

Before proceeding, let me visually summarise what we have accomplished. The lower left box illustrates the process we undertook to build the bird embedding model using PyTorch, Metaflow, and MLflow.

Following the creation of the embedding model, as shown in the lower right box, we utilised FastAPI and Docker to prepare our container for model serving. To enable image semantic search functionality, we selected LanceDB as our embedded vector database, facilitating rapid similarity searches.

Finally, the upper right box depicts the deployment-ready front-end, developed with ReactJS (Vite) and Docker, designed to showcase the system's capabilities to our users.

Search 1 - The Philippine Eagle

I was born in Cebu, an island in the middle of the Philippines, and even though I now call Australia home, I am still a proud Filipino. So the only bird that should go first is the Philippine Eagle.

Appearance: This bird has a very distinctive, almost comical appearance. It has a prominent, spiky crest of feathers on its head. Its face is pale with dark markings around the eyes. The beak is relatively short and thick. The body has a a mix of brown and white feathers.

The application displays the top 8 matches, it could have returned more, however, we wanted to display the results nice and even on the screen. After uploading the reference image, it takes a split-second for the results to show.

Notice that it was able to identify with 87.94% certatinty that the reference image is a Philippine Eagle, and the next most similar bird in the database is a low 38.73%. There will be a little bit of a delay if we deploy this to the cloud, however, this is much better in latancy and capability that any generic LLM out there. Just to show that Generative AI, as great as it is, is never the answer to all use cases out there.

For example, I showed this image to both Gemini and ChatGPT, with mixed results. While Gemini was able to identify the bird and suggest other similar birds, and although it also took less than a second, it was only able to give me text descriptions, I know its unfair to Gemini, as its a language model after all. But it’s multi-modal, so it should have been able to show me pictures too.

ChatGPT did not do too well either, though it did try to generate images of similar birds, it got stuck and produced an error generating the images after trying for a minute.

Search 2 - Parus Major

Appearance: This is a Great Tit (Parus major). It has a black cap, white cheeks, a yellow breast with a black stripe down the center, a greyish back, and a dark beak. It's a small passerine bird.

Despite our vector database containing only 50 bird images, it effectively demonstrates the similarity search process. By storing the bird images' vector embeddings in LanceDB, we can perform a similarity search by inputting a bird image.

Upon upload via FastAPI, the image is converted into a vector embedding, which is then submitted as a query to LanceDB. Due to the absence of a specified index, an exact search is performed exhaustively (brute-force) within a fraction of a second.

For databases containing millions of images, indexing is necessary to enable efficient nearest-neighbour searches, which scale to larger datasets but may result in a lower recall.

Search 3 - Java Sparrow

Appearance: This is a Java Sparrow (Lonchura oryzivora). It has a distinctive pink beak, a grey head, a pinkish cheek patch, a grey body, and a black tail. It's a small finch-like bird.

Observing both the Java Sparrow and the previously described Great Tit (Parus major), one might notice superficial similarities, such as the white cheek patches and grayish body coloration. However, these are where the similarities end.

It is noteworthy that the application accurately reflects this distinction. While it identifies both birds as among the most similar, the similarity scores clearly indicate a significant difference. The Java Sparrow is identified with a 96.98% similarity, whereas the Great Tit registers only 47.51%.

This effectively demonstrates that, despite some shared visual features, the application correctly recognises the Great Tit as distinctly different from the reference Java Sparrow.

Search 4 - Scarlet Tanager

Appearance: This is a Scarlet Tanager (Piranga olivacea). It has a bright red body, black wings and tail, and a thick, pale beak. It's a small songbird.

As with search #3, the application successfully identified and differentiated the Bornean Bristlehead as the most similar bird in the database. Beyond the shared red and black coloration, these two species exhibit significant differences.

The Scarlet Tanager is a small bird endemic to North and South America, while the Bornean Bristlehead is a medium-sized bird native to Borneo. Fortunately, neither species is endangered, and both are widespread within their respective habitats.

API Implementation for the Front End

As detailed in the previous article, we employed FastAPI to serve our similarity search system, which comprises the embedding model and LanceDB, our vector database. A notable feature of FastAPI is its built-in support for generating OpenAPI (formerly Swagger) documentation, as illustrated below. You have likely encountered this type of API documentation previously.

The OpenAPI interface provides an interactive environment for testing the API. Users can upload bird images directly through the documentation to experience the system's functionality firsthand. Please try it!

Local System Execution

Once the fine-tuned embedding model and the bird embeddings are ingested into LanceDB, executing the complete ReactJS front-end and FastAPI back-end locally is achieved through the following command:

docker compose up --build

This command initiates the build and execution of the Docker Compose network, deploying both the front-end and back-end services, thereby providing the complete system as detailed in this article.

Parting Words

Building this application has been a rewarding experience, initially intended as an educational endeavor for you, my readers and friends. However, as I've often found, these projects invariably benefit myself as well.

On numerous occasions, I've revisited my past articles, years after publication, and been surprised by the refreshed understanding I gain. It's a familiar feeling of learning and comprehension, distilled from words and images within a simple blog post.

I hope this project has sparked your interest in the possibilities of how computers can interpret images. It would be wonderful if you felt encouraged to explore these concepts further and perhaps even apply them in your own projects.

My aim was to illustrate the potential of machine learning for visual analysis and similarity search, and I hope this has motivated you to continue learning and experimenting in this exciting field.

Why don’t you give it a go!

You can find all the code for this project on GitHub. Feel free to check it out and try it for yourself – it's all open source!

If you have any questions, drop a comment here on Substack, or send me a DM on LinkedIn.

And hey, if you're interested in collaborating on something cool, let's chat! I'm always up for new ideas and working with other passionate people.

Part 1. So how do you build a vector embedding model? - Introduces vector embedding models and the intuition behind the technologies we can use to build one ourselves.

Part 2. Let's build our image embedding model - Shows a couple of ways to build embedding models - first by using a pre-trained model, and next by fine-tuning a pre-trained model. We use PyTorch to build our feature extractor.

Part 3. Modelling with Metaflow and MLFlow - Here we are using Metaflow to build our model training workflow, where we introduce the concept of checkpointing, and MLFlow for experiment tracking.

Part 4. From Training to Deployment: A Simple Approach to Serving Embedding Models - (this article) -Packaging your ML model in a Docker container opens it up to a multitude of model serving options.

Part 5. Putting Our Bird Embedding Model to Work: Introducing the Web Frontend -For our embedding model to prove useful to others, we have created a modern frontend to serve the similarity inference to our users.

Hi friends,

In our last few posts, we talked about embedding models and showed you how to fine-tune a pre-trained ResNet model using PyTorch. We also went over some MLOps ideas to set up a workflow for training this model consistently.

Now, we want to take things a step further and actually use this model in a way that's helpful for others, not just me. So, we're going to build a system around it. This means creating an API to let other programs talk to our model, and a nice, modern website for people to interact with it directly.

Getting Started: The API

FastAPI is a really popular tool for building APIs with Python. It's known for being super fast, which makes it a great choice for serving machine learning models. Plus, it has a bunch of other cool stuff built-in, like built-in validations, to make sure the data sent to the API is in the right format.

It's built on top of Pydantic, which is this incredibly popular library for checking data. Seriously, it gets downloaded like 272 million times every month! That tells you how much developers trust it and find it useful. Pydantic has tons of features, but the ones I find most handy are: making sure the data is valid, managing settings easily, and turning Python stuff into formats that can be sent over the internet (serialisation).

Making Things Easy with Docker

Docker... well, everyone knows Docker, right? For me, the biggest thing is that it makes life so much easier for me and my team. It gives us the best developer experience (DX) possible. With Docker, we can be pretty confident that if our API runs inside the container on our computers, it's going to run the same way when we put it in the cloud. No more "it works on my machine!" surprises. We also use Docker Compose to not just build the container for the API, but also to set up the whole system around it, like the website that will actually use the API.

The Power of Vector Databases

Finally, the real magic behind finding similar things in a project like this is the "semantic search" power of vector databases. In this project, we're using LanceDB, which is a vector database that keeps all its data in a file. It's open-source, has a lot of cool features, and the best part? It's super easy to get started with. Remember that feeling the first time you used SQLite? You get the same feeling using LanceDB.

My Thoughts & Where We're Headed

So, our ResNet 50 model sees those bird pictures and creates these "vector embeddings" – basically, a list of 2048 numbers that represent each image. Now that Generative AI is a big deal, these vector databases are making a comeback, and there are tons of them out there. But for what we're building, we need something solid for production: fast, easy to work with, and packed with features.

I actually first heard about LanceDB from this blog called The Data Quarry by Prashanth Rao. Specifically, it was in his article – "Embedded databases (3): LanceDB and the modular data stack." The idea of a vector database that felt like SQLite really clicked with me. Think about SQLite – it's fast, lightweight, has a lot of cool stuff, and you can use it for your local experiments, but more importantly, you can rely on it in real production systems. That's what got me excited about LanceDB.

What really surprised me was seeing how LanceDB stacked up against Elasticsearch as shown in the article. In a lot of ways, LanceDB actually performed better, and it did it using way fewer resources on a single computer. That's pretty impressive!

Using Docker containers is also a big part of our plan. It's like future-proofing our project for different ways we might want to deploy it, both while we're building it and when it's live. Because we're using Docker, we're not locked into one specific way of putting our API out there. We could use virtual machines, Kubernetes (which is super popular for managing containers), or even serverless options that can run containers, like AWS Lambda and Azure Functions. Plus, as we talked about earlier, Docker makes life so much easier for developers. We can be pretty sure that if it works in our Docker container, it'll work anywhere else.

Challenges & Unexpected Issues

Okay, so this project threw a little curveball my way. It's the first time I've really used conda to manage the Python stuff. I've always been a pyenv + python venv kind of guy. For me, that setup just feels lighter and I don't have to think about it too much anymore – it's just natural. I gotta say, conda and mamba (which is supposed to be a faster version of conda) feel a bit heavier, and it seems like there are a few more commands I need to keep in mind.

But hey, I'm willing to give it some more time. I've heard from a lot of people that conda is actually better when it comes to making sure all your project's dependencies play nicely together. So, I'll stick with it for a bit and see if it wins me over. And then there’s uv, but I will save that for another day.

Iterations & Lessons Learned

The whole point of starting this series was to get a better handle on how those vector embedding models actually work with images. With everyone talking about using vector databases for semantic search in those RAG (Retrieval-Augmented Generation) applications, I was curious if we could do the same thing with images – you know, compare them for similarity just like we do with text in NLP. It was pretty cool to find out that ResNet, which is actually a pretty old technology in the world of computer vision, can be trained to recognise objects, and that making an embedding model isn't as complicated as I first thought once you understand the basics.

Once I had a good grasp on building and training the embedding model, I moved on to creating an MLOps pipeline using Metaflow and keeping track of my experiments with MLflow. Before I really got into building models myself, I kind of thought that the model building part was the biggest chunk of creating a machine learning system. And don't get me wrong, it's definitely super important. But there are so many other things that happen around that modeling part – before you even start and after you're done – that when you look at the whole end-to-end system, the actual model building almost looks like a smaller piece of the puzzle.

After setting up the modeling pipeline, I now have a system that can handle all the steps – getting the data ready, training the model, and checking how good it is – in a way that I can repeat whenever I need to retrain in the future. That's pretty neat, but the thing is, nobody else can actually use this yet.

To make this image similarity thing available to everyone, we need an API. And that's where serving the model using FastAPI and Docker containers comes into play.

Plus, using Docker not only gives us more options for putting this thing in the cloud, but it also makes the whole development process way easier. It helps us developers focus on actually building the stuff without getting bogged down in environment issues.

In addition to more cloud deployment options, a superior developer experience is produced, which help the developer even more focus on the job at hand.

Final Results & Reflection

Whenever I build anything that uses Docker, I always make sure to use Docker Compose too. Honestly, this is mostly about making the development process smoother than anything else.

The kinds of systems I usually work on involve one or more Docker containers, and it's just so handy and reliable to have a complete setup that just works on my computer and in the cloud without having to change any code. It's a real time-saver and makes things way less frustrating.

For this project, I actually used Cursor, which is an AI-powered code editor, to help me put together the API and the ReactJS website (I'll be talking more about the website in my next post). I've used FastAPI before, and instead of the usual endless Googling and copy-pasting to get things going, Cursor really sped up the initial setup. I definitely had to guide it along the way quite a bit, but it definitely made my work easier. It's like having a coding buddy that knows a lot of the basics.

And LanceDB? It just works, plain and simple. If you ever need a solid, production-ready vector database that you can embed right into your application, I can now confidently say that LanceDB is the way to go. Even though it saves its data to your local hard drive by default, you can easily switch it to use cloud storage like AWS S3 or Azure Blob Storage if you need to. That makes it really flexible.

Building the API with FastAPI is fantastic, especially when you compare it to how you build APIs using something like AWS Lambda handlers (trust me, FastAPI is much nicer!). The built-in data validation, the easy way it handles data formats, and the fact that it can handle things asynchronously (doing multiple things at once) are all great features.

But honestly, you don't really see the magic of similarity search with vector databases until you have a user interface to play with. Seeing the search results with your own eyes is just... well, it feels like magic. You type in a description, or in our case, you'll upload a picture, and it instantly finds similar things. It's pretty cool.

Why don’t you give it a go!

You can find all the code for this project on GitHub. Feel free to check it out and try it for yourself – it's all open source!

If you have any questions, drop a comment here on Substack, or send me a DM on LinkedIn.

And hey, if you're interested in collaborating on something cool, let's chat! I'm always up for new ideas and working with other passionate people.

In today’s article, we will be talking about some software engineering best practices that will improve the developer experience as we build Generative AI applications. One of my annoyances in this space is the rapid advancements where we see new models in what feels like every week.

This means I regularly have to update my applications or scripts to accomodate this. I know that LlamaIndex and LangChain have a LLMFactory pattern that handles this scenario, however, I haven’t seen a library that does this for Instructor-based projects (which is my preference), until now.

I forked this from one of Dave Ebbelaar's repo to play with the code and understand how it works. It is awesome to have a repo filled with best practices and a well-documented codebase. I’ve been following his work for a few months now, and its still amazing to me how I continuously find diamonds in Youtube, the calibre of learning materials and generosity is top-notch.

Here are some learnings from this codebase:

• I have always used Jupyter notebooks to experiment with code, but this is the first time I have used interactive Python in VSCode, which I thought is a great experience. Although I will not be doing this all the time since I still enjoy the usefulness of Jupyter notebooks, I will be using interactive Python in VSCode more often.

• I like his use of the LLMFactory to make it easy to swap out different models, as I mentioned, frequent changes in this space, there’s always a new SOTA model around the corner. This is a great way to make the codebase more modular and easier to maintain. Also, its simple enough to extend and modify to adapt to your workflow.

• It is still using the instructor library, which I find indispensable when working with LLMs and structured outputs.

• Use Pydantic Settings as powerful way to manage and centralise your application’s settings, in this case, we are loading our application settings from an .env file, and we want to manage all from the one source file.

As I was running and playing with the code, there are a few issues that I thought would have been better, so here are the improvements I have added to this fork:

• I have improved LLMFactory so that the LLM registrations are done separately, meaning you will not need to update the LLMFactory class every time you add a new LLM. When a new model comes around, all we have to is register the new model by adding the register_llm_client decorator on the new model-specific code that returns the instructor client.

• I have added support for Amazon Bedrock to have access the Anthropic Claude models there. A bit of adjustments to Bedrock embedding models so that the calls are now identical to OpenAI or Ollama embedding models.

• I have created a similar factory class for embedding models (EmbeddingModelFactory), so that you can now mix and match different LLMs with different embedding models.

• This means I can now use say DeepSeek R1 running on Ollama locally and Titan embedding model from AWS.

• Or Anthropic Claude Sonnet 3.5 in Amazon Bedrock with Ollama local embedding model.

• Or both the LLM and embedding model from Amazon Bedrock. This is a great way to experiment with different models and not have to change a lot of code.

I have added a new decorator register_llm_client which is nothing but a lookup table for the registered LLM clients.

To support a new LLM, just decorate the function with register_llm_client, and that is all you need to do for the registration.

In the LLMFactory constructor, simply call get_llm_client, and this will return that provider’s client code!

If you want to play with LLMFactory, to check this out and play with this more, please have a look at the Dave’s repo at https://github.com/daveebbelaar/pgvectorscale-rag-solution or my fork at https://github.com/jaeyow/pgvectorscale-rag-solution.

Till then,

JO

Part 1. So how do you build a vector embedding model? - Introduces vector embedding models and the intuition behind the technologies we can use to build one ourselves.

Part 2. Let's build our image embedding model - Shows a couple of ways to build embedding models - first by using a pre-trained model, and next by fine-tuning a pre-trained model. We use PyTorch to build our feature extractor.

Part 3. Modelling with Metaflow and MLFlow - (this article) - Here we are using Metaflow to build our model training workflow, where we introduce the concept of checkpointing, and MLFlow for experiment tracking.

Part 4. From Training to Deployment: A Simple Approach to Serving Embedding Models -Packaging your ML model in a Docker container opens it up to a multitude of model serving options.

Part 5. Putting Our Bird Embedding Model to Work: Introducing the Web Frontend -For our embedding model to prove useful to others, we have created a modern frontend to serve the similarity inference to our users.

Hi friends,

In the previous article, we explored two approaches to creating an image embedding model, demonstrated through Jupyter notebooks: one for using a pre-trained model and another for fine-tuning that pre-trained model. These examples showed how to build an embedding model both by leveraging an existing pre-trained model and by fine-tuning it for our specific use case.

Image: We built an embedding model able to distinguish between bird species.

These are a good introduction, however this is still very far from having the ability to build machine learning models for real. Models that can be easily trained and prepared for online inference on the cloud, or in other words get them as close to production-ready as possible.

Introducing Metaflow

In this article, we’ll be introducing Metaflow, a fantastic machine learning orchestrator open-sourced in 2019 after being developed and battle-tested in Netflix to run their internal ML, AI and Data projects. I don’t want to have to repeat myself introducing Metaflow, their excellent documentation already does this better than I ever can.

According to their website: “Metaflow is a human-friendly Python library that makes it straightforward to develop, deploy, and operate various kinds of data-intensive applications, in particular those involving data science, ML, and AI.“

I have been using Metaflow for a while now, here’s a past project that used it as the ML orchestrator. This small project allowed me to understand first-hand the power of this tool.

Image: The stacks in Metaflow.

Metaflow super powers

The real strength of this tool lies in its ability to streamline the transition from early prototyping to production. You can start by developing your solution locally on a laptop, ensuring an intuitive and straightforward experience. Then, with minimal code changes, you can scale your code to leverage your preferred cloud platform—whether AWS, Google Cloud, or Azure—for compute and storage.

Image: Metaflow enables a simpler process from developing locally to full production system.

Here’s the Metaflow pipeline I created for training our bird species embedding model (Github link).

This is truly ingenious because it empowers data scientists to productionise their models independently. Traditionally, this responsibility often fell to Machine Learning Engineers (MLEs), who had to port and adapt the models for production. While effective, this approach introduces an additional layer of complexity, which can disrupt the process.

Data scientists, who understand their models best, should be the ones bringing their creations to life. This tool eliminates that barrier, enabling them to do so with ease. Ultimately, it allows you to achieve more with fewer resources.

Since the day it’s been open sourced, more and more materials have been made available for us, such as these great tutorial resources. They have made it even easier by providing Metaflow in a sandbox environment, so that you don’t even have to set it up in your cloud account.

Using Metaflow to build our model

So to continue the the article and the bird classification use case, I’ve built an MLOps pipeline that is almost ready for production. I say almost, because I’m not really deploying this to any cloud environment, although there will be a minimal effort to do so. Here is the python code for this flow.

With Python, Metaflow and MLFlow, we have built an ML System that:

• have the ability to build Python-based Directed Acyclic Graphs (DAGs)

• have version control as a native feature

• allow you to easily scale up your pipeline from prototype to production

• enables you to have an ML system that is library agnostic, so that you can bring in your favourite ML library, like scikit learn, PyTorch or TensorFlow.

• allows you to easily debug your pipelines, as well as create a more fault-tolerant system

• With MLFlow, we have also added the ability to log and track experiments, to ensure a system that is dependable and reproducible. This includes a model registry functionality that makes tracking your models with ease.

Image: MLFlow metrics page showing this training run’s metrics

Model Checkpoints

One of the key features I’ve added to this workflow is support for model checkpoints. Before diving deeper, let’s clarify what checkpoints are. As described in an article on Towards Data Science, a checkpoint is a snapshot of a model’s entire internal state at a given point during training. At the end of training, this state is saved into a binary file, which can later be loaded to enable predictions.

However, best practices suggest going beyond saving the model only at the end of training. It’s far more effective to periodically save checkpoints at critical stages of the training cycle.

Image: Periodic checkpointing (image by Lak Lakshmanan)

Most people think of checkpointing simply as a way to save a trained model for later inference, and while this is important, the benefits of checkpointing extend much further. One of its most significant advantages is enabling a fault-tolerant training system—a feature I cannot stress enough.

Imagine you’re running a 10-hour training cycle, and disaster strikes—be it hardware failure, a software crash, or some other catastrophic issue—9 hours in. Without checkpoints, you’d need to start training from scratch, wasting time and resources on another 10-hour cycle. However, with periodic checkpoints, you can simply resume training from the most recent saved state. For instance, if you save your model every hour, you’d only need to retrain from hour 9, saving precious time and effort.

That said, it’s important to strike a balance in how frequently you save checkpoints. Saving too often can waste storage and slow down training, while saving too infrequently may not provide enough recovery points to be useful. Thoughtful checkpointing ensures you maximise efficiency and minimise risk.

Conclusion

Here’s the Metaflow pipeline I created for training our bird species embedding model (Github link).

As discussed in previous articles, we approached this by initially treating the task as a classification problem. If you examine the code, you'll notice it follows a standard PyTorch model training process. However, the key step is removing the classification head before saving the final checkpoint. This transforms the model from a classifier into an embedding model capable of generating a 2048-dimensional embedding vector for each bird image.

We ported our model training notebooks into a production-ready workflow using Metaflow DAGs. Metaflow already works for local development, however for MLFlow, there was a little bit of configuration to make it work in a local Docker setup.

This wraps up our process for building embedding models. It was an excellent opportunity to improve our understanding of deep learning for image processing. While ResNet and similar architectures are considered "old tech" (dating back to 2015), they remain foundational and widely used in industries like manufacturing and medical diagnostics.

Till then,

JO

Part 1. So how do you build a vector embedding model? - Introduces vector embedding models and the intuition behind the technologies we can use to build one ourselves.

Part 2. Let's build our image embedding model - (this article) - Shows a couple of ways to build embedding models - first by using a pre-trained model, and next by fine-tuning a pre-trained model. We use PyTorch to build our feature extractor.

Part 3. Modelling with Metaflow and MLFlow - Here we are using Metaflow to build our model training workflow, where we introduce the concept of checkpointing, and MLFlow for experiment tracking.

Part 4. From Training to Deployment: A Simple Approach to Serving Embedding Models -Packaging your ML model in a Docker container opens it up to a multitude of model serving options.

Part 5. Putting Our Bird Embedding Model to Work: Introducing the Web Frontend -For our embedding model to prove useful to others, we have created a modern frontend to serve the similarity inference to our users.

Hi friends,

In the last article, we’ve been tasked by Bird Watch to build their bird species similarity search engine. We have a dataset of 525 bird species downloaded from Kaggle. I swear I got it originally from Kaggle, however when preparing this article, I couldn’t find it there anymore. I did find a copy in Hugging Face, so here it is.

A similarity search engine operates by extracting embedding vectors from images and calculating the distances between them. The magnitude of these distances indicates the degree of similarity between the images. To create an effective embedding model, it's essential to select a suitable image model and decide whether to use the pre-trained model as-is or to fine-tune it for your specific application. Using either will leverage the model’s feature extraction capabilities.

Let’s begin.

Option 1: Use a Pre-trained model

The breakthrough performance of AlexNet in 2012 opened the floodgates for deep learning in computer vision, leading to the discovery and development of numerous models over the years. These models not only improved accuracy but also became increasingly efficient. Notable examples include VGGNet (2014), ResNet (2015), MobileNet (2017), and Vision Transformers (ViT) (2020), all of which have pushed the boundaries of what's possible in image processing with deep learning.

For the purposes of this example, we will focus on a model from the ResNet family—specifically, ResNet-50. In my experience, ResNet-50 strikes an ideal balance between accuracy and computational efficiency, making it well-suited for this particular use case.

ResNet General Architectural Design

Follow with the Jupyter Notebook

Follow the scripts in a Jupyter notebook here.

Using a pre-trained model as an embedding model is the simplest and most efficient option. For instance, in our case, ResNet-50 has been pre-trained on the ImageNet dataset and includes a classifier head with 1000 classes. To create an embedding model, you only need to remove the final classification layer, transforming it into an embedding model that's ready to use for our bird species similarity search engine.

Similarity Search with pre-trained ResNet-50

In the following images, we used the pre-trained model as an embedding model. While it works to some extent, its performance isn't optimal. For example, it performed well for the Philippine Eagle and Parus Major but struggled with the Scarlet Tanager. Additionally, if you examine their similarity scores, you'll notice that the bird species are clustered too closely together. This is expected, as the ResNet-50 model was pre-trained on a 1,000-class dataset, including many categories beyond birds, which affects its ability to distinguish between different bird species in fine detail.

Issues with the pre-trained model

Using the ResNet-50 pre-trained embedding model presents a few key issues:

1. Similarity scores are too close together: The model fails to adequately distinguish between bird species in the dataset, with all similarity scores falling within the 0.8xx range. This lack of variation suggests the model isn't capturing enough nuanced differences between species.

2. Sensitivity to image variations: The model is highly sensitive to changes in image orientation, rotation and lighting. For instance, showing it different variations of the same bird species—such as flipped or rotated images or different lighting conditions—leads to inconsistent and poor results, indicating a lack of robustness in handling such variations.

Option 2: Fine-tune a pre-trained model

Another approach is to fine-tune the ResNet-50 model. This means starting with the pre-trained ResNet-50, leveraging the weights it learned during its initial training. Fine-tuning offers a significant advantage over training from scratch, which would require hundreds of hours and millions of images—resources we don’t have, nor need. Instead, fine-tuning allows us to quickly adapt the model for our specific use case, essentially taking a shortcut to build an effective embedding model.

Fine-tuning an embedding model follows a process similar to training a classifier. In our case, since we are working with a smaller dataset of 50 classes, we start the process as if we are training a 50-class classifier. After training, we simply remove the final classifier layer, converting the model into an embedding model ready to handle our similarity search use case.

Follow with the Jupyter Notebook

Here is the Jupyter Notebook for fine-tuning a pre-trained ResNet-50 model.

In the images below, we used the same set as in the pre-trained experiment, but this time we generated embeddings using the fine-tuned ResNet-50 model. The results show a drastic improvement, both in accuracy and the returned similarity scores.

Similarity Search with fine-tuned ResNet-50

All unknown birds were matched correctly as the top result. Moreover, the range of similarity scores is much more promising—where the top match has a similarity score close to 1, and the remaining results have scores closer to zero. This demonstrates that the fine-tuned model captures bird species embeddings with far greater precision.

For example, the unknown birds—Philippine Eagle, Scarlet Tanager, and Parus Major—all returned the correct match as the top result! It's clear we've built a highly effective bird species embedding model, and impressively, the model is just under 100MB in size.

Conclusion

While the Bird Watch project is hypothetical, the techniques we've explored in using embedding models—both pre-trained and fine-tuned—are very much applicable in the real world. The ResNet-50 model we utilized was introduced by Microsoft in 2015, so it may not be considered cutting-edge technology. However, it's important to not underestimate the power of these machine learning models. With its ideal balance of accuracy and latency, these models continue to see widespread application in the industry today.

Although we are applying this technology in a conceptual project like our bird species similarity search application, the same methods continue to be used in critical areas such as automated visual inspection and quality control in manufacturing, as well as in medical imaging for diagnosis and cancer detection.

Till then,

JO

Part 1. So how do you build a vector embedding model? - (this article) - Introduces vector embedding models and the intuition behind the technologies we can use to build one ourselves.

Part 2. Let's build our image embedding model - Shows a couple of ways to build embedding models - first by using a pre-trained model, and next by fine-tuning a pre-trained model. We use PyTorch to build our feature extractor.

Part 3. Modelling with Metaflow and MLFlow - Here we are using Metaflow to build our model training workflow, where we introduce the concept of checkpointing, and MLFlow for experiment tracking.

Part 4. From Training to Deployment: A Simple Approach to Serving Embedding Models -Packaging your ML model in a Docker container opens it up to a multitude of model serving options.

Part 5. Putting Our Bird Embedding Model to Work: Introducing the Web Frontend -For our embedding model to prove useful to others, we have created a modern frontend to serve the similarity inference to our users.

Hi friends,

It’s been an eventful few months at work since my last post, where I wrote about the time when I spent the weekend playing with SOMs. When I’m less busy work-wise, I do tend to work on interesting end-to-end projects that have something to do with machine learning. I did tell myself that for 2024, Gen AI will be my focus, and it still is, however, I will still have time for ML projects such as this.

One problem that stumped me recently was an image classification problem for 5,000+ classes. Yeah, I’ve done multiple classification problems before but not with more than a dozen classes. I also have not done a lot of computer vision ML problems so this is my chance to dig a little bit deeper and familiarise myself with the process.

A Hypothetical Problem

I am partnering with Bird Watch, a bird conservation and wildlife monitoring non-profit organization. They need a machine learning model that will help them identify bird species that had been captured by their drones and camera traps. Instead of directly classifying each bird species, I was asked to return the 10 most similar bird species, returned in order according to the most similar where naturally the most similar specie will be at the top of that list. So this is good, instead of forcing the output into predefined categories, it can actually find matches when the species presented is not part of the dataset, and can be a helpful in identifying new species.

• multiple class classification of this magnitude is not recommended as it will be a large model and will be expensive to train

• after reflecting on the problem, where we need to return the ten most similar bird species. This was not really a classification problem after all

• input is an image, and the model will need to return the 10 most similar images from the images in a database.

• so instead of an ML problem, we can actually treat this as a data retrieval problem, with the help of embedding vectors and a vector database

• So if it’s similarity search, as what we are doing here, what technology enables it? Yes, semantic search and vector databases.

• And when talking about vector databases, the thing that underpins it all is vector embeddings generated by an embedding model.

So, I don’t need a classification model?

A typical bird classification model will return with a species prediction, one out of the 525 bird species in the dataset. But what do I do if I don’t want to predict the species? What machine learning model do I need when I want to return the 10 most similar species? We need a species similarity search, and not a species classifier!

We need to build a vector embedding model!

Although vector embeddings have been around since the 1980s and 1990s, I only recently became aware of them, thanks to their rise in popularity with large language models like ChatGPT towards the end of 2022.

How to build an embedding model

To know how to produce an embedding model, we first need to understand the main parts of a Convolutional Neural Network (CNN) model. The CNN model has three main parts:

• Backbone - This is the part of the model that extracts the features from the image. This is typically the convolutional layers of the model.

• Neck - This is the part of the model that reduces the dimensionality of the features extracted by the backbone. This is typically the pooling layers of the model. This is typically the penultimate layer of the model, that outputs the features.

• Head - This is the part of the model that classifies the features extracted by the backbone and refined by the neck.

This image is true for many of the image models hosted in Hugging Face. To produce an embedding model, we need to remove the head and use the neck's output which I later realised is actually are THE vector embeddings! These embeddings are the features extracted by the backbone and refined by the neck.

It is as easy as that. Just remove the classification head, regardless if you are using a pre-trained model or a fine-tuned model, and you have just produced an embedding model.

That is exactly what we will do, perform a ‘surgery’ on the fine-tuned model that we will build for this article!

Pre-trained image model

There are a multitude of options of pre-trained image models available online - Tensorflow and PyTorch have their model hubs, and cloud providers like AWS, Google and Azure als have their own AI galleries. For this article however, we will be using timm (PyTorch Image Module), which is a PyTorch library that contains a collection (over 1,400) of models, and miscellaneous code that can help you build, train or fine tune your image models. When you use this library, it actually downloads the model weights from Hugging Face, another well known model hub that initially became popular through its hosting of NLP models. These days, you will find there almost any type of open source machine learning model to fine-tune or to use outright.

Let’s check out ResNet50

ResNet 50, short for Residual Network architecture, a deep learning machine learning model that is popular for image classification tasks. It was introduced by Microsoft Research in 2015, and it became the basis of many deep learning architectures after that. It was first introduced in this paper by He. et. al.

Feature extractor

ResNet 50 is part of the ResNet family of image models, and has been proven for its performance as pre-trained models, in a process called transfer learning - where it learns weights from large datasets, like ImageNet, and uses that to improve the accuracy in much smaller datasets, like our bird species dataset.

For our embedding model, although we will start by training a ResNet 50 classifier, once we have built our fine-tuned model, we will be discarding the classification head, leaving just the embedding model, in other words, we’re left with the feature extractor, and that’s precisely what we need.

And how about our similarity search?

Once a vector embedding model has been built, we’re halfway to our bird species similarity search engine. We can then build it in the next 3 steps:

• ingest our bird species dataset into a vector database, using the fine-tuned embedding model to generate the embedding of each

• using the same embedding model, get the vector embedding of the new bird specie image

• finally, perform a similarity search of this new bird specie image against all the bird species ingested in the vector database. There are several options of similarity criteria, but the most common are cosine similarity and euclidian distance. These similarity metrics are typical of the popular vector databases these days.

Conclusion

In this article, we have learned that the image models like ResNet 50 are used to train and fine-tune classification models. But to produce an embedding model, we can simply remove the model’s classification head, leaving behind the feature extractor and a model that produces embedding vectors - our fine-tuned embedding model.

In the next article, we’ll see how to actually build this embedding model, through a Jupyter notebook that we’ll build for this series.

Till then,

JO

A couple of weeks ago, I stumbled upon an unsupervised learning algorithm called Kohonen Self-Organising Maps. Koho-wwhattt?!!

Github project for this article

Self-organising maps are nothing novel, in fact it was first introduced by a Finnish professor Teuvo Kohonen in the 1980s. It is a type of artificial neural network, but instead of the usual backpropagation (gradient descent) used in neural networks, Kohonen SOMs use competitive learning instead. It was quite popular around 2005, however, I seldom (never) hear them being used these days. There are SOM libraries in popular languages like Python, R, and Matlab to name a few.

What attracted me to SOMs was it’s ability to create pretty abstract blobs of colour which are almost like art, if you ask me. (This colorful representation is possible if you use a 3-component vector like the RGB-component we use in this article.) Above is one such example of an output generated by our Python implementation.

What are SOMs?

Kohonen Self-Organising Maps (SOM) are a type of unsupervised learning algorithm. Kohonen Maps are typically used for clustering and visualising so that higher-dimensional data can be represented in lower dimensions, typically in 2D like in a rectangular topology or grid.

In addition to segmentation and clustering analysis, it is also a form of dimensionality reduction technique so that the high-dimensional data in the input layer can be represented in the output grid.

SOM Training

I have created a notebook, a Python implementation and a FastAPI client that I have made available in GitHub to use any which way you want. This Jupyter notebook goes through the training of self-organising maps as shown below:

1. Each node's weights are initialized.

2. We enumerate through the training data for kohonen_some number of iterations (repeating if necessary). The current value we are training against will be referred to as the current input vector

3. Every node is examined to calculate which one's weights are most like the input vector. The winning node is commonly known as the Best Matching Unit (BMU).

4. The radius of the neighbourhood of the BMU is now calculated. This is a value that starts large, typically set to the 'radius' of the lattice, but diminishes each time-step. Any nodes found within this radius are deemed to be inside the BMU's neighbourhood.

5. Each neighbouring node's (the nodes found in step 4) weights are adjusted to make them more like the input vector. The closer a node is to the BMU, the more its weights get altered.

6. Go to step 2 until we've completed N iterations.

Kohonen Input and Output Layer

A Kohonen SOM has two layers, the input layer and the output layer. The input layer is made up of the features of the data, while the output layer is made up of the nodes that will be trained to represent the input data. In this project, our input layer is made up of floats (which represent RGB colours), and the output layer is a 2D grid of nodes, each node also made up of RGB components.

The image below shows a very simple SOM with an input layer of 3 features and an output layer of 16 nodes. In this project, we only have 3 features (RGB colour components), however, you can have more features in the input layer. For example, if you are trying to segment customers, the features can be income, sex, age, race, etc.

With our input features (and output node weights) conveniently RGB components, we can easily visualise the output layer as a 2D grid of colours.

If you look at each node in the output layer, you will notice that there are 3 lines into each node. These lines represent the weight of each feature in the input layer. Each output node weight is also a 3-dimension vector, the same shape as the input layer, and also represent RGB dimension. When updating the node weights during training, the changes to the weight are easily visualised as colours.

SOM Output

Given the input layer made up of 20 random RGB colours, later you will notice that the output image will contain colour groups from this layer.

In the below left image, a 100x100 output layer is initialised with random colours. Running the SOM algorithm for 1000 iterations will produce the output on the right, where we can see roughly 20 colours grouped together.

Vectorisation

In my first attempt at implementing the Kohonen SOM algorithm, I used the typical Python nested loops following the algorithm to the letter. However, I quickly realised that increasing the iterations to 200, 500, 1000 or more would slow it to a crawl, not very exciting when deploying it to production.

The algorithm could be vectorised using Numpy, which would make it more efficient, and faster. I have implemented both versions and compared the execution times.

The vectorised implementation is around 76x faster (from around 10 minutes down to 9 seconds), so it’s worth remembering when working on your Python implementations that involve numerous loops.

Production deployment

In the spirit of full-stack machine learning, we should not disregard production deployment of the resulting API. This project uses FastAPI to implement the web API to serve our ML model.

For this project, my deployment preference is serverless first, so I would use AWS Lambda Container Images to deploy it. As with many APIs for data and machine learning applications, the required dependencies will surely be over the 250MB uncompressed limit even when using multiple Lambda Layers. All the popular Python libraries are quite chunky and will easily go over this Lambda hard limit. Numpy alone is already over 100MB, and we still have FastAPI, Uvicorn, and Mangum to add.

With Lambda Container Images, this will allow us up to 10GB container size limit for our API. I have supplied a Docker image for Lambda containers, as well as a standard image for any cloud provider that can deploy containers.

However, for applications that have serious production requirements, a more comprehensive ML serving solution such as AWS SageMaker, BentoML, or TensorFlow Serving may be more appropriate.

Till then,

JO

Handy Resources

Github project for this article

Kohonen Self Organising Map (SOM)

Self-Organising Maps

Kohonen Self-Organising Maps - A special type of Artificial Neural Network

Wikipedia - Self-organising map

Choosing the right project to work on can be as tricky as finding your way through a maze of endless options.

New AI announcements (especially Gen AI) seem to drop like clockwork these days, and the unlimited project choices can be overwhelming and paralyzing. If you know me, you'll know I'm a big fan of Formula 1. So, it wasn't hard when it came to picking my next machine learning project. I needed something that would be exciting to myself, a project I could pour endless hours into without blinking an eye.

Friends, we're about to dive into the exciting world of Formula 1 race prediction!

This isn't my first rodeo with race prediction: I dipped my toes into it during a Data Science boot camp I completed a while back. That experience taught me the ropes of ML model development, however, this time, it will be a full-on ML-based product. Buckle up as we delve into applied data science!

To start off, here are our requirements and constraints:

• serves race predictions using a production-ready (infinitely scaleable) architecture

• web application deployed on a modern stack

• historical data comes from an API, and miscellaneous race information will be scraped from online source/s

• using the FTI Pipeline Architecture as described here, we will have a feature pipeline that will be run on a specified schedule, to update racing data when they become available.

• data validation and integrity checks are part of our feature pipeline.

• a training pipeline that will be run when new features become available. As this series is not intended as a tutorial in model development, we will be using a good model (deep learning or an ML model, well, I have not made up my mind yet), we will discuss model selection criteria as we progress further.

• use a modern platform to track our training, model tuning, and models created by this pipeline

• a batch inference pipeline, that runs when a new model is ready and populates a database (or simply a file in storage) with pre-calculated predictions

• a race prediction API that returns these pre-calculated prediction results to our user interface

• our system will also have model prediction performance monitoring built-in

• use exclusively free* and serverless infrastructure.
  *note: A generous free tier is more than enough for our use case.

  ---

Tech Stack

Looking at the tech stack dump below, it seems a lot, and it is, however, this is what you would typically find with the industry or with startups building MVPs. When you are building a product, you’ll find yourself cobbling different applications together, as there’s no ONE application that can do it all.

Don’t worry, we will go through each one as we use them in the project. Here are some descriptions of each tool:

• Python - the bulk of this project will be written in Python, as it is easy to learn and maintain, and it’s such a rich ecosystem that it is the de-facto standard in machine learning.

• Metaflow - all our Python workflows will be orchestrated by Metaflow, an open-source ML tool built and battle-tested in Netflix. I’ve been using it for a while now, and just love how I can easily transition from my laptop to the cloud with very minimal changes.

• NextJS on Vercel - ReactJS is great, but NextJS is even better. It is a superset of React and is designed for production as it is super fast. I wrote a blog article about it a while back, and it was already awesome then.

• Beautiful Soup - Most of the dataset for the project comes from Ergast.com, however, we are on borrowed time as it has been deprecated, and will be available only up to the end of 2024. For the latest race results, we will be scraping it straight from the Formula 1 website.

• Hopsworks - I have mentioned that we will be using the FTI pipeline architecture, the mental model from the Hopswroks team. This architecture revolves around the use of feature stores, and we will use them here.

• DuckDB - we will be using DuckDB instead of Pandas to work with our data in SQL more easily. I will always take the chance to not have to use Pandas if I can help it.

• AWS Lambda - since our model serving is quite simple, where we’re only really serving online predictions from our cached and pre-calculated results, we don’t need to go through the complexities of using something like Seldon, for leveraging Kubernetes clusters.

• Comet - for experiment tracking and model registry, we can use trusty Comet, with its easy-to-use UI and Python API.

• Great Expectations - to enforce data validations and ensure data quality as the data is ingested into our pipeline, we use GX.

• Evidently AI - monitoring the models in production is made possible by Evidently AI, and it can be used to detect drift and help us decide whether to retrain our models or not.

  I’d like to think that our budget for this project - is anywhere between free and cheap! 🤣
  
  - JO

  ---

Architecture Diagram

We now have the architecture diagram above, which we will use as a guide as we build this project. We might add changes here and there, but the main takeaway here is that instead of a single ‘end-to-end‘ pipeline, a production-ready machine learning system is a modular, easily maintainable, and scalable system. I’ve marked the ML tooling we plan to use to give a better idea of our end goal.

I think that should be about it for now. In the next edition, we will start building our workflows.

Till then,

JO

Software development is a complex process, where development teams have to juggle creativity, customer requirements, technology, and many delivery-related issues. Furthermore, in the last few years, more and more products have incorporated machine learning, making what was already a complex undertaking even more challenging.

In one of my first articles on machine learning, I used the following image to describe a machine learning product (copyright Google), showing all its components, and how model development plays a very minor role. How can one build a product given the image below?

In the last couple of years, more and more tools have been added to the MLOps ecosystem. Why is it then that a recent Gartner survey revealed that on average only 54% of AI projects make it from pilot to production? Just one per cent up from the previous year, with all this tooling at our disposal, surely we’re missing something. Yes, many of our models just don’t make it to production.

One would think that we’d have this figured out by now. Couldn’t be anymore further from the truth. As described here and here, it seems that more tool choices do not translate to increased success in productionizing ML systems.

So what are we to do then?

Getting over software complexity

Coming from a software engineering background, I'm no stranger to the complexities of software development. To address this, the industry employs various strategies, and one notable approach is the use of architectural frameworks. For instance, the Model-View-Controller (MVC) framework gained popularity in the .NET ecosystem around 2010, as it facilitated web developers in creating web applications with a modular and shared understanding of web application development. (As an aside, it’s not something you would want to use anymore these days as there have been many advancements since then)

Consequently, it garnered widespread adoption within the industry because it made web application development more approachable and manageable. With a common understanding of the system, everyone involved had a better grasp of the process. This, in turn, led to an abundance of developers well-versed in the framework, making it easier and more cost-effective to find collaborators.

How about machine learning?

As I mentioned earlier, the arrival of all these ML and MLOps tools had minimal impact on the ability to productionize ML systems. Well, I dare think that it might have even contributed to decision fatigue, as being confronted by all these choices made the decision even more difficult to make.

Can we achieve the same outcome in machine learning as our developer cousins?

I believe so. The folks at Hopsworks developed the FTI architecture for machine learning systems. Instead of the usual single “end-to-end pipeline“, where the mental model has everything including the kitchen sink, they have recommended three separate pipelines - the Feature pipeline, the Training pipeline and the Inference pipeline.

Although Uber Engineering pioneered using multiple pipelines to manage the scalability and complexity in machine learning systems, it looks like Hopsworks first came up with FTI architecture.

But how do multiple pipelines help?

At first glance, having multiple pipelines is counter-intuitive. Wouldn’t a single pipeline be simpler for our system, and easier to build and maintain? Well, yes it is a simpler system no doubt, however, typical ML systems deal with dynamic data, and we should have a system that supports that from the start.

A Feature pipeline regularly creates ‘features‘ from incoming data and saves it to a shared storage called a feature store. Typical systems have regular (if not streaming) data ingestion pipelines, so there is a need to operate the feature pipeline regularly, on-demand for streaming systems, or on a schedule for batch systems.

A Training pipeline reads training data from the feature store, trains the model and saves the trained model in a model registry. And because this pipeline is stand-alone, it can be 1) developed by someone else, and 2) invoked by a different schedule from the other pipelines.

And finally, the Inference pipeline is responsible for facilitating the prediction service. Call it on a schedule for batch inference systems, or if used for online inference, use any of the model serving infrastructure made available by many varied service providers.

Benefits of the FTI architecture

• simple mental model of realistic machine learning systems

• enables the development of maintainable ML systems in a relatively short period

• reduced cognitive load when thinking about the whole ML system

• modular pipelines provide clear boundaries and interfaces and promote collaboration between the multiple personas in an ML product team

• open architecture - enables it to be cloud and tool-agnostic to avoid lock-in

Where do we go from here

In the next few editions, let’s develop a machine learning system from the ground up guided by this architecture. It will be fun, trust me.

Till then,

JO

I’m JO Reyes.

Originally from Cebu Philippines, I now call Sydney Australia home.

I’m an experienced software engineer and have been skilling up on machine learning systems for the past couple of years. I’m also midway through my Masters of Data Science at UNSW which is going well.

I love building things, and writing about it. I have a blog at Full Stack Developer Tips which I still regularly update, however Full Stack ML is for all my machine learning content.

With Full Stack ML, I hope to connect with many of you ML and AI folks, and learn from each other. I plan to cover topics related to building machine learning systems, specially as it makes its way into production.

Anyhow, let’s keep this short today, as this is just a welcome message! The real newsletter will start next week.

Thanks for stopping by, and see you later!

Regards,

JO