Tools like Copilot, Cursor, and numerous other “agents” like Claude Code & co. can generate boilerplate in seconds, spin up a REST API, scaffold with a single prompt, or even summarise a 5,000-line pull request. I have personally used this to port logic from a language I don’t want to deal with — no hate to C++, but sometimes, while working with embedded devices, micropython is such a time saver. Regardless, with the power of LLMs, I can condense various miscellaneous work into minutes, which would require hours of my focus time otherwise.

I have seen the evolution of these tools over the last 2 years, and I can confidently say that they are here to stay. They are not perfect, but they are getting better every day. I have been daily-driving Copilot and Claude Code since they were released and I can claim that I’m somewhat of a power-user myself. Some day I’ll write a blog post about my experience with these tools, but for now, I want to focus on a different aspect of this evolution.

The effectiveness of AI coding assistance depends not only on the model’s intelligence, but also on the quality of the codebase it interacts with.

Pretty straightforward, right? But let me explain why I think this is a big deal, and also try to address some of the FUD going around these tools on Twitter every day.

The Rise of Coding Automation

Early “AI pair programmers” were mostly autocomplete on steroids. They could spit out snippets, but context was limited. Today’s LLM-based tools ingest entire projects, understand inter-dependencies, and adapt to your coding patterns. This means I can completely offload repetitive work like DTO generation, CRUD endpoints, and unit test scaffolding to the machines. I spend less time grinding through boilerplate and more time focusing on domain logic and architecture.

But this doesn’t mean developers are “done for.” It means our jobs have shifted from writing every line to more designing, curating, and maintaining code that humans and machines can reason about.

Cutting Through the Noise

“AI will replace all developers.”

Just describe features in English and boom—software ships itself.

This is fantasy. Software is more than code; it’s architecture, systems thinking, tradeoffs, and a touch of empathy for the future users.

As someone writing software for a living, I wish it were that easy. I would love to run a bunch of Claudes and mint money selling software to the world.

I write code that makes LLMs go brrr, so trust me when I say that they are utterly useless for some of the things I do. I routinely turn-off copilot and force-kill Claude for about 20-30% of my work, when it starts to generate garbage on parts of code that work on novel parts — for example, OpenAI’s responses API. It was just released a night back and Claude had no clue. It generated wonderfully consistent garbage, but garbage nonetheless. It just hasn’t seen the API yet, and no amount of prompts and web searches could fix that.

“LLMs are useless.”

Because they sometimes hallucinate or generate brittle code, critics just dismiss them outright.

That ignores the productivity gains already proven in industry settings. I have bootstrapped complete features while on in-flight WiFi with Claude. The truth is in between. LLMs are tools, not magic wands. Neither they make you Hermione to open door locks with “Alohomora”. Their usefulness scales with the structure and readability of the code they touch.

The Sweet Spot for Humans and LLMs

Uncle Bob didn’t write “Clean Code” with LLMs in mind, but it might as well have been. (Let’s ignore politics and Clean Code dogmatists for now. It’s a nice book, read it if you haven’t)

Consider a few principles:

• Small, single-responsibility functions make it easier for humans to test, and easier for models to “understand” scope.

• Use meaningful variable and class names that act as semantic breadcrumbs for LLM reasoning.

• Have a consistent structure. Predictable patterns make it simpler for models to complete code correctly.

Now flip the scenario, Get a file from a poorly written legacy monster with 30,000 LOC files, cryptic variable names, and spaghetti logic. Working on such code is brutal for humans and impossible for even the smartest LLM to navigate effectively within context windows.

Good code isn’t just a human courtesy anymore.

A Peek Under the Hood — How LLM Tools Parse Your Code

LLMs don’t know words; they know tokens. Each token is a piece of the input text, and the model processes these tokens to understand context and predict responses. With enough training data, LLMs have learned to recognise patterns and make educated guesses about what comes next. So models, in essence, take X tokens in and generate Y tokens out. The more context (X) you can provide, the better the output (Y) tends to be. However, each model has its own context window limitations. Today’s models have an advertised limit of around few million tokens, but in practice, only a fraction of that is usable for coding tasks. It’s much better than GPT-3.5’s 32K limit.

But our massive projects won’t fit in a single context window. Even if they did, you don’t want to do that, because computing each token costs time and money. It’s not just one call to an API — agents run in a loop, making multiple calls to refine outputs, and all their memory (including what they read and what they wrote) is going to be part of the context window and you will run out of it pretty quickly.

So, coding agents have to be smart about what parts of your codebase to load into the model at any given time. Much of the heavy work is done by LLMs, which, like any competent developer, would try to guess the structure of the code they’re working with.

Next time you kick off “Agent mode”, look carefully at the tool calls the Cursor makes. You will find a lot of searches across files with patterns matching file names, classes and functions. If I gave you a codebase and no documentation, you would do the same thing. There are optimisations like vector search and embeddings, but fundamentally, it’s the same process.

If your project has massive, tangled files, the assistant has to make hard tradeoffs about what to load. That means it may drop critical logic, or worse, misinterpret and hallucinate stuff. But if your code is modular, well-named, and neatly segmented, the assistant can pull just the right piece and you get faster, cheaper, and more accurate results.

Think of it as coding with an additional teammate: one who doesn’t complain, but has strict memory limits.

Coding for Humans and Machines

The lesson is simple: clean code has never mattered more. I’m not saying to follow “Clean Code” dogmatically, but strive for clarity, modularity, and good naming. Readable, modular, and well-structured code isn’t just about making your future self (or a junior dev) grateful. It’s also about enabling the growing ecosystem of AI coding tools to work effectively. The better your codebase, The better these tools perform, the more they boost your productivity and reduce friction.

So, in 2025 and beyond, don’t just code for humans. Code for humans and machines. That’s how you’ll get the best of both worlds.

People clapped. We thought it was just a clever tool and didn’t know how it would change everything.

Soon, something strange began to happen. The machines weren’t just helping. They were thinking, writing, and fixing mistakes.

Picking up where humans left off.

Then came the Cursor. Windsurf. Devin. Names that seemed harmless, like pets or weekend hobbies. But these were not toys. They read code like old detectives read letters. They poked into corners, solved problems, and sometimes rewrote the story.

It sounds like science fiction: a machine that chats, edits files, runs commands and corrects itself and keeps going even after it stumbles.

But there is no secret. No magic.

It is just a loop. A model. And lots of words and some clever API calls.

Now, we are going to build one. From scratch. Piece by piece.

Let’s chat!

A coding agent is a chat interface connected to a Large Language Model (LLM) like OpenAI’s GPT models or (my preferred) Anthropic’s Claude Sonnet models. You need an API key from OpenAI, Anthropic or Gemini, a system prompt to define the agent’s persona, ask the user for input in a loop, send the conversation (messages) to the OpenAI chat completions API endpoint, get a result, and print it.

import json
import requests


def load_config(config_path):
    with open(config_path, 'r') as file:
        config = json.load(file)
    return {
        "endpoint": config["endpoint"],
        "api_key": config["api_key"],
        "model": config["model"]
    }


def chat_completion(endpoint, api_key, model, messages):
    request = {"model": model, "messages": messages}
    headers = {"Authorization": f"Bearer {api_key}"}
    response = requests.post(
        f"{endpoint}/chat/completions",
        json=request,
        headers=headers
    )
    response.raise_for_status()
    return response.json()


def main():
    config = load_config('config.json')

    system_prompt = """You are a helpful chat assistant."""

    messages = [{"role": "system", "content": system_prompt}]

    print("Chat with the assistant. Type 'exit' to quit.")

    while True:
        user_input = input("> ")
        if user_input.lower() == "exit":
            break

        messages.append({"role": "user", "content": user_input})

        response = chat_completion(
            endpoint=config["endpoint"],
            api_key=config["api_key"],
            model=config["model"],
            messages=messages
        )

        choice = response["choices"][0]
        message = choice["message"]
        messages.append(message)

        print(message["content"])


if __name__ == "__main__":
    main()

Just 60 lines, and you have ChatGPT Pro! It is missing things like chat persistence, image inputs, and all other UI niceties, but you get the idea. The core is simple.

Note: I’m not using any LLM SDK here, as they are plain HTTP calls underneath. If you use them, you can make this code even smaller.

AI, follow my instructions!

The GPT 3 model, more or less a glorified autocomplete, was the precursor to ChatGPT (GPT 3.5). ChatGPT was a fine-tuned GPT 3 model that was good at following instructions. If you told ChatGPT to rewrite something in the style of King James’ Bible, it complied.

Soon, we figured we could also instruct the models to generate and understand “structured” outputs like JSON. You probably have done things like asking ChatGPT to make tables from JSON.

A LLM with tools is an “agent”. It now has agency to work on things we tell it to do. You can prompt the models to generate a structured response, parse them, call your custom code, pass the prompt again to the agent, and get your formatted response. Modify the system prompt in the chat code we created before and see how it works.

You are a helpful home assistant, answer the users queries based on your knowledge and provided the tools about home automation

Always format your response with tags.
DO NOT include any other content before or after the tags
- Use the speak tag to return a user-facing message
- Use the call tag to output a json which we will use to call a tool to do something
- You SHOULD respond with multiple call tags if needed

Example:

<speak>content to speak</speak>
<call>{ "name": "function_name", "args": { "arg1": "value", "arg2": "value" } }</call>


You have access to following tools:
1. temperature(): number in degree celsius
2. humidity(): number from 0 to 1 to denote relative humidity
3. list_devices(): { deviceId: string, name: string}, name and deviceId of all configured devices
4. get_device_properties(deviceId: string): Record<string, any> of device properties
5. set_device_properties(deviceId: string, properties: Record<string, any>): boolean, set device properties

When you request to run a tool, you will receive JSON formatted tool output in the next message
as { "output": "tool output" } or { "error": "tool error" }

Read the tool output and emit a helpful error message

If you ask a question, the model will respond with a structured response now. Now respond with a JSON value and see the model format the values appropriately in natural language. If you ask a general query, the model also answers with its intelligence.

$ python chat.py
Chat with the assistant. Type 'exit' to quit.
> what is the temperature and humidity
<speak>Let me check the current temperature and humidity for you.</speak>
<call>{ "name": "temperature", "args": {} }</call>
> { "output": 32 }
<speak>The current temperature is 32 degrees Celsius. Now, I will check the humidity.</speak>
<call>{ "name": "humidity", "args": {} }</call>
> { "output": 0.78 }
<speak>The current humidity level is approximately 78%.</speak>                                                                                     > C compilers are written in C! how was the first compiler created then?
<speak>The first compiler was created as a separate program written in a different programming language or directly in machine code. For example, the earliest compiler for a high-level language like Fortran was initially written in assembly language or machine-specific code before being rewritten in the target language itself. This process is known as "bootstrapping," where a simpler or existing compiler is used to create a more advanced one. Over time, this foundational work enabled the development of self-hosting compilers written in the language they
compile.</speak>

Building our first tool

Parsing model responses was so 2022. It’s 2025 now, and model providers have done a ton on RLHF to ensure models use tools effectively and also provide a way to specify which tools your models call in the chat completions API itself.

Let’s start by defining our first tool read, which reads a file.

def read(path: str):
    with open(path, 'r') as f:
        return f.read()

Then, modify our chat_completion code to accept tools and pass it to the API.

def chat_completion(endpoint, api_key, model, messages, tools):
    request = {"model": model, "messages": messages, "tools": tools}
    headers = {"Authorization": f"Bearer {api_key}"}
    response = requests.post(
        f"{endpoint}/chat/completions",
        json=request,
        headers=headers
    )
    response.raise_for_status()
    return response.json()

The OpenAI API spec wants us to use JSON schema to describe the tool params, so let’s define our tools and add a description of our read tool.

When the model wants to call a tool, it will specify it in its finish_reason; let’s handle that and call our tool. Since we need to pass the result of our tool immediately call back to the LLM and generate further responses, let’s add a nested loop to process tool calls and yield the prompt back to the user only when the model completely processes your request.

# ... rest of the code, same as before
def main():
    config = load_config('config.json')

    system_prompt = """You are a helpful chat assistant."""

    messages = [{"role": "system", "content": system_prompt}]

    # Our tools go here...
    tools = [{
        "type": "function",
        "function": {
            "name": "read",
            "description": "Read a text file",
            "parameters": {
                "type": "object",
                "properties": {
                    "path": {
                        "type": "string",
                        "description": "File path"
                    }
                },
                "required": ["path"]
            }
        }
    }]

    tool_functions = {
        "read": read
    }

    print("Chat with the assistant. Type 'exit' to quit.")

    while True:
        user_input = input("> ")
        if user_input.lower() == "exit":
            break

        messages.append({"role": "user", "content": user_input})

        # Our inner tool-calling loop goes here
        processing = True
        while processing:
            response = chat_completion(
                endpoint=config["endpoint"],
                api_key=config["api_key"],
                model=config["model"],
                tools=tools,
                messages=messages
            )

            choice = response["choices"][0]
            message = choice["message"]
            messages.append(message)

            print(message["content"])

            # Handle tool calls
            if choice["finish_reason"] == "tool_calls":
                tool_calls = message["tool_calls"]
                for tool_call in tool_calls:
                    tool_call_id = tool_call["id"]
                    tool_call_function = tool_call["function"]
                    tool_name = tool_call_function["name"]
                    tool_args = tool_call_function["arguments"]

                    tool_args_dict = json.loads(tool_args)
                    # Tell we are executing the function
                    print(f"{tool_name}({str(tool_args_dict)})")
                    # Execute the tool function
                    result = tool_functions[tool_name](**tool_args_dict)

                    messages.append({
                        "role": "tool",
                        "content": result,
                        "tool_call_id": tool_call_id
                    })


            # If the LLM ran out of output tokens
            elif choice["finish_reason"] == "length":
                messages.append({
                    "role": "user",
                    "content": "Please continue."
                })

            # We are done doing our task, yield back to the user
            elif choice["finish_reason"] == "stop":
                processing = False

That’s our first tool. It doesn’t do much, but now the LLM can read files and answer based on that. Let’s try it out.

$ python chat.py
Chat with the assistant. Type 'exit' to quit.
> what does chat.py do?
read({'path': 'main.py'})
The chat.py script is designed to operate as a helpful assistant. It interacts with users via prompts and can execute various
tools to facilitate development workflows

Bringing it all together

Think about the elementary set of tools you, as a programmer, need to write code: list all other files, read the files, write your changes and run commands to test your code. Let’s add a ls and a write tool to explore the codebase and write any changes you ask the agent to make.

# Add tool implementions

def ls(path: str):
    if not os.path.exists(path):
        return "Path does not exist."

    if os.path.isfile(path):
        return f"File: {path}"

    items = os.listdir(path)
    items_str = "\n".join(items)
    return f"Directory: {path}\n{items_str}"

def write(path: str, content: str):
    with open(path, 'w') as file:
        file.write(content)
    return "File written successfully."

# Register our tools

def main():
    # ... rest of the code
    tools = [
        {
            "type": "function",
            "function": {
                "name": "ls",
                "description": "List files and directories.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "path": {
                            "type": "string",
                            "description": "Directory path"
                        }
                    },
                    "required": ["path"]
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "read",
                "description": "Read a text file",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "path": {
                            "type": "string",
                            "description": "File patb"
                        }
                    },
                    "required": ["path"]
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "write",
                "description": "Write content to a file.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "path": {
                            "type": "string",
                            "description": "File path"
                        },
                        "content": {
                            "type": "string",
                            "description": "The content to write to the file."
                        }
                    },
                    "required": ["path", "content"]
                }
            }
        }
    ]

    tool_functions = {
        "ls": ls,
        "read": read,
        "write": write
    }
    # ... rest of the code

While building any agent, think of what you, as a human, would use to complete your tasks. You can start with a basic set of tools and supply more specialized ones to make your agent more efficient.

For example, let’s use the agent we just created to introduce a new search_replace tool to make more minor file edits instead of rewriting everything.

$ python chat.py
What would you like to do?
> Add a new search_replace tool to main.py
ls({'path': './'})
read({'path': 'chat.py'})
To add a new search_replace tool to `chat.py`, we will:

1. Add a Python function called `search_replace(path, search_text, replace_text)` that reads a file, replaces all instances of `search_text` with `replace_text`, and writes back the changes.
2. Add the `search_replace` tool specification to the `tools` list.
3. Add the mapping in `tool_functions`.

Would you like these lines added for you and the file updated, or do you want to copy and paste them manually?
> Go ahead
read({'path': 'chat.py'})
write({'path': 'chat.py', 'content': 'import json... (trimmed)'})
The `search_replace` tool has been added to `simplified_main.py`.
You can now use this tool to find and replace text in any file via your
assistant interface. If you need to test its usage or want an example command, just let me know!

If you notice, our agent generates a lot of markdown content, asks for confirmation, etc. By tweaking the system prompt, we can change our agent’s personality and make it behave as we want.

Try the following system prompt. You’ll see your agent perform a lot smarter.

You are a helpful assistant that assist users with software development tasks.
- Always analyze the user's codebase before providing a solution.
- Use the tools provided to you.
- If you are unsure about the user's intent, ask clarifying questions.
- Keep your responses concise and to the point, avoiding unnecessary verbosity.
Respond in plain text, and do not include any code blocks or formatting.

And this is prompt engineering 101, no secret sauce. Experiment till you convince your model to behave as you want.

Here is the complete 250 lines of our basic agent.

Play around with our simple agent, and you will find that it’s pretty smart already. It can make edits, write documentation, and review code. If you add specialized tools, like a bash tool to execute commands or a web search tool to look for information on the web, you can make it even more capable.

Then what makes Cursor, Windsurf or <insert AI coding tool here> any special?

Nothing.

This architecture for building agents isn’t just for toy projects. All the cutting-edge copilots work this way: orchestrating LLMs and practical function calls. The trick is to start simple, add chat and tools, feed a lot of tokens to the models and then the richer interactions. I started writing this article earlier, but OpenAI’s Codex CLI release this week just confirmed the theory.

Some of these agents are integrated into editors and IDEs as plugins, which gives them a nice UI and a varying level of tool integration.

This scrappy script does the job, but you can make it much more efficient. You can extend this by adding version control tools or semantic code searches. I have been implementing a more extensive version of this agent with a language server and debugging capabilities.

The real magic: context engineering

If you’ve read this far, you’re either in awe or disbelief, asking, “No, there must be something, right?”

As we just saw, there is no secret recipe for a coding agent, and as the models have gotten smaller, there is no moat in the system prompts as well. However, these agents have one serious disadvantage compared to humans: memory, or in LLM terms, the context length.

Each time you say something or run a tool and provide its output, we keep appending the messages and feeding them to the next API call. This sequence of messages is the model’s context to generate the next set of output tokens. The original ChatGPT model had a token length of 16,384 tokens, translating to roughly 12,000 words. Later models have improved the token length significantly, with Google’s Gemini topping the charts with 2 million tokens, and the newest OpenAI models have a 1 million context size.

But even for a small codebase, you will quickly run out of tokens if you iterate enough. Also, with a large token length, the models tend to lose alignment with your instructions and increase the hallucination rate. My personal experience with Claude, which has 200K token sizes, starts to degrade its performance at around 70K tokens.

As we are paying tokens in + tokens out in every API call, a large context length will not only increase latency, as the model has to read through the entire conversation to generate its output but also the costs, which grow quadratically as the context grows.

So, a lot of creative engineering is involved in managing the model context as the agent performs its tasks. Some agents do summarization once they are done with a task, while some mix vector searches to mutate the context.

With a large enough context window and lots of tokens (and $$$), we won’t need this either. But for now, this is where the real creative magic is.

Exciting times are coming.

Today, I'm going to step aside from the usual technical jargon and talk about a concept that's relevant in the world of software development and our daily lives.

It's called the theory of broken windows.

What is the theory of broken windows?

The theory of broken windows is a criminological theory that suggests that visible signs of crime, anti-social behaviour, and civil disorder create an urban environment that encourages further crime and disorder, including serious crimes.

The theory was introduced in 1982 by social scientists James Q. Wilson and George L. Kelling. They argued that maintaining and monitoring urban environments to prevent small crimes such as vandalism, public drinking, and fare evasion helps create an atmosphere of order and lawfulness, thereby preventing more serious crimes.

Their theory states that if a window in a building is broken and left unrepaired, people walking by will conclude that no one cares and is in charge. As a result, they will be more likely to break windows and commit other acts of vandalism.

In everyday life

Scholars have applied the broken windows theory to various aspects of life beyond criminology. For example, they have used it to explain how small problems can escalate into larger ones if addressed.

A common example is if there is litter on the street, people are more likely to litter. If you leave graffiti on a wall, more graffiti will follow. If you walk around your neighbourhood, you will notice how this principle plays out in real life.

It also applies to individual behaviour. Think about a messy room. If you leave a piece of clothing on the floor, you are likely to leave more clothes on the floor, and soon enough, your room will be a mess. The same goes for an unkempt garden, a cluttered desk, or a neglected car.

In software development

In software development, you can apply the theory of broken windows can be to code quality. If a codebase has a lot of inconsistencies, bad practices, and outdated code, developers are more likely to continue writing poor code. They might not feel the need to refactor or improve the codebase because it already looks messy and disorganised.

On the other hand, if a codebase is well-maintained, clean, and follows best practices, developers are more likely to adhere to those standards. They will be more inclined to write clean code, refactor when necessary, and maintain the overall quality of the codebase. It creates a positive feedback loop that reinforces good practices.

I've seen this theory play out in various software projects throughout my career. Teams prioritising code quality, consistency, and maintainability tend to produce better software in the long run. They are more productive, have fewer bugs, and can effectively respond to changes. On the other hand, teams that neglect these aspects often struggle with technical debt, bugs, and slow development cycles. Since they are used to this laxity, it becomes challenging to convince them to adhere to standard best-practices.

What does it tell us?

The theory of broken windows is a powerful concept that highlights the importance of maintaining order and addressing small problems before they escalate into larger ones. Whether in crime prevention, everyday life, or software development, the principle remains the same: small things matter. By paying attention to the details and keeping things in order, we can create an environment that fosters positive outcomes and discourages negative ones.

So the next time you see a broken window, whether a literal or metaphorical one, remember the importance of fixing it before more windows get broken. It saves you a lot of trouble in the long run.

Unless you tame them!

Step 1: Store Logs in a Centralized Location

Logs are only helpful if you can access them easily when you need them. If logs are scattered across multiple servers, formats, and locations, they are hard to access, search, analyze and reason about.

The first step in taming logs is to store them in a centralized location, which makes them accessible from a single location. You can use a log management service that comes with your cloud provider, such as AWS CloudWatch Logs, Google Cloud Logging, or Azure Monitor. You can also use a third-party log management service, such as Datadog, Loggly, Papertrail, Splunk, Sumo Logic or Logz.io.

You can also use an open-source log management solution, such as Fluentd, ELK stack, GrayLog, Grafana + Loki, or ClickHouse.

Such services will require installing and configuring an agent to ship logs to your log server. Some agents will also ship system logs, allowing you to observe other parts of your servers, such as system services.

Step 2: Format Logs in a Standardized Way

If logs are in different formats, with varying fields and structure, they are hard for humans and machines to parse. Free-form log text only allows you to search using a full-text index, which is both expensive and limited in terms of expressiveness.

By formatting logs in a standardized way, you can define a standard parsing schema that you can use to extract essential fields from logs, such as timestamp, log level, message, and metadata. Logs with metadata make them easier to search and filter. Log agents will also add some metadata about the source automatically. If you use “fluentd” or “logstash” (from elk-stack), you can use “grok patterns” to parse logs and extract even more fields.

You can define a straightforward log format such as:

<timestamp> <log level> <message> [key1=value1 key2=value2 ...]

Your log agent can then parse this format, extract the timestamp, log level, message, and metadata fields, and forward them to your log server.

Step 3: Define a correlation ID

Your application will have multiple concurrent executions, each generating various logs. To make it easier to trace the logs generated by a single execution, you can define a correlation ID unique to each execution at the very start of invocation. You can pass this correlation ID from one component to another and print it in every logging statement to trace the logs generated by a single execution across multiple components.

Let's add a correlation ID to our log format:

<timestamp> <correlation ID> <log level> <message> [key1=value1 key2=value2 ...]

Your log agent can then extract the correlation ID field and forward it to your log server.

Step 4: Define a trace ID

In a distributed system, a single request can span multiple services. To make it easier to trace the logs generated by a single request across various services, you can define a trace ID unique to each request at the very start of the request. Your code can pass this trace ID between services, and you can reuse the same trace ID while printing logs.

Let's add a trace ID to our log format:

<timestamp> <trace ID> <correlation ID> <log level> <message> [key1=value1 key2=value2 ...]

Step 5: Define log levels

Logs can be noisy and generate a lot of information that is not useful. To make logs more useful, you can define a log level that indicates the importance of a log message. You can then filter logs based on their log level so that you only see the log messages that are important to you.

Define good log levels such as:

1. DEBUG verbose; valuable information for debugging and troubleshooting

2. INFO good to know something happened

3. WARN something unexpected happened, but the application can recover

4. ERROR something unexpected happened, and the application cannot recover

5. CRITICAL something unexpected happened, and the application will crash

Review the log lines introduced in your code base and ensure they use the correct log level. Logging too much information at a higher log level can make finding important logs harder.

Step 6: Implement structured logging

If logs are in free-form text format, they are challenging to search and filter. By implementing structured logging, you can define a standard schema that allows you to use off-the-shelf tools to extract essential fields from logs, such as timestamp, log level, message, and metadata.

For example, for logs associated with a particular order, you can add the order ID to the log message as structured metadata instead of embedding it in the log string itself so that you can search for all logs associated with that order.

2021-01-01T00:00:00Z 1234 5678 INFO Order created order_id=ABCD1234

To emit structured logs, you can use a structured logging library such as logrus or zap for Go, logback for Java, or Structlog for Python. JSON is a widely used format for structured logs, and logfmt is also a good option.

Your log agent can then parse this format, extract the timestamp, log level, message, and metadata fields, and forward them to your log server. Your log server can then index these fields and make them searchable. You can even create dashboards and alerts based on these fields.

In JSON format, the log message would look like this:

{
  "timestamp": "2021-01-01T00:00:00Z",
  "correlation_id": "1234",
  "trace_id": "5678",
  "log_level": "INFO",
  "message": "Order created",
  "metadata": {
    "order_id": "ABCD1234"
  }
}

While JSON is a great candidate format for structured logs, it can be verbose. You can use a more compact format such as “logfmt”:

timestamp=2021-01-01T00:00:00Z correlation_id=1234 trace_id=5678 log_level=INFO message="Order created" order_id=ABCD1234

If your logging library does not support structured logging, you can still use a log format by defining a simple wrapper for formatting your log messages accordingly.

Here is an example in Go that prints logs in logfmt:

package main

import (
	"fmt"
	"time"
)

func logfmt(level string, message string, metadata map[string]string) {
	fmt.Printf("timestamp=%s log_level=%s message=%q", time.Now().Format(time.RFC3339), level, message)

	for k, v := range metadata {
		fmt.Printf(" %s=%q", k, v)
	}

	fmt.Println()
}

You can then use this wrapper to log messages in a structured format:

logfmt("INFO", "Order created", map[string]string{"order_id": "ABCD1234"})

⚠️ Some people find JSON unyielding, especially during development. You can configure your logging library to log in JSON format in production and a more human-readable format during development. Libraries like Logback and Structlog have a developer friendly “console” format that you can use during local development.

Step 7: Add context to your logs

Log messages are context-dependent. They depend on the application's state when and where you generate the log message. By adding context to your log messages, you can provide additional information that can help you understand the log message better.

You can add context such as:

• Request URL

• Request method

• Source IP address

• Status code

• Any other relevant information that can help you understand the log message better

Use a middleware to add this context to your log messages. This middleware can extract and add information from the request to the log message.

Step 8: Cut down on log verbosity

Don't log everything. Logs can generate a lot of information that is not useful. To make logs more useful, reduce log verbosity by logging only the vital information.

Log something only if something is:

1. Useful for understanding the state of the application

2. Useful for debugging and troubleshooting

3. Useful for monitoring and alerting

4. Useful for auditing and compliance

Step 9: Check logs for sensitive data

In 2019, Facebook (now Meta) was under fire for inadvertently logging the passwords of hundreds of millions of users due to a misconfiguration. While there was no evidence of abuse or unauthorized access, they received major flak from the global security and privacy community regarding the company's security practices and the broader implications for user privacy.

Logs can inadvertently capture and expose sensitive information such as passwords, API keys, or personal data. Regularly review your logs to ensure that sensitive data is either masked or excluded from logs entirely. Implement automated tools to scan logs for such information. Sanitizing logs is critical to prevent security breaches and comply with data protection regulations.

Many log management services provide tools to scan log files in real-time, flagging or redacting any information that matches known patterns, such as credit card numbers, social security numbers, emails, or other personally identifiable information (PII). Logging libraries also provide hooks and post-processors to enable such detection at the application level and also offer remediation features like automatically masking or removing sensitive data before it can be stored or transmitted.

Step 10: Consider supporting changing log levels at runtime

Changing log levels at runtime can be helpful for debugging and troubleshooting. It allows you to change the log level of your application without restarting it. This feature can be beneficial if you want to increase the log level of your application to debug an issue and then decrease it once you have resolved the issue.

You can implement a log-level change endpoint in your application to change the log level of your application at runtime. You must protect this endpoint by authentication and authorization to prevent unauthorized access.

Java libraries like Slf4j and Logback support changing log levels at runtime using JMX or Jolokia. You can easily implement this feature in other languages as well by exposing secured management endpoints.

Step 11: Monitor your logs

Logs can be a valuable source of information about your application's state. While they are no alternative to dedicated observability tooling like metrics and application traces, by monitoring your logs, you can detect issues before they become critical. You can also create dashboards and alerts based on log messages to monitor your application's health.

Log management services provide matching, filtering and aggregation capabilites to utilize patterns to match log occurences and emit metrics or trigger alerts.

For example, most databases provide something called as a slow query log, where they log queries with execution time greater than a configured value. By monitoring slow query occurences, your logging system can alert you about a potential issue in your application.

Step 12: Implement log rotation and log retention policies

Logs can consume a lot of storage which can get very expensive especially when using managed log management services. Define sensible archival and deletion policies for logs. You can choose to store logs in “hot” storage — like in ClickHouse DB for 60 days and then move it to “cold” storage like “S3” for storing it upto a year. Bulk object storage is cheaper than storing it in an actively searchable database. Tools like Amazon Athena and Loki still allow you to query archived logs when needed.

… And the list doesn’t end here

There are many more ways you can employ to make logs more useful. Following these steps one-by-one will give you a robust baseline to start from.

Do you employ some other creative ways to make the best use of logs? Let me know in the comments below!

Leave a comment

The evolution(?) of cables 🔌

The great thing about the cables of yesteryears was that they either worked or did not work. And when they failed, they failed spectacularly due to their analogue signalling. If you ever had a loose mouse connector that skipped constant white noise from speakers or a bad VGA cable producing an occasional rainbow on your monitor, you know what I'm referring to.

However, for good or bad, we have moved to a digital world where we can fail and gracefully degrade. You only know when you try to push the limits, and then you wonder whether it's a faulty cable, connector, or software!

A few years ago at work, we had these Thunderbolt 3-enabled monitors that also supported USB C Displayport Alt mode + data (I will explain these later). Thunderbolt uses the same interface as USB C but only works if you plug it in a particular orientation. However, the monitor works virtually identically in both modes, with the refresh rate being lower and USB ports dropping to Gen 2 speeds (480 MBps) if you have connected it to the USB C mode by plugging the cable wrong. There is no other indication unless you notice the jankyness in the mouse (I wasn't used to 120Hz screens then, so I couldn't see much) or slowness in data transfer speeds over USB. How confusing it was!

One cable to rule them all

USB Type C is everywhere! It charges devices, transfers data, connects peripherals, and has a powerful alt-mode that allows you to use some of the pins for transmitting signals other than USB. It will enable you to connect displays and even transmit analogue audio (which is how many Androids removed the headphone jack). It's the holy grail of connectors. However, it's a super convoluted mess!

The USB-IF messed up big time. With so many optional capabilities, it resulted in maddening permutations and finding cables that can support them is a task. Also, let's not talk about naming now!

As it turns out, USB C is simply the connector form factor. USB 1 - 4 are generations that determine speed, signalling, power delivery, and other capabilities. A USB C cable can be of any generation from 2 to 4 and support any power delivery mode, which gets electronically marked with a marker chip. If the chip fails, it gracefully degrades to USB 2 capabilities. The Thunderbolt standard also uses the USB C connector, another high-speed connectivity solution by Apple and Intel.

Here are some combinations that I know exist:

1. A USB C cable, by default, supports 5V/1.5A (7W) current, but most cables usually support 15W of power. If your phone charges slowly, try a different cable, like a laptop's power cable. Those things are rated at 100W (20V/5A). Have a look at the USB PD Wikipedia article for more combinations of power delivery support. Your cable may support any power level.

2. USB C cables can support USB 2/3/4. The thin and long ones that come for charging are usually USB 2. They only support transmitting data @ 480 Mbps and don't support audio, video, or any other protocol in alt mode.

3. USB C cable "certified" for USB 3, which was first renamed USB 3.1 and then renamed USB 3.2 Gen 1, supports 5 Gbps data transfer and transmitting audio or video. USB 3.2 Gen 2 supports 10 Gbps, USB 3.2 Gen 2 x2 supports 20 Gbps, and USB4 supports 40 Gbps in addition to alt-mode. (What the hell is this USB-IF!?) During alt-mode, the cables either fall back to USB 2 or 3 speeds depending on the generation and the devices.

4. Thunderbolt is a different beast. It looks like a USB cable but only supports Thunderbolt at 40/80 Gbps for Gen 3 and Gen 4, respectively. If connected to non-Thunderbolt ports, it falls back to USB 2. Thunderbolt supports audio and video output, but you cannot use a Thunderbolt cable to connect to displays over DisplayPort alt mode.

These are the combinations I have personally verified with different hardware and a box full of USB C cables. Want more madness? See the Wikipedia article for USB C.

Display cables

You can connect displays using HDMI or DisplayPort (DP). These are much more robust than analog-based VGA or component connectors. Currently, DP 2.0 is faster, with 80 Gpbs supporting 8k displays at a 120 Hz refresh rate. HDMI is widely used with media equipment and can transfer audio or video. DP is more popular on computers. Both standards are always playing catch-up in terms of capabilities. But simply connecting two devices with a compatible cable is not enough.

For HDMI and DP, devices typically advertise the version of the connector they support, each being backwards compatible. For example, DP 1.4 supports 4K@120 Hz at 10-bit colour, and HDMI 2.0 supports 4K@60 Hz at 8-bit colour. However, for some reason, god knows what, the HDMI and the VESA alliance explicitly forbid cable manufacturers from stating the maximum version the cable is certified to work with. You must now tally the speed or designator to find the corresponding version.

1. A high-speed HDMI cable or an HBR DP cable supports 4K@30 Hz using HDMI 1.4 or DP 1.2

2. A premium high-speed HDMI cable or an HBR2 DP cable supports 4K@60Hz using HDMI 2 or DP 1.4

3. An HBR3 DP cable supports 4K@120 Hz using DP 1.4

4. A UHBR DP cable or an ultra high-speed HDMI cable supports 8K@60 Hz using DP 2.0 or HDMI 2.1

5. You can connect a DP output to an HDMI input. The HDMI version would be the nearest compatible version that the port support.

Also, both HDMI and DP can work in DisplayPort Alt-mode, adding another layer of compatibility complexity.

The harbringing of USB Type C and the evolution of HDMI and DisplayPort have simplified the cable conundrum by allowing a single connector to handle multiple functions. However, this advancement has also introduced a new layer of complexity with multiple modes of operation, power delivery support, and varying data transfer speeds. While these modern connectivity solutions offer incredible versatility, users need to clearly understand these cables' diverse capabilities and limitations to ensure seamless and optimal device connectivity.

As technology advances, we'll likely see further refinements in these standards to streamline the user experience and reduce cable compatibility confusion. Things are already in motion. The Thunderbolt and USB4 specifications merged, so there is one less permutation and some good news to cheer on!

Timestamps and Time zones

A timestamp is a label that indicates the specific date and time an event occurred. It's like you time-captured a moment and stamped it for reference. It denotes a point in history. But how you read it will depend on which “time zone” you are using. A timestamp which reads 12:00 am on British Standard Time, will read 5:30 am on Indian Standard Time.

Time zones are geographical regions that follow a standard time defined by their offset from the Universal Coordinated Time (UTC). However, the rules governing these time zones can sometimes be a convoluted conundrum to decipher.

The UTC Advantage

Coordinated Universal Time (UTC) is the global standard for regulating clocks and time. It establishes a reference for the current time and forms the basis for civil time and time zones.

UTC is deterministic and unambiguous. It's determined from highly precise atomic clocks and is not subject to the whims of time zones or daylight saving time.

Coding for UTC eradicates daylight saving time, saving you from the horrors of ambiguous time. Moreover, the fixed rules make operations like calculating differences between timestamps a breeze. Extra brownie points for no vanishing or extra hours!

Users vs Servers

Your users are everywhere in the world. Some of them are also wanderers, traveling across time zones. Their primary concern lies in knowing what time an event happened. Servers, on the contrary, remain static. If you tell a user that they withdrew money at 3:00 AM tomorrow, they will be perplexed.

Daylight Saving Time: More Trouble Than It's Worth?

Daylight Saving Time has long been thought to be a beneficial strategy. But the reality seems to tiptoe on the contrary. Opting for this leads to an unexpected hour popping out of or vanishing into oblivion—not to mention the excessive chaos it causes!

Ignoring UTC: A Pandora's Box

Should you shun the UTC and stick to local time, prepare for a swarm of bugs, angry users, and disgruntled developers on your plate. Your coded masterpiece could morph into an unsolvable labyrinth filled with timezone turbulence, Daylight Saving Time debacle, and other trickery of time manipulation. Consequently, a chunk of your time debugging, with little to spare for actual development.

Using UTC in Your Code

Set your Server to UTC

Ensure that you set your servers to UTC. This will save you the hassle of converting time zones and daylight saving time. By default, every piece of code you write will be using UTC.

To do this in a Linux server with systemd, you can set the timezone using the timedatectl command:

timedatectl set-timezone UTC

Set your Application to UTC

If you can't set your server to UTC, consider using a library to handle time in UTC. Java, for instance, has a Clock you can utilize to get the current time in UTC. Similarly, other languages allow you to set the timezone for your application.

Some examples:

class Example {
    public static void main(String[] args) {
        // A clock that returns the current time using UTC time-zone.
        Clock clock = Clock.systemUTC();
        LocalDateTime now = LocalDateTime.now(clock);
        System.out.println(now);
    }
}

class Example {
    public static void main(String[] args) {
        // Set default timezone to UTC
        TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
        // Get the current time,
        LocalDateTime now = LocalDateTime.now();
        System.out.println(now);
    }
}

from datetime import datetime
from pytz import utc

# Get the current time in UTC
# This will return a timezone-aware datetime object
# with the timezone set to UTC
now = datetime.now(utc)

Timestamps in databases

When storing timestamps in databases, always use UTC. It will ensure consistency across your application and prevent timezone-related bugs. Most databases use the TIMESTAMP data type to store timestamps in UTC. When displaying them on the front end, you can format them to the user's timezone.

Timestamps in APIs

Adopt the ISO 8601 format with UTC timezone for your APIs. Make it a point to state the timezone explicitly in your response.

{
  "timestamp": "2024-01-01T12:00:00Z"
}

Using ISO 8601 with UTC timezone will ensure your timestamps are unambiguous and consistent across different systems.

Displaying Time to Users

When displaying timestamps to users, always convert them to the user's timezone. It will ensure that the user sees the time in their local time, making it easier to understand. You can use libraries like `moment.js` in JavaScript to handle timezone conversions easily.

const timestamp = "2024-01-01T12:00:00Z";
const localTime = moment(timestamp).local();
console.log(localTime.format("YYYY-MM-DD HH:mm:ss"));

If needed, you can also display the timezone and timestamp to avoid confusion. You can also provide an option for users to change the timezone in your application settings.

Scheduling Events: An Exception to the Rule

If future schedules are required, use user-defined time, date, and timezone. Do not convert them to UTC; store them as they are.

But Why?

Users anticipate the event to occur at that precise time in their respective time zones. An unexpected shift due to conversion to UTC, courtesy of the absurd rules of DST, can be a source of confusion.

Timezone Handling: (not) A Cake Walk

Correct and effective timezone is hard. However, therein lies the art of coding—tackling problems head-on and emerging victorious. And using UTC for your timestamps definitely makes that journey a tad bit less winding. So save yourself from the time zone madness and just F**ing use UTC!

I wanted to build a custom Zsh prompt that was lightning-fast, minimalistic, and focused on performance. I wanted to understand how Zsh works under the hood and how I could leverage its features to build a fast and minimal prompt.

Let's go! 🚀

How Zsh prompts work

Traditionally, shells define the prompt using the PS1 environment variable. Zsh supports this for compatibility but also provides more advanced ways to define prompts using the PROMPT and RPROMPT variables.

The PROMPT variable defines the prompt's left side, while the RPROMPT variable defines the right side. These variables can contain a mix of text and escape sequences that Zsh expands when the prompt is displayed.

Most Zsh prompts use a combination of escape sequences and custom functions that execute when the prompt is displayed. These functions can display dynamic information like the current Git branch, the current Python virtual environment, or the current NodeJS version.

Building a custom Zsh prompt

Through years of using Zsh, I've come to appreciate minimalistic prompts that are fast and discreet. I wanted to build a prompt that displayed only the essential information, like the current working directory, git branch and status. Something like this:

~/projects/my-project (branch) [status]

$

Zsh uses escape sequences to display dynamic information in the prompt. For example, the %n escape sequence expands to the current username, the %~ escape sequence displays the current working directory, and the %# escape sequence expands to a # if the current user has superuser privileges.

To quickly iterate and test the prompt, you can set the `PROMPT` variable in the current shell session. For example, the default Zsh prompt in macOS is "%n@%m %~%# ", which displays the username, host name, working directory, and a `#` if the user has superuser privileges. It would be equivalent to setting the PROMPT variable to:

PROMPT='%n@%m %~%# '

Using this as a starting point, I decided to get rid of the username and host name (as I'm the only user on my machine) to end with something like this:

PROMPT='%~ %# '

Going multi-line

I'm a big fan of multi-line prompts, as they allow me to display long paths and more information without cluttering the prompt.

My first try was to sneak in a newline character (\n) in the PROMPT variable, but that didn't work. Zsh doesn't support escape sequences in the PROMPT variable, so I used the precmd hook to print the CWD and display the prompt on the following line.

The precmd hook executes before each prompt is displayed and can be used to run custom commands or functions. I used the print with -P inside it to interpret prompt expansion sequences, just like in the PROMPT variable.

precmd() {
  print -P "%~"
}

PROMPT='%# '

It worked like a charm; I could display the current working directory on a new line.

Adding Git information

Zsh has built-in support for displaying Git information in the prompt. You can use the vcs_info module to display the current Git branch and status in the prompt.

To enable the vcs_info module, you must load it into your Zsh session using the autoload command.

autoload -Uz vcs_info

Once loaded, you must invoke the vcs_info function in the precmd hook to update the Git information before displaying each prompt. The function then populates the vcs_info_msg_0_ variable with the Git information, which you can display in the prompt.

precmd() {
  vcs_info
  print -P "%~ ${vcs_info_msg_0_}"
}

Except that it didn't work. Zsh does not enable the expansion of variables in prompts by default, so I had to enable it using the setopt prompt_subst command.

setopt prompt_subst

Voilà! I now had a custom Zsh prompt that displayed the current working directory and Git information.

Customizing the VCS info

You can customize the Git information displayed in the prompt by setting the vcs_info_msg_0_ variable to a custom format string. The format string can contain escape sequences that Zsh expands when the prompt is displayed.

I wanted to have the current branch and staged/unstaged status. So, using zstyle, I changed the format string to:

zstyle ':vcs_info:*' enable git
zstyle ':vcs_info:git:*' formats '%b%f %m%u%c %a'

This format string displays the current branch (%b), the staged (%c) and unstaged (%u) changes, any ongoing action like rebase (%a) and miscellaneous info (%m).

I wanted to display symbols instead of text for the Git status, so I set the stagedstr and unstagedstr variables to + and !.

zstyle ':vcs_info:*' check-for-changes true
zstyle ':vcs_info:*' stagedstr ' +'
zstyle ':vcs_info:*' unstagedstr ' !'

Colours and formatting

I got the prompt working, but it was bland. I wanted to add some colours and formatting to make it pop ✨.

Zsh supports escape sequences for colours and formatting, which you can use to style the prompt. The %F{color} escape sequence sets the foreground colour, the %K{color} escape sequence sets the background colour, and the %B escape sequence sets bold text.

I wanted to display the current working directory in bold, the git branch in blue, and the git status in green and red for staged and unstaged changes. I also wanted to display Powerline symbols for the prompt.

So I headed over to Nerd Fonts, downloaded the patched “Fira Code” font, and set it as the font in iTerm.

I then used escape sequences to set the colours and formatting in the PROMPT and vcs_info styles.

Here's the final prompt. The Powerline symbol for git branch \uE0A0 is unfortunately not supported in any other font and hence appears as an unrecognized square.

zstyle ':vcs_info:git*' formats " %F{blue}%b%f %m%u%c %a "
zstyle ':vcs_info:*' enable git
zstyle ':vcs_info:*' check-for-changes true
zstyle ':vcs_info:*' stagedstr ' %F{green}✚%f'
zstyle ':vcs_info:*' unstagedstr ' %F{red}●%f'

precmd() {
    vcs_info
    print -P '%B%~%b ${vcs_info_msg_0_}'
}

PROMPT='%B%(!.#.$)%b '

And here's how it looks:

I use the Snazzy colour scheme in iTerm, which makes the prompt colours look even better. My Zsh configuration has some more customizations, like history search, syntax highlighting, and autosuggestions, but I'll save that for another post.

And that's it! I now have a custom Zsh prompt that displays the current working directory, Git branch and status in a minimalistic and colourful way.

Email is an essential part of how we communicate in the 21st century. It's also one of the most misunderstood technologies on the Internet. I'll explore how email sending works behind the scenes in this post.

🪄 Trivia: History of email

Email is the most common form of electronic communication, with over 2.5 billion users worldwide. It was invented in 1971 by Ray Tomlinson, who sent the first email from one computer to another.

It took place on ARPANET --the precursor to what we now know as the Internet, using a protocol that later developed into FTP, a file-transfer protocol that was popular before HTTP took over. The early days of Email were much more rudimentary than what we use now.

🔎 Anatomy of an email address

An email address is a string of characters that looks something like this: joe@example.com

The domain name is the part after the @ symbol, which would be "example.com." The domain can be one word or multiple words, which functions like an address for your email. Anyone sending you an email will send it to the email servers associated with your domain. You can sign up at one of the many public email services like Gmail or your ISP's service, which provide you with a mail server, or you can build and run your own.

🚀 Sending email

Sending email is a two-step process. First, you find the mail server of your recipient. Then, you use the SMTP protocol to connect with the mail server and send your message. Sounds simple? Yes, it is!

🔎 Finding the mail server

You can find out which mail server to send your email to by looking up its domain name in DNS MX records: pick one randomly if it has multiple addresses listed.

Technically speaking, there is a priority that servers should follow - the number before the email hostname, but we can ignore it for our example.

If you have a Mac or a Linux system, you can run this command to see the MX records for a given domain. Here is an example for Gmail. You can see 5 SMTP servers listed in the DNS response.

$ dig -t MX gmail.com

; <<>> DiG 9.10.6 <<>> -t MX gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25845
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gmail.com.			IN	MX

;; ANSWER SECTION:
gmail.com.		2531	IN	MX	20 alt2.gmail-smtp-in.l.google.com.
gmail.com.		2531	IN	MX	10 alt1.gmail-smtp-in.l.google.com.
gmail.com.		2531	IN	MX	40 alt4.gmail-smtp-in.l.google.com.
gmail.com.		2531	IN	MX	5 gmail-smtp-in.l.google.com.
gmail.com.		2531	IN	MX	30 alt3.gmail-smtp-in.l.google.com.

;; Query time: 53 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Nov 20 12:20:51 IST 2023
;; MSG SIZE  rcvd: 161

✉️ Sending Email

Once connected, you can send emails using the SMTP protocol. The SMTP protocol is a simple text-based protocol running on TCP, which is how mail servers and clients talk to each other to send and receive email.

Since it's text-based, you can simulate an SMTP exchange using the telnet command. If you have telnet installed on a terminal, you can email yourself.

⚠️ Your success may vary based on whether your ISP allows you to send emails directly and your mail server enables you to send emails without a relay.

$ telnet gmail-smtp-in.l.google.com 25

Trying 2404:6800:4003:c1a::1b...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP d12-20020a170902654c00b001c746b986e2si852655pln.346 - gsmtp
> HELO localhost
250 mx.google.com at your service
> MAIL FROM: <root@localhost>
250 2.1.0 OK d12-20020a170902654c00b001c746b986e2si852655pln.346 - gsmtp
> RCPT TO: <your.email@gmail.com>
250 2.1.5 OK d12-20020a170902654c00b001c746b986e2si852655pln.346 - gsmtp
> DATA
354 Go ahead d12-20020a170902654c00b001c746b986e2si852655pln.346 - gsmtp
> Test message from telnet
> .
250 Message accepted for delivery 
> QUIT 
221 mx.google.com closing connection

🔥 Modern-day email

Most people use email through a web client, which abstracts away all the complexities of sending and receiving email by performing all necessary transactions behind the scenes. Email servers, however, offer IMAP, POP3 and SMTP access if you want to use other email software (including mobile apps like Gmail Outlook) to send and receive emails.

You can still send and receive emails from a laptop by running an SMTP server on your local machine and setting the required MX records.

It's hard to believe that email is more than 40 years old. The first message was sent in 1971; ever since then, it has become essential to our everyday lives. It's easy to forget how many things we do with our inboxes--from keeping track of bills and receipts to communicating with friends worldwide. But even though we take this technology for granted now, its origins were quite humble: They started with two people who just wanted an easier way to send messages back and forth between each other!

In the next part, we will discuss what happens when you receive the email, including verification, spam filtering and archiving until you access it on your computer.

We’ve all been there: You’re trying to find an error that doesn’t make sense! You changed this one thing last week, but now it’s broken? 🤬WHAT IS GOING ON?!

Testing is an integral part of developing software

Testing enables you to make sure that your code works as expected, and it can even help with regression testing ✅ — a process where you test existing features to ensure they still work after adding new ones or modifying existing ones. Ultimately, you catch bugs 🐞 before they reach production.

Testing early and often is the best practice for writing software because it allows you to find problems before they become too large or complex for quick fixes later in development.

🔵 Unit testing - test code in isolation

Unit testing is the process of testing individual units of source code. Each time you complete working on a piece of code, you write a unit test for it. It checks the correctness of that particular piece of logic you wrote. Or even better, come up with test cases before writing code, and make sure they pass!

🤔 But code is rarely independent. Therefore in unit tests, you employ mocks and stubs to “fix” the behaviour of other parts of your code and focus on the behaviour of the code-under-test. Your language of choice would have a framework that would do it, like Mockito for Java, Jest and Sinon for JS etc.

For further reading:

👉 You should write unit tests to provide quick feedback. They should be fast ⚡️ and avoid expensive/external operations (using mocks and stubs).

Example of a unit test

You wrote some code that sorts a list of records based on the alphabetical order of first and last names in case of a tie. Your unit tests should contain cases for - empty, few records without a common first name, few records with the same first name etc. and then assert the output.

🟡 Integration testing - test code that interacts with other modules

Integration testing is a form of black box testing that involves testing code against real-world scenarios. The right time to write integration tests is when you have just completed a feature with APIs and interactions.

👉 You should perform integration tests at the module level (micro-service level if you follow the micro-services pattern), where you validate the contract between components. You should validate your code with all its dependencies to start, run and serve your test expectations.

⚠️ You should use real servers for internal dependencies, like 🗄️ databases and 🗃️ caches. You can use a mock server like Wiremock or point them against an actual deployment for external services. Using a mocking library to mock external interactions does not serve the purpose of an integration test.

🐳 Docker and test containers help in spinning up servers for integration testing.

Also, as your code matures, you should build a test harness to test your application against real deployments automatically.

Example of an integration test

You call an API that stores some data in the DB, publishes several events to a queue and returns an HTTP 200 response. You write a test to assert these behaviours.

End-to-end testing and manual testing

👉 You perform end-to-end testing at your system’s outermost, user-accessible part. Suppose your system has multiple microservices powering a single dashboard. In that case, you should concentrate the bulk of your end-to-end testing efforts on testing the behaviour of the dashboard as a whole when deployed with other modules in a test environment.

🔧 Manual testing is the most common type of software testing. It’s also the most expensive and time-consuming but also the most effective. As part of manual testing, you must manually test each feature in your application and ensure it works as expected. Lots of 🖱️ clicking around and ⌨️ entering data, just like the end user would. But those few clicks would touch lots of layers of code at the same time.

The pyramid of testing

The pyramid of testing visually represents the order in which you should run tests. You can also use it to help you decide what tests are appropriate for your codebase. The bottom level represents unit tests, followed by integration tests and manual (UI) testing at the top.

The bottom levels are narrow and cover a limited amount of code at a time, but it is easy to write and debug failures. As you move up, the code under test increases with lots of logic stacking over each other. A single run covers a lot of code at a time, but debugging failures become proportionately 🌵 tricky due to many more ⚙️ moving parts.

How to assert the quality of your test suite

✅ You must design your tests according to the specification of your software. You should easily understand what each test is doing and why it’s essential. A wrong test is a source of bugs if not reviewed correctly.

⚠️ If a test has any dependencies on external systems (such as databases), you should document or mock them to ensure they don’t affect the rest of your tests.

❗️Your test should also be easy to write and read so that if you need to change something in your codebase, it’s easy for someone else (or yourself!) who isn’t familiar with all your classes right away can understand what’s going on in each unit test file.

If you have a piece of code which is complicated to test, there is probably a code-smell hiding.

⚡️ Finally, all tests must run quickly — if a single test takes too long, developers will avoid running it frequently because it adds time to their build process and slows them down when making changes elsewhere in the codebase.

👀 Use code coverage to measure the quality of your test suite objectively. A high code coverage indicates that you have accounted for the various conditions and control flow. Test frameworks usually come with a bundled coverage tool like JaCoCo for Java, coverage.py for Python and Istanbul for JS.

While testing is essential to developing software, it can be challenging to get it right. There are many different types of tests, each serving another purpose. The most important thing is to understand what kind of testing is appropriate at the moment so that you can make sure that tests cover all of your code before deploying it into production and squash those 🐞 bugs!

What's Wrong with "The Trinity"?

🔴 Take Phonepe. I can never find the "scan" or "pay to a phone number" button without spending a hot minute reading through their number of useless offers. It often would be tucked in a small corner among coloured buttons, littered across the screen, designed to draw your attention away from what you wanted to do — may a payment.

🔴 Now take its major competitor, Paytm. They force full-page gambling ads and weird offer banners, hiding the payment buttons. And after you manage to locate it, you receive yet another barrage of ads, spin wheels, cashback animations, and more products to buy, even before confirming your payment 🤦‍♂️. The only good thing is they have the best reliability.

🟠 Google Pay fares better in this regard, but still too many distractions with energy, diyas, mangoes or whatever mini-game they want to develop that week — still a massive welcome change from the Phonepe-Paytm chaos. Unlike others, You can still locate the pay button, which isn't lost in the sea of offers. However, the only big negative of Google Pay — is the neverending saga of payment failures 🤷‍♂️.

🏦 Bank apps are a bug fest 🐞, and by the time you reach the UPI page, they time out before you complete payment due to sEcUriTy 😡!!!

And BHIM, the OG UPI app, which was dead simple in 2016 even for my grand-parents has slowly turned to the Paytm or Phonepe way with an even worse experience because they don't use the system keyboard (for sEcurITy ofc) and typing anything is a nightmare.

Is there an app that is simple, easy to use and distraction-free?

At this point, I’m actually hoping if Apple Wallet launches in India with UPI. There has been talks about an Apple Card in partnership with HDFC Bank, I am going to assume they won’t have an agenda to sell products (other than the hardware), hence probably provide a service that’s simple and useful.

Why?

The real reason why all this happens is the 0 MDR regime. Unlike the case of MDR in credit or debit cards, our government doesn't allow UPI payment service providers to charge a transaction fee.

So, if you as an app want to make money, you must not allow users to make a payment and instead sell whatever you have at any cost and hope some users bite the bullet. Be it BNPL, loans or bill pay. 💸 No wonder some payment apps are struggling too — remember FreeCharge and MobiQuik?

PayZapp is the only app I know that still maintains the sanity of a payments app and offers all essential features (Scan, Pay to mobile, Credit cards, BBPS etc.). It has been my primary for a while now. The transaction failure rates are slightly worse since it runs on the HDFC network.

You might have heard a lot lately about “language models”. They’re computer programs that can understand and generate human-like language. Pretty impressive, right? Let’s see what all this hype is about 👀.

What is a Language Model?

A language model is a statistical model that predicts the next word in a sentence. It takes as input some context and outputs an embedding - a vector representation for each term in your vocabulary. 😵‍💫

Simply put, language models predict what word will come next based on what they have already seen.

For example, if you say, “I am hungry”, and then “I want to eat” 🤤, the language model may predict that you are going to say, “Pizza” 🍕.

N-gram models

N-gram models are one of the first language models developed by NLP researchers.

➡️ These models look at the previous n words in your sentence and predict what word will come next.

Unigram models ( n = 1) are the simplest language models. They are simply the probability of occurrence of each word in the corpora.

Models with higher n-count try to improve over unigram models and encode “context”. Bigram models (n = 2) and Trigram ( n = 3) look at your sentence’s previous two or three words and predict the next word.

✅ Auto-correction, text-to-speech and keyboard suggestions all utilize n-gram models.

Markov Models

Markov Models are an extension of n-gram models. They try to capture language structure by representing each word with a state and thus learning the probability distribution over states.

For example, if your sentence is “I like to drink coffee” ☕️, then the first word might be a state that represents the word “I”. Then, adding the following word (like “like”) becomes a new state representing both words together. It continues until you have added every possible word combination into states in your corpus.

Continuous Space Language Models

With computing becoming cheaper and more accessible, researchers started using neural networks to model language.

They found that training the neural network on a large text dataset could create “continuous space language models” that don’t need to be constrained by discrete states like the Markov Model does. Instead, they can represent all possible words in a sentence as points in a high-dimensional space.

➡️ This means that your model will learn higher-level features about your data and not just individual words or phrases alone to model language.

Word2Vec, published in 2013, was one of the first neural language models to use continuous space representations. The paper’s authors explained that this allowed them to model words more accurately and efficiently, which was a huge step forward in NLP research.

Later, recurrent neural networks like LSTMs took over the NLP world. The advent of attention mechanisms allowed language models to focus on specific parts of sentences while learning what they mean.

➡️ This makes it possible for them to learn about complex relationships between words and the meaning of sentences without requiring human intervention.

Transformers

The current state-of-the-art was introduced in 2017 by researchers from Google and the BERT (Bidirectional Encoder Representations from Transformers). Replacing the prior neural network models, they quickly overtook the NLP world.

➡️ They also allowed researchers to train models on much larger datasets than before, making it easier to build models from scratch using pre-trained model versions.

So, what’s the hype about?

Language models have been around for a while and are routinely used everywhere - from your keyboard auto-complete to the hundreds of chatbots 🤖 you see.

⭐️ But they have started getting incredibly “large” in the past few years - thanks to the fantastic success of transformer models. BERT, RoBERTa, GPT 2/3/chatGPT, and LamBDA are all large language models, some with over several hundred billion training parameters.

⭐️ Being trained on massive training data, these models performed impressively in generating responses to various prompts. They picked up a lot of patterns leading them to answer some complicated queries.

👉 They could come to simple logical conclusions that misled many people about “intelligence” and thus the hype.

AI is not taking away our jobs anytime soon.

The models are still pattern-matching algorithms with a lossy memory of a large chunk of the internet. They often regurgitate made-up or incorrect facts; we all saw what happened with the Google Bard demo, didn’t we?

Is that all?

Large language models are just the tip of the iceberg of what is yet to come. Their most significant contribution - easing communication with humans.

All of today’s tools and algorithms won’t be obsolete by them or any future AI model. Instead, future AI research will focus on bridging them.

⭐️ A few weeks after the ChatGPT sensation, Meta (Facebook) researchers developed a new pattern for this, aptly called Tool Former. They aim to train language models to generate input for a specialized subsystem, retrieve the results and convert them back to natural language.

⭐️ Frameworks like LangChain and Auto-GPT aim to build “agents” that work towards a particular goal by using the power of these LLMs. They start with a description and iteratively devise steps to reach their target by clever prompting. Some of the tasks the community has done are amazing. 🪄

⭐️ Similarly, OpenAI has made available plugins that integrate ChatGPT with external systems over a REST API. A task like asking for available flights will query a flight information system and respond with the most up-to-date results.

⭐️ Google’s Bard is now being integrated across the entire spectrum of GSuite products — maybe finally, we will realize the dream we were sold through “Ok Google” and “Alexa” when they truly become smart assistants 🤖.

The future is exciting! ✨

Here are some libraries that will make your life easier by removing all the grunt work in writing unit tests.

Java provides several frameworks for writing tests, but one library stands out above all others: JUnit. There are alternative libraries like Spock, but JUnit is the most used, so I will limit my discussion to libraries that work well with JUnit.

Mocking frameworks: Mockito & PowerMock

It can be difficult to test when writing code with many dependencies. That's where mocking comes in. A mocking framework allows you to replace real objects with fake ones during testing to isolate the code under test from its dependencies and ensure it works correctly without worrying about external factors like network connections or databases.

Mockito is the de-facto Java library used for creating and testing mocks in software development. It provides a simple and intuitive API for creating mock objects, which can be used in unit tests to simulate the behaviour of objects that are not yet available or are difficult to obtain.

With Mockito, you can mock classes, interfaces, and even individual methods. You can also specify how the mock objects should behave, such as returning a specific value for a given method call or verifying that a method was called a certain number of times.

import org.junit.Test;

import static org.junit.Assert.*;
import static org.mockito.Mockito.*;

class CalculatorTest {
    @Test
    void testAdd() {
        Calculator calculator = mock(Calculator.class);
        when(calculator.add(1, 2)).thenReturn(3);

        assertEquals(3, calculator.add(1, 2));
    }

    @Test
    void testMultiply() {
        Calculator calculator = mock(Calculator.class);
        when(calculator.multiply(2, 3)).thenReturn(6);

        assertEquals(6, calculator.multiply(2, 3));
    }
}

Mockito works in most cases, but it expects you to write code in complete idiomatic gang-of-four style code. Often you will find yourself needing to test a piece of code which bends around those rules, such as needing to mock a static method. Mockito stays away from byte-code black magic required for these cases. Time to summon the dark wizard - PowerMock!

PowerMock has helpful additional features, such as mocking final methods and static methods that Mockito does not support. It may be beneficial when writing tests involving these types of declarations (e.g., mocking static initializers). In this example, we will mock some static methods of java.lang.Math

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

import java.util.ArrayList;
import java.util.List;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.junit.jupiter.MockitoExtension;

@ExtendWith(MockitoExtension.class)
class MathTest {
    @Test
    void testMax() {
        Math mockMath = mock(Math.class);
        when(mockMath.max(1, 2)).thenReturn(2);

        assertEquals(2, mockMath.max(1, 2));
    }
}

Assertion libraries: AssertJ & Hamcrest

While JUnit's built-in assertions will suffice most of the trivial assertions, there are times when you need something more powerful. AssertJ is an assertion library that provides additional features, like fluent assertions and method chaining, for better readability.

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.within;

import org.junit.jupiter.api.Test;

class AssertJAssertionTest {
    @Test
    void testAssertThat() {
        int actual = 1;
        assertThat(actual).isEqualTo(1);
    }

    @Test
    void testWithin() {
        double actual = 0.99;
        double expected = 1.0;
        double delta = 0.01;
        assertThat(actual).isCloseTo(expected, within(delta));
    }
}

Hamcrest is a library for writing “matchers” of various kinds, including matching on the presence or absence of a particular element, matching against the text content of elements, and matching against the structure of elements. It provides a fluent interface for writing these matches in a way that is more readable than using JUnit's built-in Matchers class.

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.CoreMatchers.equalTo;
import static org.hamcrest.number.IsCloseTo.closeTo;

import org.junit.jupiter.api.Test;

class HamcrestAssertionTest {
    @Test
    void testIs() {
        int actual = 1;
        assertThat(actual, is(1));
    }

    @Test
    void testEqualTo() {
        String actual = "hello";
        assertThat(actual, equalTo("hello"));
    }

    @Test
    void testCloseTo() {
        double actual = 0.99;
        double expected = 1.0;
        double delta = 0.01;
        assertThat(actual, closeTo(expected, delta));
    }
}

Data faking: Instancio & DataFaker

While writing, your tests need a lot of fake data. You can hand-roll the objects, but it becomes tiresome rather quickly. Instancio is an open-source library that allows you to create random instances of your classes without writing a single line of code.

public record User(String firstName, String lastName, String email) {}

// use Instantio to create a list of 10 users
List<User> user = Instancio.ofList(User.class).size(10).create();

If you need real-looking data for populating a database or testing validations, use DataFaker. It generates real-looking data for unit tests, such as names and addresses or bank account numbers. It can also generate random data for integration tests.

import net.datafaker.Faker;

// ...

Faker faker = new Faker();

String name = faker.name().fullName(); // Miss Samanta Schmidt
String firstName = faker.name().firstName(); // Emory
String lastName = faker.name().lastName(); // Barton

String streetAddress = faker.address().streetAddress(); // 60018 Sawayn Brooks Suite 449

Web service mocking: WireMock

WireMock is a Java library that allows you to quickly mock, stub and validate web services. It helps you test the interaction between your application and the remote services it depends on when deployed. Using WireMock, you can define HTTP routes and test HTTP calls with their expected request bodies and mock responses.

import static com.github.tomakehurst.wiremock.client.WireMock.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
import com.github.tomakehurst.wiremock.WireMockServer;
import com.github.tomakehurst.wiremock.core.WireMockConfiguration;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;

public class ClientServiceTest {
    private static WireMockServer wireMockServer;

    @BeforeAll
    public static void setup() {
        wireMockServer = new WireMockServer(WireMockConfiguration.options().port(8080));
        wireMockServer.start();
    }

    @AfterAll
    public static void teardown() {
        wireMockServer.stop();
    }

    @Test
    public void testClientService() {
        wireMockServer.stubFor(get(urlEqualTo("/service/endpoint"))
            .willReturn(aResponse().withBody("{\"data\":\"mocked data\"}")));

        ClientService client = new ClientService();
        String response = client.callService();
        assertEquals("mocked data", response);
    }
}

WireMock can simulate slow or unavailable services to test how your application handles them, and you can verify if your code responds correctly in such situations.

Mocking application dependency with TestContainers

TestContainers is a lightweight, open-source library that lets developers quickly mock and test interactions between their code and popular software systems. It's great for testing with real, local dependencies like databases or caches. It also simulates third-party services like Amazon S3 or Google Cloud Storage (using their corresponding local emulators). Here is an example with a PostgreSQL container spun up for use during the test.

import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

@Testcontainers
public class TestContainersExample {
    @Container
    private static final PostgreSQLContainer database = new PostgreSQLContainer();

    @BeforeEach
    public void setup() {
        String jdbcUrl = database.getJdbcUrl();
        // Use jdbcUrl to setup connection to the database container
    }

    @AfterEach
    public void teardown() {
        // Close the database connection
    }

    @Test
    public void testDatabaseOperations() {
        // Perform database operations and assertions
    }
}

TestContainers is built on the Docker API and works with any Dockerfile or image. The library provides ready-made definitions for popular software such as PostgreSQL, MySQL, and Redis and allows you to define your mocks using a simple, declarative syntax. Then it uses these definitions in conjunction with the official docker run command to create new containers that behave like the real thing but provides sandboxed environments for testing purposes.

Unit testing can be a powerful tool for ensuring that your code works correctly, but it also takes time to learn how best to use it. I've covered some great libraries that make unit testing more effortless than ever, and they're free! You don't have any excuse not to try them out today!

This is a conversation space in the Substack app that I set up exclusively for subscribers — like a group chat or live hangout. I’ll post short prompts, thoughts, and updates that come my way, and you can jump into the discussion.

To join our chat, you’ll need to download the Substack app, now available for both iOS and Android. Chats are sent via the app, not email, so turn on push notifications, so you don’t miss a conversation as it happens.

How to get started

1. Download the app by clicking this link or the button below. Substack Chat is now available on both iOS and Android.

Get app

2. Open the app and tap the Chat icon. It looks like two bubbles in the bottom bar, and you’ll see a row for my chat inside.

3. That’s it! Jump into my thread to say hi, and if you have any issues, check out Substack’s FAQ.

Join chat

My (other) stack!

For a while, I have started recording my rides on Strava (a platform for cyclists) to get the most out of my training sessions. Initially, I started with an Apple Watch (Series 7, LTE model). The LTE connectivity allowed maps to work and helped to not take the phone out on every other turn to look at the directions. It worked fine until I started doing long rides (4 - 5h) and discovered that the watch’s battery life was not enough. The Apple Watch only provided GPS and heart-rate data for my rides. However, I wanted more data - such as speed, power & cadence, and heart rate. I bought a Garmin Edge 520 Plus Bundle with Speed/Cadence sensor and a chest-strap heart-rate monitor.

The Garmin Edge 520 Plus is a fabulous bike computer. It has a built-in GPS with navigation and interfaces with BLE and ANT+ sensors, so you don’t need to worry about bringing along your smartphone. It can upload rides directly to Strava, which I use for tracking my progress and sharing my rides with friends. It also interacts with dozens of interesting apps, such as Kumoot and Trailforks, which help plan and share routes.

The heart rate monitor makes it easy for me to keep an eye on how hard I work during each ride, ensuring that I’m not overdoing or getting lazy. I also use the power/cadence sensors to get more accurate information about my training performance!

Since I discovered the sensors run on Bluetooth Low Energy, I looked for documentation of their working and, to much joy, found out that entire specs are standardized and documented. I wanted to get creative to build beautiful charts and have fun with React hooks!

A primer of Bluetooth LE and Web Bluetooth

You have used Bluetooth Headphones and maybe Bluetooth Keyboard and Mice. Bluetooth LE (BLE), similar to Bluetooth, is a low-power wireless technology that allows heart rate monitors and power/cadence sensors to communicate with your phone. BLE devices require very little energy - literally! The cycling sensors run on CR2032 coin cell batteries for years.

Web Bluetooth is a new browser API that allows web pages to communicate directly with Bluetooth LE devices. That means you can use your browser to pair your heart rate monitor or bike computer! However, support for the API is currently limited to Google Chrome and other Chromium derivatives on all desktops and Android.

How BLE devices work

When your device searches for BLE devices, it discovers nearby devices and their advertised services using the Generic Application Profile protocol.

Once you initiate pairing, you can discover what “services” the devices expose. Specifications published by Bluetooth SIG define BLE services. Once you have discovered the services, you can send and receive data via a “characteristic” like an address on the street. The sensor sends data as characteristic values (e.g., heart rate). At the same time, your phone listens for changes in these values over time (e.g., heart rate readings every second). If you are interested to read more, look up the Generic Attribute Profile.

Here’s the technical TLDR; Sensors act as GATT servers, and your devices as GATT central. Servers publish “characteristics” on multiple “services” that your device central continuously listens to. If you want to dig deeper into BLE services and characteristics, the GATT spec in XML is your friend.

Using the Web Bluetooth API

The Web Bluetooth API lets you interact with BLE devices from a web page. It’s pretty bleeding edge and only works in Chrome, but there are plans to support it in other browsers soon.

Check for Web Bluetooth support with:

const bluetoothSupported = 'Bluetooth' in navigator;

Once you have determined you have Bluetooth support available, it’s time to discover some BLE devices and pair them with them. The current shape of Web Bluetooth API allows you to request a single device at once that advertises a list of services, and the browser provides the pairing UI. However, improvements to auto-connect and remembering chosen devices are coming soon. Let’s search for a heart-rate monitor and pair it with it.

async function readDeviceInfo(btDevice: BluetoothDevice): Promise<DeviceInfo> {
  const deviceInfo = await btDevice.gatt!.getPrimaryService(
    "device_information"
  );
  if (!deviceInfo) {
    return {};
  }

  const decoder = new TextDecoder("utf-8");

  const [manufacturer, model] = await Promise.all([
    deviceInfo.getCharacteristic("manufacturer_name_string"),
    deviceInfo.getCharacteristic("model_number_string")
  ]);

  return {
    manufacturer: manufacturer
      ? decoder.decode(await manufacturer.readValue())
      : undefined,
    model: model ? decoder.decode(await model.readValue()) : undefined
  };
}


const btDevice = await navigator.bluetooth.requestDevice({
  filters: [
    { services: ["heart_rate"] }
  ],
  optionalServices: ["device_information"]
});

const server = await btDevice.gatt!.connect();

const [services, deviceInfo] = await Promise.all([
  server.getPrimaryServices(),
  readDeviceInfo(btDevice)
]);

The readDeviceInfo function decodes the device_info characteristic and returns basic info such as BLE device manufacturer, device name and model number. When requesting scanning devices, you can specify a string such as heart_rate or its corresponding UUID 0000180d-0000-1000-8000-00805f9b34fb. The GATT XML specifications contain the values and the corresponding UUIDs in the descriptors.

Once we have discovered the services, we can read the characteristics and display them on the screen.

const service = await server.getPrimaryService("heart_rate");
const characteristic = await service.getCharacteristic("heart_rate_measurement");

await characteristic.startNotifications();

characteristic.addEventListener("characteristicvaluechanged", (event: Event) => {
  // @ts-ignore
  const value = event.target!.value as DataView;
  const flags = value.getUint8(0);
  const result: HeartRateMonitorValue = {
    heartRate: 0,
  };

  let index = 1;

  const rate16Bits = flags & 0x1;
  if (rate16Bits) {
    result.heartRate = value.getUint16(index, true);
    index += 2;
  } else {
    result.heartRate = value.getUint8(index);
    index += 1;
  }
  const contactDetected = flags & 0x2;
  const contactSensorPresent = flags & 0x4;
  if (contactSensorPresent) {
    result.contactDetected = !!contactDetected;
  }
  const energyPresent = flags & 0x8;
  if (energyPresent) {
    result.energyExpended = value.getUint16(index, true);
    index += 2;
  }

  document.querySelector('#heartRate').innerHTML = result;
});

The heart_rate_meaturement characteristic emits 2 or 3 octets for each measurement. For the uninformed, one octet is 8 bits or 1 byte in popular convention, but historically, bytes had different lengths. The first octet is a flag, the first bit of which says whether the heart rate value is 1-bit or 16-bit (human values can fit in 8-bit, but what if you want to measure the heart rate of your cat?). The preceding two bits represent whether the sensor is in contact and whether it emits calories consumed as a value. Depending on the flag values, we read the heart rate as an 8-bit or 16-bit value and energy expended as a 16-bit value if present.

Similarly, we can also read cadence (rotation per minute) values from speed and cadence sensors (CSC). However, interpreting these sensor values involves a bit more calculation. Instead of providing the raw values like the heart rate monitor, the CSC sensors provide wheel rotations and crank rotations along with the last rotation time. Using these, we have to derive speed and cadence.

const service = await server.getPrimaryService("cycling_speed_and_cadence");
const characteristic = await service.getCharacteristic("csc_measurement");

await characteristic.startNotifications();

const TYRE_CIRCUMFERENCE_METRES = 2.155; // I ride a 32c tyr
const MAX_UINT16 = 65535;
const CSC_TIME_FACTOR = 1024;

let lastWheelRevolutions = 0;
let lastWheelEventTime = 0;
let lastCrackRevolutions = 0;
let lastCrankEventTime = 0;

characteristic.addEventListener("characteristicvaluechanged", (event: Event) => {
  const value = event.target!.value as DataView;
  const flags = value.getUint8(0);

  let index = 1;

  const wheelRevolutionDataPresent = flags & 0x1;
  if (wheelRevolutionDataPresent) {
    const cumulativeWheelRevolutions = value.getUint32(index, true);
    index += 4;
    const wheelEventTime = value.getUint16(index, true);
    index += 2;

    const speed =
      (((lastWheelRevolutions - cumulativeWheelRevolutions) *
        TYRE_CIRCUMFERENCE_METRES) /
        diff(
          lastWheelEventTime,
          wheelEventTime,
          MAX_UINT16
        )) *
      CSC_TIME_FACTOR;

    document.querySelector('#speed').innerHTML = speed;
    lastWheelRevolutions = cumulativeWheelRevolutions;
    lastWheelEventTime = wheelEventTime;
  }

  const crankRevolutionDataPresent = flags & 0x2;
  if (crankRevolutionDataPresent) {
    const cumulativeCrankRevolutions = value.getUint16(index, true);
    index += 2;
    const crankEventTime = value.getUint16(index, true);
    index += 2;

    const crankCadence =
      (diff(
        lastCrackRevolutions,
        cumulativeCrankRevolutions,
        MAX_UINT16
      ) /
        diff(
          lastCrankEventTime,
          crankEventTime,
          MAX_UINT16
        )) *
      CSC_TIME_FACTOR;

    document.querySelector('#cadence').innerHTML = crankCadence;
    lastCrackRevolutions = cumulativeCrankRevolutions;
    lastCrankEventTime = crankEventTime;
  }
});

Similar to heart_rate_monitor characteristics, the cycling_speed_and_cadence also emits a flag value that says whether or not wheel rotation data and crank rotation data are present. Many sensors (especially frame-integrated magnetic ones) have a single unit for wheel and crank revolution data. Still, some units, like Garmin, have separate wheel (speed) and crank (cadence) sensors, which connect individually.

If the wheel revolution data is present, we subtract the cumulative value as a 32-bit integer from the last value and multiply it by the tyre circumference to find the distance travelled. The difference in timestamps gives the time and thus speed (= distance/time). Note that the time is also a 16-bit value representing 1/1024 of a second and will overflow quickly. Similarly, the crank rotation value is also a 16-bit value (why?) along with a 16-bit timestamp which provides the current RPM (= rotations/time).

There is a separate diff function to calculate the difference between 16-bit values with consideration for overflow. Here it goes:

function diff(last: number, current: number, overflow: number) {
  const diff = current - last;
  if (diff < 0) {
    return diff + overflow;
  }
  return diff;
}

It’s a simple function that checks if the last number is less than the current. If yes, it assumes an overflow and adds the overflow value to the difference.

Putting it all together

Putting all this as multiple React Hooks with a tiny front end gave me an excellent little bike computer that runs as a web app! It doesn’t have all the features that my Garmin has, but hey, it works, and with access to JS and raw data, we can go as creative as we want. Power meter support would be a relatively simple addition. Support for recording and exporting rides and showing multiple graphs at the end of the ride would be fantastic as well. Turn-by-turn GPS Navigation? Difficult, but still doable with OpenStreetMaps!

The online version is available at bike-computer.pages.dev, and the code is at github.com/recrsn/bike-computer.

I started this project as a playground for experimenting with HRM and CSC BLE characteristics, and it turned out well. The next step is to use iOS CoreBluetooth or Dart and build a fully functioning mobile app using the lessons learnt here. But this is a showcase of the modern web platform.

If you find yourself wondering whether your last change will kill your servers or receive extreme bashing from your users, you need feature flags.

Feature flags are a great way to be fast and safe while developing software. They can be used for many things, including A/B testing, canary releases or staged rollouts. You can have your code do one thing for some customers and another for others; you can keep track of how well things are working out; and take appropriate action if something goes wrong.

Feature flagging is an approach to continuous delivery in which we release features by modifying a system configuration rather than changing code. Rather than release all the features simultaneously, you can release specific features to specific users in different environments.

Feature flags are configurable parameters that function as gatekeepers for enabling or disabling one or more features based on some condition that can be static or dynamic. They’re ideal for managing user experience or feature rollouts since they allow you to control when new features are activated or deactivated. Combining them with product analytics and error monitoring, you can also monitor how users interact with them before starting them for everyone.

def get_homepage_content(request):
    if feature_flags.enabled('new_homepage', request):
        return new_homepage(request.user)
    else:
        return homepage_content(request.user)

How are Feature Flags helpful to you?

Feature flags are a tool that can help you deliver complex products faster. They provide safety, agility, release control, and experimentation support.

Safety: Being able to turn off things that aren’t working immediately. The ability to scale up safely.

Turning off things that aren’t working immediately is an obvious benefit, but it’s not the only one. You can also increase your team’s velocity by enabling them to implement changes faster without worrying whether those changes will ruin everything else. When you’re using feature flags and can scale down quickly, you don’t have to worry about removing unpopular features from your codebase if they don’t work out. And if there are any bugs or issues with a new piece of code or feature, it’s easy to disable it while you resolve them instead of having your whole system go down when something breaks unexpectedly at runtime!

Agility and Release Control - release your new features faster while not affecting the functionality of your current features.

You can now safely test new features without affecting the functionality of existing features. You can release features to a small subset of users and see how they work. If you find something that does not work as expected, you can quickly shut it off for all users or just for that subset.

You don’t need to code features that you want to experiment with in advance or choose between deploying sooner but potentially risking downtime or delaying your deployment and allowing other teams’ work to back up behind you. You can test new features in an isolated environment and then release them as soon as they’re ready—without having to plan for it months in advance.

Feature flags work exceedingly well with the CI/CD philosophy, which empowers users to deploy code once it’s merged with the mainline and passes all acceptance tests by toggling off all the new changes. Once all the features pass QA, you can turn them on.

Experimentation: determine whether a new or changed feature works as intended and whether it should be a part of the general release.

Another benefit of feature flags is experimenting with a new or revised feature and determining whether it works as intended. If you don’t need the new or revised feature for the general release, you can turn it off entirely and avoid any negative impact on the user experience.

In addition, if there are features that work well but aren’t making their way into your general releases, this allows you to test out new ideas and see if they are helpful before deploying them widely. And finally, by getting feedback from users who opt into these experiments, you can get a sense of what they want to help shape future updates.

• A/B Testing: You might think you don’t need feature flags for this—you’re just going to run a couple of experiments simultaneously. But what if you wanted to run a test where not all customers see the same variation? Or what if you wanted to measure the impact of each variation independently? Feature flags let you do both of these things quickly!

• Canary Releases: Canary releases are a great way to get early feedback on new software. You can use feature flags to enable the new code for only part of your customer base and then gradually roll it out further as you collect data and ensure everything works as expected. 

• Staged Rollouts: This is similar to A/B testing, except that instead of measuring how different groups respond differently to the same thing, you measure how the same group reacts differently over time.

Implementing feature flags

Classifying feature flags

We can classify feature toggles into multiple classes based on their longevity and dynamism. Based on this, we can bin them into

1. Release flags - set during build time to enable/disable code sections.

2. Experiment flags - enable A/B testing.

3. Ops flags - the ability to turn features off if something goes wrong.

4. Permission toggles - allow certain features for a subset of users.

Release flags

They are the least dynamic of the bunch, and you set them during build time. You can use it to disable features that need to be conditioned or fail a late-stage QA review—often used by teams following trunk-based development to exclude changes that are not yet ready.

You can implement release flags by language/build tool features such as #IFDEF macros in C++ to include and exclude code.

A famous use of release flags is noticeable in APK teardowns, where insights into strings disabled in application code give a sneak peek of what’s coming in future versions but is not currently ready for prime-time.

Experiment flags

Used for A/B testing, enable a feature based on which cohort a user belongs to. You can serve two versions of your code to different cohorts of users and observe the performance of the versions. Since the flags depend on some properties of the current user, you decide the feature flag value on runtime. Typically, you call an API with user details, which gives the flag’s value. There are several libraries and third-party services that help implement feature flags.

Ops Flags

These flags control operational aspects of our code. You may consider an ops flag when rolling out a new feature that risks breaking a critical flow or has unclear performance implications. If the feature misbehaves, developers can quickly disable the recent changes using these flags.

Such flags are designed to be relatively short-lived - once we achieve confidence in the new change, we remove the flag. However, systems also have a few “Kill Switches” that allow disabling non-critical system functionality when the system is under unusually high load or dealing with a failing third party such as a payment gateway.

Since ops flags don’t depend on user context, You can implement ops flags by reading them from environment variables or configuration stores such as Consul or Etcd.

Permission toggles

You may want to expose newly introduced features to a small group of users - such as internal users or alpha/beta users for early feedback based on which you may further evolve the product. Such flags are the most dynamic of the bunch are depend on the user’s configuration.

A typical way of implementing this would be to tag features based on their availability and decide to expose them based on whether the user belongs to the group for which we expose this functionality.

Considerations while implementing feature toggles

Separating decision points from decision logic

While developing feature flags, it is increasingly tempting to splatter a bunch of if-else all over your code and control features based on this. It may seem reasonable, but it tightly couples feature flags with features. For example, consider yourself developing many changes for a v2 iteration of a workflow.

def generate_report(query):
    if feature_flags.enabled("v2_release"):
        add_performance_data()

    # ...

    if not feature_flags.enabled("v2_release"):
      # do some deprecated work.

Such code forces you to release everything together or nothing at all. However, if you want to release a subset of features, you can only do so with extensive refactoring. Also, breaking into many small feature flags would be too much to handle.

Instead, a more innovative way would be to introduce another abstraction that does context-specific decisions using feature flags.

Let’s refactor the above code.

# report_features.py
def performance_data_enabled():
    return feature_flags.enabled("v2_release")

def query_source_table():
    return not feature_flags.enabled("v2_release")

# generate_report.py

def generate_report(query):
    if report_features.performance_data_enabled():
        add_performance_data()

    # ...

    If report_features.query_source_table():
      # do some deprecated work

Embracing Inversion of Control

Decision-based strategies like the above work well for simple changes, but adding so many branching points would be a maintenance headache when changes become non-trivial.

Feature flags go well with the Open-Close principle of SOLID practices. Instead of changing existing code, you can develop and switch a new implementation based on feature flags.

class V1ReportGenerator:
    def generate_report():
        # do work

class V2ReportGenerator:
    def generate_report():
        # do more work

def create_report_generator():
    if feature_flags.enabled("v2_release"):
        return V2ReportGenerator()
    else
        return V1ReportGenerator()

Storing feature flag data

1. If possible, prefer storing them alongside your code in your SCM and re-deploy whenever you enable or disable a feature. Managing flags in code gives the same benefit as infrastructure as code and contains a log of what changes you deployed at any time.

2. You can store flags in the app database for more dynamic features. The schema for storing flags can be straightforward – a feature name, value, and additional columns for conditions and audit metadata. You can also use other data stores such as Consul or Etcd.

Managing overrides

There will be cases where you must toggle a flag for a minimal subset of users. Having a proper overriding mechanism can help here.

1. If you are using a database, you can maintain an override table with additional constraints on which you perform a left join to get the value of a feature flag.

2. You can also opt for additional information from the request context - such as an API version or a special HTTP header to control feature flags. Doing this moves the control of flags to the consumer of your service.

There are many different ways to use feature flags, but the main idea is simple: you can use them to roll out new features without breaking your site, which means no more worrying about code deployment and issues with testing. If a component isn’t ready for prime time, flip the switch off until it’s ready.

The concept of feature flags is pretty simple: they allow you to control which users see each feature on any given day or week as it rolls out slowly over time rather than all at once when deployed and live. This means fewer bugs in production because tests are more accessible and faster than ever!

You will find lots and lots of guides about monitoring your applications. However, most of them are just focused on setting up a framework using some integration with some language. What they don’t discuss is what metrics you should be monitoring. This blog post exclusively targets that topic.

For monitoring our micro-services, we can start with the four golden signals, as per the Google SRE book — latency, traffic, errors, and saturation.

Health checks

This is the most basic form of monitoring that you should be implementing for your application. You are most likely already using health checks if you use a load-balancer or Kubernetes with liveness probes or Consul.

Implementing a health check is relatively simple. Your service should respond within a reasonable timeout. You can design a “ping” endpoint for an HTTP service that returns a 200 when called. For other protocols, you can similarly create a ping functionality. This checks the basic reachability of your application. Depending on your code, you may want to return success only when pings to required services are green.

Infrastructure metrics

Your second step should be to monitor infrastructure metrics exported by your platform. Collect and monitor the data collected from the OS, such as CPU utilization, memory utilization, IO utilization, and network utilization. Your cloud provider may provide some data as well. If you are using Docker or Kubernetes, then collect those metrics as well. Your monitoring system would likely have integrations that export this data with little effort. If you are using Prometheus, installing node_exporter will collect data from all machines in your infrastructure. Similarly, the Prometheus Kubernetes operator will fetch metrics from your Kubernetes

Response codes and errors

We want as few errors as possible. So we start counting them. A simple way of tracking these is by utilizing your framework and adding middleware to track requests and responses. For an HTTP micro-service, you may track total requests, success and failures, and the time taken by each request.

Depending on your monitoring system, you will want to assign different labels or tags to indicate the context of each event so that your monitoring system will allow you to drill down stats for each endpoint. Some tags that I frequently find adding are the URI, response status codes to requests, some shard indicators for sharded services (like country code, app version), etc. It depends on your requirements.

Latency and timings

Timing is a crucial indicator of performance — you certainly want to track this. You should monitor your request and response timings across your stack in a fine-grained manner as possible. The most accessible place to start is your application server. Similar to the request/response status monitoring, you can begin with a middleware that logs request and response times. Extend that to any proxies or load-balancers. You have to break down what time is spent on routing decisions, network latency, SSL, etc.

After you have a high-level picture, focus on the time spent by your supporting infrastructure and business logic. If you have some number crunching, you should track those timings separately. If you are hitting a database or another service, track their timings as well.

Database performance metrics

I have come across it multiple times — if something is slow, start checking with your database. Databases are massive beasts that do a lot of time-consuming IO and unfortunately don’t scale that well. However, your database likely exports a ton of data that can be very useful in debugging issues or predicting a future performance regression. Stats like table sizes, query timings, tuples read per query, tuples returned, etc., are some things you should be monitoring. Database monitoring is so vast that it deserves its own post altogether.

Cache metrics

Cache helps to speed up your services by keeping frequently-used data readily available. Monitor the cache size, hit rate, miss rate, evict rate, and cache miss ratio. The goal of a cache is to maximize the number of hits. The miss ratio should be as low as possible without risking the storage of slate data. If it is too high, you should reconfigure your cache and maybe even look at your caching strategy.

Queue metrics

If you are utilizing message queues to process jobs, you must monitor them. Producer and consumer counts, queue produce rates, consumption rates, lag, and failures are critical numbers to track. Setting alerts when any of these metrics are abnormal is also essential. It will provide an early warning to any potential failure.

Additionally, monitor the additional metrics published by your queue software (Kafka or RabbitMQ) related to your queue’s performance.

Business Metrics

Apart from the generic metrics we discussed above, your application will likely have custom metrics that you want to track. Things like the number of completed orders, transactions in your application are application-specific business-specific. These numbers give an assurance about how the overall system is performing. In case of an incident, these metrics also help to assess the real impact.

Such metrics vary on a case-to-case basis. It could be a counter or a gauge, or a timer. You can use the libraries provided by your metrics platform to expose these metrics from your application.

Having good metrics is essential for setting up a reliable architecture. Metrics form the source of truth of service alerts and performance numbers. It gives a direction of what to do next in order to improve, fine-tune and optimize the performance and stability of your services.

In conditional HTTP requests, the result of an HTTP request can be changed by comparing the affected resources with the value of a validator.

REST uses HTTP extensively — HTTP verbs in combination with URLs have special implied meanings and status codes for errors. Conditional requests can be used to validate the content of a cache or determining whether the current document being edited is valid.

Making a conditional HTTP request

Properly configured web servers and CDNs instruct browsers to cache most of the static content to avoid re-downloading big chunks of assets, improving page loading speed.

The easiest way to cache static content is to instruct the browser to cache something with the HTTP Expires header. It directs browsers, on an intermediate proxy (CDN Edge), to assume the downloaded content to be fresh until the given time and use the same when the subsequent identical request arrives.

This approach has some drawbacks, such as difficulty in changing content once cached by browsers, etc. These can be mitigated with well-known cache-busting techniques — the benefits outweigh the problems.

Last-Modified, ETags and 304 Not Modified

One of the ways to compromise between unconditional caching and not caching is validating cached content. By supplying a timestamp with the Last-Modified header, you can instruct browsers to send an If-Modified-Since header with the timestamp of the cached response. If the server has the same content, it responds with a 304 Not modified with an empty body, and the browser proceeds to use content from it’s cache. If the content has changed, the server sends a new body with a new Last-modified value with a 200 OK response and the new body.

Sometimes you don’t want to (or cannot) use timestamps to track the state of a resource. You can use a state-identifier such as a checksum (or anything) in a header called ETag which the browser uses with the If-None-Match header. If the value sent by the browser matches the Etag at the server, then the server sends a 304 Not modified status like before.

Conditional requests in APIs

Most API calls, however, are not cached by default. This is true for calls originating from the browser and server-to-server calls. Every time you repeat a GET request, the server fetches the resource from its data store and transfers the body over the network.

Using the date headers — Expiry, Last-Modified and If-Modified-Since

A good example would be an API GET /recommendations for new article recommendations. Your client polls the endpoint for new items to be available. For every call, your API server fetches content from the database, serializes them to JSON, and returns the result. The client then compares the response with the previous and detects changed items.

The recommendations are generated daily using a batch job at midnight, which means that the article recommendations are valid till midnight. You can attach an Expiry header to indicate that the content is fresh till midnight.

In some cases, you may not be able to determine the expiry in advance, such as the API GET /articles/<id> for returning the content of an article. In this case, you can return a Last-Modified header with the last updated timestamp from your database. For subsequent requests for the same article, the client sends an If-Modified-Since with the timestamp sent by the server earlier. The server responds with a 200 OK or with a response of 304 Not Modified depending on whether the article has been modified or not.

Optimistic Locking with ETags/If-Match or If-Unmodified-Since

Let’s continue with our example. We have an update API PATCH /articles/<id> which can be called simultaneously when multiple editors are working on the same article. Without any optimistic locking, we will have something called a lost-update problem.

1. A and B started editing the same article at 9:00 am. Both see identical contents.

2. A makes some changes and decides to save them at 9:15 am.

3. B takes a little longer and saves at 9:30 am.

Without optimistic locking, the content of B will overwrite the changes of A, and the updates made by A will be lost.

To prevent this, you can use a version number as the ETag or serve the Last-Modified timestamp. While making the PATCH request to update the article, the client also sends If-Match or If-Unmodified-Since headers. The server now checks if there is any modification since the value specified those headers headers. The server either returns a 412 Precondition failed if there has been an update since 9:00 am. The client can decide to fetch the latest value and attempt to merge with the latest changes.

Implementing conditional requests

The client

If you are calling APIs from a browser, you do not have to set up anything special. Set the appropriate header, and the browser will do the heavy lifting for caching. If you need advanced control, you can control the cache behavior using fetch API.

For server-side clients, it a bit complex. You need to configure a store, such as “Redis” to store the response and associated metadata. Before sending an HTTP request, you need to fetch the values for a given URL and add it to the headers. Libraries such as OKHttp contain pluggable providers to easily integrate such a mechanism.

The server

The server must understand these headers and perform the appropriate action. For example, in the GET /article/<id> API described above, your server can return 304 Not modified if the “last-modified” timestamp on the database is older than the timestamp sent an If-Modified-Since header.

To prevent “lost updates”, the server can use a version number as the Etag or use the If-Unmodified-Since header and assert it to be newer than the value in the your database. If the condition is violated, send a 412 Precondition Failed status. The client can then initiate appropriate conflict-resolution action such as displaying a diff and letting the user merge the differing versions of the article.

Conclusion

Leveraging HTTP conditional requests is an often ignored aspect of REST APIs. An adequately designed API can improve performance by reducing unnecessary network traffic. By implementing proper locking semantics, you can ensure APIs perform safe operations. This translates to thousands of dollars of money saved for a moderate-scale service.

In this blog, we will look at building a search system for a web app that indexes various videos. Each video has a title, description, and associated metadata. We want our system to be searchable over video title and description. We will be using SQLAlchemy as our ORM layer for the underlying PostgreSQL database.

PostgreSQL and FTS

PostgreSQL is a very capable RDBMS with batteries included. It comes with full-text functionality baked-in with tsquery and tsvector constructs.

Let’s build a table to store our videos.

CREATE TABLE video ( id PRIMARY KEY BIGSERIAL, title VARCHAR(100) NOT NULL, description TEXT NOT NULL, published_at TIMESTAMP DEFAULT NOW() );

The Video model will look like the following snippet.

class Video(db.Model): id = db.Column(db.Integer(), primary_key=True, autoincrement=True) title = db.Column(db.String(255), nullable=False) description = db.Column(db.Text(), nullable=False) thumbnail_url = db.Column(db.String(2048), nullable=False) published_at = db.Column(db.DateTime(), nullable=False)

To implement a basic search, we can use the SQL LIKE operator.

select id, title, description, published_at FROM video WHERE title LIKE %term% OR description LIKE %term%

To do that in SQLAlchemy,

results = Video.query.filter(Video.title.like(term) | Video.description.like(f'%{term}%')).all()

It’s a simple pattern matching. However, this is a start. For many purposes like username autocomplete, this is enough.

However, they have several shortcomings when we try to build a practical search solution.

• There is no language awareness. It is not practically feasible to search based on synonyms and derived words using patterns.

• It is difficult to provide a ranking of search results, which renders them ineffective when thousands of matches are found.

• They do not support indexes and hence are slow.

Enter tsquery and tsvector

A tsvector is a sorted list of distinct lexemes, which have been normalized to merge different variants of the same word. It is a form optimized for search. It is used to store the target contents in a search system.

A tsquery is a set of lexemes meant to be searched against a group of vectors. They are used to represent search queries and can use boolean operators to combine the match effect.

Let’s make the above search a little more efficient.

select id, title, description, published_at FROM video WHERE to_tsvector(title || ' ' || description) @@ to_tsquery(term)

This does not have a direct SQLAlchemy equivalent, but you can perform full-text searches for a single column using matchfunction.

results = Videos.query.filter(Video.description.match(term)).all()

First, this query concatenates the title and description columns. It then converts the result to a tsvector and matches it against the search term. This gives good results, but given an average-sized database, such ad-hoc searches will not provide acceptable results.

Inverted Indices, TSVector Columns, and Generated Columns

To speed up the search, we need to pre-compute the tsvectors and store them in a gin index. A GiN or Generalized Inverted Index is a type of index used for full-text search. Like the “glossary” pages at the end of the book, an inverted index says which term is present in which documents.

PostgreSQL provides us with a column type called tsvector which we will be using to store our pre-computed search document. We will use generated columns which will automatically populate this column whenever we insert a new row. The contents will be to_tsvector('english', title || ' ' || description)

The english is essential as it enables PostgreSQL to generate a predictable tsvector every time. However, this also means that you will be limited in handling English documents only. To work around this limitation, you will need to build your own indexing mechanism, which is out of the scope of this article.

So our final alter table looks like this:

ALTER TABLE video ADD COLUMN __ts_vector__ tsvector GENERATED ALWAYS AS (to_tsvector('english', title || ' ' || description)) STORED;

SQLAlchemy natively does not support tsvector so we need to create a custom type called TSVector, as shown in the following snippet.

import sqlalchemy as sa from sqlalchemy.dialects.postgresql import TSVECTOR

class TSVector(sa.types.TypeDecorator): impl = TSVECTOR

TypeDecorator is an abstract class with few abstract methods, your IDE and/or linter will warn about unimplemented methods, but you can safely ignore them. Since we will not be manipulating TSVector columns directly from Python, we can leave those undefined.

Now we will use the TSVector type in your Video model.

from sqlalchemy import desc, Index

from .ts_vector import TSVector from ..db import db

class Video(db.Model): id = db.Column(db.Integer(), primary_key=True, autoincrement=True) title = db.Column(db.String(255), nullable=False) description = db.Column(db.Text(), nullable=False) thumbnail_url = db.Column(db.String(2048), nullable=False) published_at = db.Column(db.DateTime(), nullable=False)

    __ts_vector__ = db.Column(TSVector(),db.Computed( "to_tsvector('english', title || ' ' || description)", persisted=True))

    __table_args__ = (Index('ix_video___ts_vector__', __ts_vector__, postgresql_using='gin'),)

We have defined a new column __ts_vector__, auto-generated from the title and description.

We can now use this with the match function to perform full-text searches. By default, it will be ordered by rank (relevance). However, you can add additional ORDER BY clauses as well.

results = Video.query.filter(Video.__ts_vector__.match(term)).all()

PostgreSQL full-text search is quite powerful and is sufficient for many use cases. It also allows further customization, such as weighted columns and keywords. Head over to the documentation to learn more. For more advanced use-cases such as high-concurrency or analytics, you may need to invest in a dedicated search setup such as Elasticsearch or a SaaS such as Algolia.

You can find the demo project using PostgreSQL, SQLAlchemy and implementing full-text search at

recrsn/video-gallery

What is load balancing?

Load balancing, as the name suggests, is distributing work to multiple agents. In our case, we distribute incoming HTTP requests from browsers, apps or other servers between a pool of web servers that can serve this request.

How does load balancing work?

Load balancing works primarily in two ways:

1. Client-side

2. Server-side

Client-side load balancing

In client-side load balancing, we expect the client to distribute requests to each of the servers capable of serving the request. We provide the client with a list of target servers and the client chooses a server by some load balancing algorithm for each request it makes.

The simplest example is a DNS based load-balancing. Most websites (including Medium) use multiple web servers to handle the traffic. Every time you visit a website, the browser asks the DNS servers to provide a list of IP addresses of servers hosting the website’s content. If the DNS responds with multiple servers, the browser chooses one and sends the request. For subsequent requests, the browser would choose another server from the list of servers returned by the DNS earlier.

Client-side load balancing is also used in microservices utilizing a service discovery framework like Consul or a service mesh like Istio.

Advantages

1. No additional infrastructure required

2. Simple to implement

Disadvantages

1. Unreliable, as we do not have control over external clients like browsers

2. There will be an uneven load distribution

Server-side load balancing

In server-side load balancing, we run an additional server called a “Load Balancer”, a reverse-proxy that is responsible for distributing requests among application servers that can handle the request. Application servers register themselves with the load-balancer. Client requests first reach the load balancer and it routes to one of the registered application servers based on its configuration. Server-side load balancing is much more common. We use servers such as NGINX, Envoy, HA Proxy or services like Amazon ELB, Cloudflare etc for load balancing.

Advantages

1. More control over load balancing

Disadvantages

1. Additional infrastructure requires cost and maintenance

Load balancing algorithms

A load-balancing algorithm determines how requests are distributed among the origin servers.

Round-robin

This is the simplest load balancing algorithm. It assigns requests to servers in a pool one after the other. For example, in a pool of three servers, the first request will be sent to upstream 1, second to upstream 2, third to upstream 3 and again fourth to upstream 1 and so on.

Least Connection

This algorithm assigns requests based on which server has the least amount of active connections (load).

Weighted

This is a variant of any of the above load balancing algorithms with a slight modification. Every server known to the load balancer now has a weight associated with it. This weight can be based on how close or how powerful the application server is to the load balancer etc. The higher the weight, the higher number of requests get assigned to the origin server.

Adaptive

This algorithm adapts its routing based on metrics provided by the application servers usually involve monitoring response times or some form of telemetry from the application servers.

Sticky

Sometimes (esp legacy web-apps) web application maintains some server-side local state in the form of sessions. This forces you to route the subsequent requests to the server that handled the first request by the client. Such routing is called sticky load balancing. Avoid such architecture, it does not scale well.

This was a small intro about load balancing. The best way to learn is to spin-up a few servers and experiment. You can use ready-to-spin docker containers or virtual machines to deploy a copy of an app and a load-balancer and watch it in action.

If you liked this post, subscribe to my newsletter to never miss the next interesting round of tech stories.

Before you ask “I thought you are a backend developer in working primarily on Java, how come you did not use Hibernate?” Yes, I am a Java developer. I write Java code for a living daily and no, I don’t use Hibernate often. I create REST APIs as the part of my daily job, but I don’t use Spring or Hibernate often. My Company uses JDBi extensively so I never had the chance to fully learn how Hibernate works. I’m well aware of how Postgres and JDBC work, so on the first day when I jumped ship to learn Hibernate for fun, I was up for a shock!

You need a private/protected default constructor

Hibernate uses reflection to read and write classes to and from database rows. It uses proxy classes to facilitate lazy loading. For this, it needs to instantiate the class using a parameter-less constructor which must be protected or private. Without a parameterless constructor, you will meet a weird error message. IDEA has a warning, if enabled, will help you prevent these mistakes.

Quoting table names

Create an Entity called User, connect it with a Spring DaoUserDetailsProvider watch the world burn.

Because apparently, Hibernate doesn’t quote the table names while issuing queries when you try authenticating to your Spring application, your code will fail with org.hibernate.exception.SQLGrammarException: could not extract ResultSet. This means that your database gave a reply Hibernate didn’t expect. If you dig down the stack trace you will find the root exception — something similar to column user0.id doesn't exist.

This is cryptic indeed but the fix is simple, whenever you use a column or table that uses some SQL restricted keywords, don’t forget to quote it manually. For Postgres we use double quotes(“”). So go ahead a add a @Table("\"user\"") and everything will work!

You have to create your own migrations

Java has tools that will help you to run migrations but you must author them yourself. While I believe this is the safest approach as framework generated code may be sub-optimal, devoid of aesthetics and sometimes outright wrong (more on this later). If you have a dozen of new entities with dozen of new fields, authoring migrations manually quickly becomes a hassle.

There is no fix for it. Playing around with JPA properties and forcing Hibernate to dump the crude SQL to a file is a good start, you have to modify them to suite to your tastes. Add the following code to automatically dump what Hibernate is thinking to execute to update your schema. This file serves as a good starting point, and you will definitely want to review/edit these files and put it with a migration library such as Flyway.

spring.jpa.properties.javax.persistence.schema-generation.scripts.create-target=src/main/resources/db/migrations/schema.sql spring.jpa.properties.javax.persistence.schema-generation.scripts.action=update

Understand how two-way relationships behave

If you are using a bidirectional one-to-many/many-to-one relationship to model your data, you must be careful. A bi-directional relationship is traditionally modeled as a field on the one-to-many side — called the owning side. The model with many-to-one relationship is called inverse-side. However, the database only stores data at one of the sides and the other side is populated by the ORM layer. Making changes on the owning side will not automatically make changes in the inverse side until you flush the data.

Database ID generation is different

Hibernate uses its own ID generator by default which uses a value from a table and computes IDs for each entity you insert. This means that your ID ranges will not be sequential or continuous. This is not an issue that affect applications (I use a UUID anyway), but is surprising. You can make hibernate to use your database’s native method of generating IDs though by specifying a different GenerationStrategy.

Automatic code-generation for JPA repositories

For JPA repositories, Hibernate can automatically generate SQL for you just from the function name and type parameters.

For example, if you have a method called Article findByCreatedBy(User u) Hibernate will generate a query like, SELECT * FROM article WHERE article.created_by_id = :user_id

But, as with any smart system that does things “automagically”, it may generate sub-par queries. Make sure to enable query logging to see what queries does Hibernate generate for you. If it is very suboptimal, write queries manually.

These are the few things that surprised me when I started using Hibernate. Hope this helps to remove some “surprises” from your experience with it.