The Lost Art of Writing Makefiles

You may come across a Makefile today or you may be writing one yourself. The Makefile will have some targets such as build, test, and install. When invoked, these tasks run a bunch of commands. Sadly, such Makefiles are no better than a task runner. Makefiles are much, much more capable.

Photo by Chester Alvarez on Unsplash

The problem with average Makefiles

With fast disks and CPUs, we do not have a strong need to prevent duplicate work anymore. A webpack build takes seconds, so small that you will be mostly okay with it running over and over.

But what if your tasks are not trivial enough?

You may have seen a Makefile like this. Here, we have a Python project with source code inside awesome_package and tests inside tests. We run pylint on the source files and use pytest to run our tests.

lint: pylint awesome_package

test: pytest tests

This Makefile is no better than having a shell script or an NPM or Pipenv command. Make is (ab)used only for running few commands when make <something> is invoked. Also, this has another glaring defect. If I create a file called lint or test, Make will refuse to run the tasks. Don’t believe me? Try it out. Weird, isn’t it?

No, it isn’t. Turns out Make was explicitly designed to behave like this. Make was developed back in the 70s UNIX days to minimize the amount of work required after any change.

Make uses a concept of target and dependencies. A target requires a set of dependencies that need to be up-to-date. When you invoke make <something>, it builds a dependency graph of the targets it needs to run to execute the given target. However, the targets are expected to be files.

If the target file is absent or the target file’s timestamp is older than the dependencies, the target is executed.

This explains why Make will not execute my test task if I have a file called test. Make finds the file and thinks the task is up-to-date and hence does nothing.

So how is this going to help me?

You can modify the Makefile to use real files as targets and run tasks out of date. Let’s change our example.

awesome_package: awesome_package/__init__.py awesome_package/code.py pylint $? touch awesome_package

tests: awesome_package tests/test_code.py pytest tests touch tests

This says that the target folder awesome_package depends on the files __init__.py and code.py Make compares the timestamps for these files and runs pylint with the changed files only if any of those are changed.

The tests task only runs if test_code.py or any of the files under awesome_package is changed. The touch is there to update the directory timestamp so that it’s newer than the files. This pattern is called “empty target”.

But do we need to specify every file in our project?

No, that would be awful. Make supports variables and can execute shell commands to populate them, which you can use as a list of targets. Here is an updated Makefile that uses the find command to discover the available python files.

SRC=$(shell find awesome_package -type f -name '*.py') TEST_SRC=$(shell find tests -type f -name '*.py')

.PHONY: all lint test

all: lint test

lint: awesome_package

test: tests

awesome_package: $(SRC) pylint $? touch awesome_package

tests: awesome_package $(TEST_SRC) pylint $? pytest tests touch tests

I also have set up something called a phony target. This says that the targets marked .PHONY will always run. Phony is useful for tasks that you want to run regardless of your code state. For example, you may want to have a clean task that deletes all cached files.

If you run this Makefile without any target arguments, it executes the first target called all, which in-turn performs lint and test, which are aliases to the actual filesystem-based targets.

So go ahead, change your Makefiles and stop Make abuse! Makefile is very powerful, and when written correctly, it will shave tons of time from your development work.