Skip to main content

need: for when you just need simple build dependencies

Comments

Sometimes I need to express a simple dependency between files. For example, if I have a directory full of images, I want to generate a thumbnail for each image, and only regenerate the thumbnails that are out of date.

just is great as a generic command runner. I frequently set up a justfile with common targets like test, format, and clean.

But just doesn't track file dependencies. I could write a recipe that loops over all the images:

thumbs:
    for f in images/*.jpg; do magick "$f" -thumbnail 200x200 "thumbs/${f#images/}"; done

clean:
    rm -rf thumbs

This regenerates every thumbnail each time I run just thumbs, even if most of them haven't changed.

Why not Make?

Make can track those file dependencies:

.PHONY: all-thumbs clean
all-thumbs: $(patsubst images/%.jpg,thumbs/%.jpg,$(wildcard images/*.jpg))

thumbs/%.jpg: images/%.jpg | thumbs
    magick $< -thumbnail 200x200 $@

thumbs:
    mkdir -p thumbs

clean:
    rm -rf thumbs

I've always found Make annoying for this kind of thing.

  1. You have to mark command-like targets such as clean and all-thumbs as phony, or Make may mistake a file with that name for an up-to-date target.
  2. You have to create the output directory yourself.
  3. Recipes need literal tabs. I've lost far too much time because a Makefile was missing one.

Why you need need

I wrote need to address these annoyances. Need only handles file dependencies. It doesn't try to be a command runner.

In need, I express dependencies between files and give it a recipe for building the outputs from the inputs. need compares file contents, so changing an input makes its outputs stale even if the timestamps are misleading. It also creates output directories automatically.

So I keep my justfile for running commands and add a needfile to express file dependencies. The thumbnail example becomes:

# justfile
thumbs:
    need get -j 'thumbs/%.jpg: images/%.jpg' -- images/*.jpg

clean:
    need clean

# needfile
thumbs/%.jpg: images/%.jpg
  magick {{in}} -thumbnail 200x200 {{out}}

Here, the shell expands images/*.jpg into the input filenames. need get maps each images/name.jpg to thumbs/name.jpg, then builds those targets. The -j option lets independent thumbnails build in parallel. If nothing changed, Need has nothing to do.

Need can also publish outputs atomically, so a failed recipe doesn't leave a partially written thumbnail in place. Its dependency signatures use file contents, not timestamps alone. For a bigger example, it can use compiler depfiles to track headers discovered while compiling.

I still use just for commands like test and format. need handles the file artifacts those commands depend on.

need on GitHub

need-tool on crates.io

Comments