2

I am trying to create a docker pull in my Makefile using the following script:

ARCH=amd64 IMAGE=k8s.gcr.io/debian-base TAG=v1.0.0 all: docker pull $(shell echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$(TAG)~g"); 

When I run make command, I am able to pull the image:

# make docker pull k8s.gcr.io/debian-base-amd64:v1.0.0; v1.0.0: Pulling from debian-base-amd64 39fafc05754f: Pull complete Digest: sha256:5f25d97ece9076612b64bb551e12f1e39a520176b684e2d663ce1bd53c5d0618 Status: Downloaded newer image for k8s.gcr.io/debian-base-amd64:v1.0.0 

I would like to add a little caveat to it. Where I want to create manifest for multiple arch images with multiple tags, something like:

ARCH=amd64 ppc64le IMAGE=k8s.gcr.io/debian-base export DOCKER_CLI_EXPERIMENTAL=enabled all: for tag in v1.0.0 v1.0.1 ; do \ docker manifest create $(IMAGE):$$tag $(shell echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$$tag~g"); \ done 

This unfortunately fails with the error:

# make for tag in v1.0.0 v1.0.1 ; do \ docker manifest create k8s.gcr.io/debian-base:$tag k8s.gcr.io/debian-base-amd64: k8s.gcr.io/debian-base-ppc64le:; \ done invalid reference format invalid reference format make: *** [Makefile:6: all] Error 1 

If you see above the tag field doesn't get populated properly. Is this possible to do in Makefile? Should I be doing it some other way or some modification required here? TIA.

5
  • What make prints on the standard output is exactly what it passes to the shell. So it is what you want (it is the shell's role to expand $tag). What happens if you copy-paste this output on the command line? Commented Jan 18, 2020 at 6:22
  • @RenaudPacalet Thanks, that helps. If I understand correctly, $tag would need to exported to the shell. Any idea how can I do that as part of Makefile? Btw I get the same output in the shell as well, i.e. the tag is not expanded: # echo $ARCH | sed -e "s~[^ ]*~$IMAGE\-&:$tag~g" k8s.gcr.io/debian-base-amd64: Commented Jan 18, 2020 at 6:38
  • I think you do not understand what I tried to explain. If you copy paste the output of make but put echo before docker you will see that the $tag is correctly expanded. I did and I got two lines. The first one was docker manifest create k8s.gcr.io/debian-base:v1.0.0 k8s.gcr.io/debian-base-amd64: k8s.gcr.io/debian-base-ppc64le: and the second one was docker manifest create k8s.gcr.io/debian-base:v1.0.1 k8s.gcr.io/debian-base-amd64: k8s.gcr.io/debian-base-ppc64le:. Commented Jan 18, 2020 at 12:01
  • 1
    So, I think that your problem comes from these commands that you crafted, not from make. What happens if you try to execute them directly from the command line? Commented Jan 18, 2020 at 12:03
  • Instead of looping at all, it would be simpler give your tags as separate targets of the same rule, and specify them both (all) as prerequisites for the default target. Within the rule's recipe, the automatic variable $@ represents the name of the target being built. Commented Jan 18, 2020 at 15:15

2 Answers 2

2

You appear to have an order-of-execution problem. You expect make to expand the $(shell) function each time execution of the loop in the rule's recipe reaches the point where the $(shell) reference appears lexically, but if you think about it, that cannot work. make hands off each command in the recipe to the shell for it to execute, at which point it is out of make's hands. Therefore, it must (and does) expand make function calls before passing the command to the shell.

The whole for loop in your recipe is and must be a single command for this purpose (that's what the trailing backslashes are about), so the $(shell) function is expanded before the loop variable is ever set. Within that command, $tag expands to nothing.

And aside from execution order, the execution of the code in the $(shell) function happens in its own shell, where the $tag variable wouldn't be set anyway.

There are several good alternatives:

Alternative 1: Get rid of the loop

You have multiple things you want to build. Great! That's right in make's wheelhouse. Let make help you. For example:

# Note: make syntax permits whitespace around the "=" in variable assignments ARCH = amd64 ppc64le IMAGE = k8s.gcr.io/debian-base export DOCKER_CLI_EXPERIMENTAL = enabled TAGS = v1.0.0 v1.0.1 all: $(TAGS) $(TAGS): docker manifest create $(IMAGE):$@ $(shell echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$@~g") .PHONY: $(TAGS) 

This uses the fact that a rule with multiple targets is applied separately for building each of those targets. It does not require an explicit iteration variable because within any make recipe, the automatic variable $@ expands to the name of the target presently being built.

The $(shell) function call herein is still expanded before the command runs, but that is not a problem because the make variables within, including $@, are expanded first.

Alternative 2: Use shell command substitution instead of make's $(shell) function

It's honestly pretty obtuse to use $(shell) inside a recipe unless to intentionally make use of the order of execution properties attending that, because the shell feature on which that make function is modeled is almost always a simpler and more appropriate choice. For example:

ARCH = amd64 ppc64le IMAGE = k8s.gcr.io/debian-base export DOCKER_CLI_EXPERIMENTAL = enabled all: for tag in v1.0.0 v1.0.1 ; do \ docker manifest create $(IMAGE):$$tag $$(echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$$tag~g"); \ done 

After expansions, the command that make passes to the shell in that case is equivalent to*

for tag in v1.0.0 v1.0.1 ; do \ docker manifest create k8s.gcr.io/debian-base:$tag $(echo amd64 ppc64le | sed -e "s~[^ ]*~k8s.gcr.io/debian-base\-&:$tag~g"); \ done 

In shell code, the construct $(any command) is called a "command substitution". The command inside the parentheses is executed, and its standard output is captured and substituted. Using this leaves no question about order of execution.

Alternative 3: both of the above

There's not much more to say than that it comes out like this:

ARCH = amd64 ppc64le IMAGE = k8s.gcr.io/debian-base export DOCKER_CLI_EXPERIMENTAL = enabled TAGS = v1.0.0 v1.0.1 all: $(TAGS) $(TAGS): docker manifest create $(IMAGE):$@ $$(echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:$@~g") .PHONY: $(TAGS) 

And, I guess, that this is the alternative I like best so far. I don't particularly care for make functions generally, as they are a GNU extension, and many of them tend to produce a blurred programming paradigm. Or maybe "a programming paradigm" would be better wording, as I don't usually think of writing makefiles as "programming" per se.

Alternative 4: also improve the command substitution command

sed is a bit overkill for just appending a string to multiple other strings, and its expression syntax is a bit arcane. I'm actually very fond of sed, but for use in a makefile I value clarity very highly. For that reason, something along these lines is probably what I would do myself:

ARCH = amd64 ppc64le IMAGE = k8s.gcr.io/debian-base export DOCKER_CLI_EXPERIMENTAL = enabled TAGS = v1.0.0 v1.0.1 all: $(TAGS) $(TAGS): docker manifest create $(IMAGE):$@ $$(for arch in $(ARCH); do echo "$(IMAGE)-$$arch:$@"; done) .PHONY: $(TAGS) 

* equivalent, but not identical, because make will perform the line joining instead of leaving that for the shell. I present the line-split version instead for easier reading.

Sign up to request clarification or add additional context in comments.

Comments

0

Your Makefile should work fine if you add \ in front of the second $$tag. This is because the contents in $(shell is passed to a shell twice (once with $(shell call and once in docker manifest ... call).

ARCH=amd64 ppc64le IMAGE=k8s.gcr.io/debian-base export DOCKER_CLI_EXPERIMENTAL=enabled all: for tag in v1.0.0 v1.0.1 ; do \ docker manifest create $(IMAGE):$$tag $(shell echo $(ARCH) | sed -e "s~[^ ]*~$(IMAGE)\-&:\$$tag~g"); \ done 

Perhaps letting make to build up names would be simpler?

ARCH=amd64 ppc64le IMAGE=k8s.gcr.io/debian-base all: for tag in v1.0.0 v1.0.1 ; do \ docker manifest create $(IMAGE):$$tag $(foreach a,$(ARCH),$(IMAGE)-$a:$$tag); \ done 

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.