45

I am aware of following thread and supposedly an answer to it. Except an answer is not an answer in generic sense. It tells what the problem was in one particular case, but not in general.

My question is: is there a way to debug ordering cycles in a generic way? E.g.: is there a command which will describe the cycle and what links one unit to another?

For example, I have following in journalctl -b (please disregard date, my system has no RTC to sync time with):

Jan 01 00:00:07 host0 systemd[1]: Found ordering cycle on sysinit.target/start Jan 01 00:00:07 host0 systemd[1]: Found dependency on local-fs.target/start Jan 01 00:00:07 host0 systemd[1]: Found dependency on cvol.service/start Jan 01 00:00:07 host0 systemd[1]: Found dependency on basic.target/start Jan 01 00:00:07 host0 systemd[1]: Found dependency on sockets.target/start Jan 01 00:00:07 host0 systemd[1]: Found dependency on dbus.socket/start Jan 01 00:00:07 host0 systemd[1]: Found dependency on sysinit.target/start Jan 01 00:00:07 host0 systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start Jan 01 00:00:07 host0 systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start

where cvol.service (the one that got introduced, and which breaks the cycle) is:

[Unit] Description=Mount Crypto Volume After=boot.mount Before=local-fs.target [Service] Type=oneshot RemainAfterExit=no ExecStart=/usr/bin/cryptsetup open /dev/*** cvol --key-file /boot/*** [Install] WantedBy=home.mount WantedBy=root.mount WantedBy=usr-local.mount

According to journalctl, cvol.service wants basic.service, except that it doesn't, at least not obviously. Is there a command which would demonstrate where this link is derived from? And in general, is there a command, which would find the cycles and show where each link in the cycle originates?

Improve this question

3 Answers 3

Reset to default
46

You can visualise the cycle with the commands systemd-analyze verify , systemd-analyze dot and the GraphViz dot tool:

systemd-analyze verify default.target |& perl -lne 'print $1 if m{Found.*?on\s+([^/]+)}' | xargs --no-run-if-empty systemd-analyze dot | dot -Tsvg >cycle.svg

You should see something like this:

enter image description here

Here you can see the cycle: c.service->b.service->a.service->c.service

Color legend: black = Requires dark blue = Requisite dark grey = Wants red = Conflicts green = After

Links:

Improve this answer
7
  • systemd-analyze verify does not exist here on a debian 8 install. Commented Oct 4, 2016 at 11:30
  • @sjas, systemd-analyze verify available since v216. try systemd-verify. Does it exist? Commented Oct 4, 2016 at 13:23
  • hm, it does not exist on Jessie: anonscm.debian.org/cgit/pkg-systemd/systemd.git/tree/debian/… Commented Oct 4, 2016 at 13:32
  • 3
    systemd-analyze verify default.target on its own does a decent job in showing the loop... Commented Apr 10, 2017 at 12:01
  • A better option for dot dot -Tx11. So you don't have to open a separate viewer. Commented Apr 1, 2023 at 14:26
28

Is there a command which would demonstrate where this link is derived from?

The closest you can do is systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After cvol.service, which will show the resulting (effective) dependency lists for a given unit.

is there a command, which would find the cycles and show where each link in the cycle originates?

To my knowledge, there is no such command. Actually systemd offers nothing to aid in debugging ordering cycles (sigh).

According to journalctl, cvol.service wants basic.service, except that it doesn't, at least not obviously.

First, the requirement dependencies (Wants=, Requires=, BindsTo= etc.) are independent of ordering dependencies (Before= and After=). What you see here is an ordering dependency cycle, i. e. it has nothing to do with Wants= etc.

Second, there is a number of "default dependencies" created between units of certain types. They are controlled by DefaultDependencies= directive in the [Unit] section (which is enabled by default).

In particular, unless this directive is explicitly disabled, any .service-type unit gets implicit Requires=basic.target and After=basic.target dependencies, which is exactly what you see. This is documented in systemd.service(5).

Improve this answer
3
  • Command you quoted worked perfectly, and indeed revealed dependency on basic.target. It's a shame that the toolset for systemctl is so lacking, but oh well, it's a new project Commented Apr 1, 2015 at 18:00
  • 2
    @galets: Judging from my experience, there are very few examples of such shortage... Maybe someday I'll get around increasing verbosity of the cycle reporter, adding some useful information to the log. Meanwhile, actually, you may use systemd-analyze verify UNIT to check unit's correctness. Behind the scenes, this command creates a virtual systemd instance and tries to load given UNIT as the initial transaction (as if it was default.target). This will not reveal any new information (compared to logs), but at least you won't have to reboot with the unit enabled to see if it fails. Commented Apr 1, 2015 at 23:12
  • systemd request for enhancement (RFE): increase verbosity of the cycle reporter Commented Jul 29, 2016 at 21:40
6

step 1: run verification command for default.target

systemd-analyze verify default.target

step 2: observe which service or target mentioned in the message "systemd Breaking ordering cycle by deleting job" and display it's complete dependency list

systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After <service or target name mentioned in the "breaking cycle" message>

step 3: look at "after" and "before" groups inside the service or target file usually defined in 

/lib/systemd/system

and find the services or targets well known to be sequential but are in outbound order for this one.

example: 

dbus.service

is usually marked "after" 

multi-user.target

but "before" 

sockets.target

such dependency could be easily observed by calling

systemctl list-dependencies default.target

however if the file

/lib/systemd/system/dbus.service

contain lines like:

Before=multi-user.target

or

After=sockets.target

or both simultaneously, means dbus.service is defined outbound and it's causing an systemd endless cycle.

the cure is simple - change word "After" to "Before" and vice-versa if necessary.

Improve this answer
1

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.