What is "subsystem login"?

The subsystem login seems indeed to be a poorly-documented feature. A bit of source code surfing may be needed to figure things out. Let's look in Debian 11's login.c.

The part that deals with a subsystem login starts at line 1151:

if (pwd->pw_shell[0] == '*') { /* subsystem root */ pwd->pw_shell++; /* skip the '*' */ subsystem (pwd); /* figure out what to execute */ subroot = true; /* say I was here again */ endpwent (); /* close all of the file which were */ endgrent (); /* open in the original rooted file */ endspent (); /* system. they will be re-opened */ #ifdef SHADOWGRP endsgent (); /* in the new rooted file system */ #endif goto top; /* go do all this all over again */ } 

If an asterisk is detected at the beginning of the shell field, the asterisk is removed (so you'll never actually see a *-bash in a process listing) and the subsystem() function is called for this user's login. Here pwd is is a structure containing the information that is equivalent to the user's line in /etc/passwd.

When a subsystem login is triggered, all the password and group entry handling is also terminated. From the comments here you should be able to guess that the subsystem login has something to do with chroot().

goto top; jumps back to line #717, effectively restarting the authentication and session set-up process, now using configuration files inside the chroot.

The subsystem() function is defined in sub.c - in fact, that is the only function in that file:

/* * subsystem - change to subsystem root * * A subsystem login is indicated by the presence of a "*" as * the first character of the login shell. The given home * directory will be used as the root of a new filesystem which * the user is actually logged into. */ void subsystem (const struct passwd *pw) { /* * The new root directory must begin with a "/" character. */ if (pw->pw_dir[0] != '/') { printf (_("Invalid root directory '%s'\n"), pw->pw_dir); SYSLOG ((LOG_WARN, BAD_SUBROOT2, pw->pw_dir, pw->pw_name)); closelog (); exit (EXIT_FAILURE); } /* * The directory must be accessible and the current process * must be able to change into it. */ if ( (chdir (pw->pw_dir) != 0) || (chroot (pw->pw_dir) != 0)) { (void) printf (_("Can't change root directory to '%s'\n"), pw->pw_dir); SYSLOG ((LOG_WARN, NO_SUBROOT2, pw->pw_dir, pw->pw_name)); closelog (); exit (EXIT_FAILURE); } } 

And here we have it: a subsystem login is a login that is chroot()ed to the user's home directory as specified in /etc/passwd. For example, if you have user subsys defined like this in /etc/passwd:

subsys:x:999:999:Subsystem login example:/home/subsys:*chrooted* 

When this user logs in using login (i.e. from a text console or via a serial port), the user will be chrooted so that they will see their home directory /home/subsys as /. Then the authentication process is repeated using the configuration files in /home/subsys/etc.

Note that in case of a subsystem login, the contents of the shell field of the actual /etc/passwd after the first * character are effectively ignored. Once the chrooting is done, the login process will read /home/subsys/etc/passwd for the real shell that will be used for this user.

As always when setting up a chroot, you'll have to ensure all the necessary library files, configuration files and devices exist within the chroot, as programs started within the chroot won't be able to access anything outside the chroot. So in this example case, you would have to create at least minimal /home/subsys/lib, /home/subsys/etc (including at least /home/subsys/etc/passwd, /home/subsys/etc/shadow, /home/subsys/etc/pam.d/login and possibly other files too) and usually at least /home/subsys/dev/null.

Depending on what you're using the subsystem login for, you might need other devices too: for shells, you are likely to need /home/subsys/dev/ptmx and the devpts filesystem in /home/subsys/dev/pts. You might set them up like this:

# these steps need only be done once: mkdir -p /home/subsys/dev/pts mknod /home/subsys/dev/null c 1 3 mknod /home/subsys/dev/ptmx c 5 2 chmod 0666 /home/subsys/dev/null /home/subsys/dev/ptmx # this needs to be re-done after every boot: mount -t devpts none /home/subsys/dev/pts 

So your /home/subsys/etc/passwd would need to have an entry for the chrooted subsys user, something like this:

subsys:x:999:999:A subsystem user:/subsyshome:/bin/bash 

This would mean the user's actual home directory would be at /home/subsys/subsyshome, and you would also need to provide /home/subsys/bin/bash and all the libraries needed by it.