`poweroff` does not work properly when multiple systemd enabled distros are running

[chemwolf6922]

Windows Version

Microsoft Windows [Version 10.0.26310.28018]

WSL Version

2.7.8.0

Are you using WSL 1 or WSL 2?

• WSL 2
• WSL 1

Kernel Version

6.18.33.1-1

Distro Version

Debian (13.0), Ubuntu (22.04.5)

Other Software

No response

Repro Steps

• Open Ubuntu as the other distro.
• Open Debian as the repro distro.
• Edit /etc/systemd/system.conf to modify the following entries:

[Manager]
DefaultTimeoutStopSec=infinity
DefaultTimeoutAbortSec=infinity

This isolates the issue and provides enough time to monitor and collect information after the stuck.

• Apply changes with sudo systemctl daemon-reexec
• run sudo poweroff

Expected Behavior

The Debian distro should terminate immediately. The same behavior when the Ubuntu distro is not running.

Actual Behavior

The Debian distro never terminates.
From the journal, units with their own timeout defined slowly dies out due to kill timeout. For example, the user@1000.service only exits after it's 2 min timeout.

Jun 22 14:16:42 feng-workstation systemd[255]: Finished systemd-exit.service - Exit the Session.
Jun 22 14:16:42 feng-workstation systemd[255]: Reached target exit.target - Exit the Session.
Jun 22 14:16:42 feng-workstation (sd-pam)[257]: pam_unix(systemd-user:session): session closed for user feng
Jun 22 14:16:42 feng-workstation systemd-logind[141]: Removed session c2.
Jun 22 14:18:42 feng-workstation systemd[1]: user@1000.service: Processes still around after final SIGKILL. Entering failed mode.
Jun 22 14:18:42 feng-workstation systemd[1]: user@1000.service: Failed with result 'timeout'.
Jun 22 14:18:42 feng-workstation systemd[1]: Stopped user@1000.service - User Manager for UID 1000.

Units without their own timeout uses the infinity timeout set earlier. And they never exit. For example, the getty@tty1.service.

Diagnostic Logs

The full shutdown journal as far as it gets:

Jun 22 14:16:42 feng-workstation sudo[318]:     feng : TTY=pts/0 ; PWD=/mnt/c/Users/wangfen ; USER=root ; COMMAND=/usr/sbin/poweroff
Jun 22 14:16:42 feng-workstation sudo[318]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jun 22 14:16:42 feng-workstation systemd-logind[141]: The system will power off now!
Jun 22 14:16:42 feng-workstation systemd-logind[141]: System is powering down.
Jun 22 14:16:42 feng-workstation sudo[318]: pam_unix(sudo:session): session closed for user root
Jun 22 14:16:42 feng-workstation systemd[1]: Stopping session-c1.scope - Session c1 of User feng...
Jun 22 14:16:42 feng-workstation systemd[1]: Removed slice system-modprobe.slice - Slice /system/modprobe.
Jun 22 14:16:42 feng-workstation login[245]: pam_unix(login:session): session closed for user feng
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped target graphical.target - Graphical Interface.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped target multi-user.target - Multi-User System.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped target getty.target - Login Prompts.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped target timers.target - Timer Units.
Jun 22 14:16:42 feng-workstation systemd[1]: apt-daily-upgrade.timer: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped apt-daily-upgrade.timer - Daily apt upgrade and clean activities.
Jun 22 14:16:42 feng-workstation systemd[1]: apt-daily.timer: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped apt-daily.timer - Daily apt download activities.
Jun 22 14:16:42 feng-workstation systemd[1]: dpkg-db-backup.timer: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped dpkg-db-backup.timer - Daily dpkg database backup timer.
Jun 22 14:16:42 feng-workstation systemd[1]: e2scrub_all.timer: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped e2scrub_all.timer - Periodic ext4 Online Metadata Check for All Filesystems.
Jun 22 14:16:42 feng-workstation systemd[1]: logrotate.timer: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped logrotate.timer - Daily rotation of log files.
Jun 22 14:16:42 feng-workstation systemd[1]: systemd-tmpfiles-clean.timer: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopping cron.service - Regular background program processing daemon...
Jun 22 14:16:42 feng-workstation systemd[1]: systemd-udev-load-credentials.service: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped systemd-udev-load-credentials.service - Load udev Rules from Credentials.
Jun 22 14:16:42 feng-workstation systemd[1]: cron.service: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped cron.service - Regular background program processing daemon.
Jun 22 14:16:42 feng-workstation systemd[1]: session-c1.scope: Deactivated successfully.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopped session-c1.scope - Session c1 of User feng.
Jun 22 14:16:42 feng-workstation systemd-logind[141]: Session c1 logged out. Waiting for processes to exit.
Jun 22 14:16:42 feng-workstation systemd[1]: Stopping user@1000.service - User Manager for UID 1000...
Jun 22 14:16:42 feng-workstation systemd-logind[141]: Removed session c1.
Jun 22 14:16:42 feng-workstation systemd[255]: Activating special unit exit.target...
Jun 22 14:16:42 feng-workstation systemd[255]: Stopped target default.target - Main User Target.
Jun 22 14:16:42 feng-workstation systemd[255]: Stopped target basic.target - Basic System.
Jun 22 14:16:42 feng-workstation systemd[255]: Stopped target paths.target - Paths.
Jun 22 14:16:42 feng-workstation systemd[255]: Stopped target sockets.target - Sockets.
Jun 22 14:16:42 feng-workstation systemd[255]: Stopped target timers.target - Timers.
Jun 22 14:16:42 feng-workstation systemd[255]: Closed dbus.socket - D-Bus User Message Bus Socket.
Jun 22 14:16:42 feng-workstation systemd[255]: Closed dirmngr.socket - GnuPG network certificate management daemon.
Jun 22 14:16:42 feng-workstation systemd[255]: Closed gpg-agent-browser.socket - GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jun 22 14:16:42 feng-workstation systemd[255]: Closed gpg-agent-extra.socket - GnuPG cryptographic agent and passphrase cache (restricted).
Jun 22 14:16:42 feng-workstation systemd[255]: Stopping gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation)...
Jun 22 14:16:42 feng-workstation systemd[255]: Stopping gpg-agent.socket - GnuPG cryptographic agent and passphrase cache...
Jun 22 14:16:42 feng-workstation systemd[255]: Closed keyboxd.socket - GnuPG public key management service.
Jun 22 14:16:42 feng-workstation systemd[255]: Stopping ssh-agent.socket - OpenSSH Agent socket...
Jun 22 14:16:42 feng-workstation systemd[255]: Closed gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation).
Jun 22 14:16:42 feng-workstation systemd[255]: Closed gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
Jun 22 14:16:42 feng-workstation systemd[255]: Closed ssh-agent.socket - OpenSSH Agent socket.
Jun 22 14:16:42 feng-workstation systemd[255]: Removed slice app.slice - User Application Slice.
Jun 22 14:16:42 feng-workstation systemd[255]: Reached target shutdown.target - Shutdown.
Jun 22 14:16:42 feng-workstation systemd[255]: Finished systemd-exit.service - Exit the Session.
Jun 22 14:16:42 feng-workstation systemd[255]: Reached target exit.target - Exit the Session.
Jun 22 14:16:42 feng-workstation (sd-pam)[257]: pam_unix(systemd-user:session): session closed for user feng
Jun 22 14:16:42 feng-workstation systemd-logind[141]: Removed session c2.
Jun 22 14:18:42 feng-workstation systemd[1]: user@1000.service: Processes still around after final SIGKILL. Entering failed mode.
Jun 22 14:18:42 feng-workstation systemd[1]: user@1000.service: Failed with result 'timeout'.
Jun 22 14:18:42 feng-workstation systemd[1]: Stopped user@1000.service - User Manager for UID 1000.
Jun 22 14:18:42 feng-workstation systemd[1]: Stopping user-runtime-dir@1000.service - User Runtime Directory /run/user/1000...
Jun 22 14:18:43 feng-workstation systemd[1]: mnt-wslg-run-user-1000.mount: Deactivated successfully.
Jun 22 14:18:43 feng-workstation systemd[1]: Unmounted mnt-wslg-run-user-1000.mount - /mnt/wslg/run/user/1000.
Jun 22 14:18:43 feng-workstation systemd[1]: run-user-1000.mount: Deactivated successfully.
Jun 22 14:18:43 feng-workstation systemd[1]: Unmounted run-user-1000.mount - /run/user/1000.
Jun 22 14:18:43 feng-workstation systemd[1]: user-runtime-dir@1000.service: Deactivated successfully.
Jun 22 14:18:43 feng-workstation systemd[1]: Stopped user-runtime-dir@1000.service - User Runtime Directory /run/user/1000.
Jun 22 14:18:43 feng-workstation systemd[1]: Removed slice user-1000.slice - User Slice of UID 1000.
Jun 22 14:18:43 feng-workstation systemd[1]: Stopping dbus.service - D-Bus System Message Bus...
Jun 22 14:18:43 feng-workstation systemd[1]: Stopping systemd-logind.service - User Login Management...
Jun 22 14:18:43 feng-workstation systemd[1]: dbus.service: Deactivated successfully.
Jun 22 14:18:43 feng-workstation systemd[1]: Stopped dbus.service - D-Bus System Message Bus.
Jun 22 14:18:43 feng-workstation systemd[1]: systemd-logind.service: Deactivated successfully.
Jun 22 14:18:43 feng-workstation systemd[1]: Stopped systemd-logind.service - User Login Management.

sudo systemctl list-jobs --no-pager
JOB UNIT                                                             TYPE  STATE
595 networking.service                                               stop  waiting
698 usr-lib-wsl-drivers.mount                                        stop  waiting
620 systemd-remount-fs.service                                       stop  waiting
696 mnt-wslg-distro.mount                                            stop  waiting
588 systemd-user-sessions.service                                    stop  waiting
683 final.target                                                     start waiting
636 swap.target                                                      stop  waiting
701 mnt-d.mount                                                      stop  waiting
627 user.slice                                                       stop  waiting
688 tmp-.X11\x2dunix.mount                                           stop  waiting
571 systemd-poweroff.service                                         start waiting
572 shutdown.target                                                  start waiting
673 systemd-modules-load.service                                     stop  waiting
640 sysinit.target                                                   stop  waiting
677 systemd-tmpfiles-setup-dev-early.service                         stop  waiting
703 run-shm.mount                                                    stop  waiting
591 slices.target                                                    stop  waiting
667 systemd-sysctl.service                                           stop  waiting
691 init.mount                                                       stop  waiting
699 mnt-wslg.mount                                                   stop  waiting
686 umount.target                                                    start waiting
606 systemd-binfmt.service                                           stop  waiting
694 mnt-c.mount                                                      stop  waiting
695 mnt-wslg-doc.mount                                               stop  waiting
704 mnt-wsl.mount                                                    stop  waiting
604 getty@tty1.service                                               stop  running
705 run-lock.mount                                                   stop  waiting
618 system-getty.slice                                               stop  waiting
689 tmp.mount                                                        stop  waiting
570 poweroff.target                                                  start waiting
598 local-fs-pre.target                                              stop  waiting
675 systemd-ask-password-wall.path                                   stop  waiting
590 sockets.target                                                   stop  waiting
679 remote-fs.target                                                 stop  waiting
692 run-user.mount                                                   stop  waiting
654 basic.target                                                     stop  waiting
687 usr-lib-modules-6.18.33.1\x2dmicrosoft\x2dstandard\x2dWSL2.mount stop  waiting
577 systemd-tmpfiles-setup.service                                   stop  waiting
661 systemd-tmpfiles-setup-dev.service                               stop  waiting
697 usr-lib-wsl-lib.mount                                            stop  waiting
615 paths.target                                                     stop  waiting
646 systemd-hostnamed.socket                                         stop  waiting
633 systemd-ask-password-console.path                                stop  waiting
700 mnt-wslg-versions.txt.mount                                      stop  waiting
643 dbus.socket                                                      stop  waiting
678 network.target                                                   stop  waiting
585 local-fs.target                                                  stop  waiting

47 jobs listed.

cat /sys/fs/cgroup/user.slice/user-1000.slice/cgroup.events
populated 1
frozen 0

cat /sys/fs/cgroup/system.slice/system-getty.slice/getty\@tty1.service/cgroup.events
populated 1
frozen 0

Both are populated by the running ubuntu distro.

[github-actions]

Logs are required for review from WSL team

If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'.
Otherwise, please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.

How to collect WSL logs

Download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:

Invoke-WebRequest -UseBasicParsing "https://github.com/ghraw/microsoft/WSL/master/diagnostics/collect-wsl-logs.ps1" -OutFile collect-wsl-logs.ps1
Set-ExecutionPolicy Bypass -Scope Process -Force
.\collect-wsl-logs.ps1

The script will output the path of the log file once done.

If this is a networking issue, please use .\collect-wsl-logs.ps1 -LogProfile networking instead, following the instructions in Collect WSL logs for networking issues

Once completed please upload the output files to this GitHub issue.

See Collect WSL logs (recommended method).

If you choose to email these logs instead of attaching them to the bug, please send them to wsl-gh-logs@microsoft.com with the GitHub issue number in the subject, and include a link to your GitHub issue comment in the message body, and reply with '/emailed-logs'.

[chemwolf6922]

Since the wsl --terminate <distro> command also uses the poweroff command internally. It will always wait for the timeout and fallback to the force shutdown.

[OneBlue]

This seems to be the root cause: Jun 22 14:18:42 feng-workstation systemd[1]: user@1000.service: Processes still around after final SIGKILL. Entering failed mode.

Do we know which processes are still alive and preventing that unit from terminating ? That feels like the right place to start investigating.

[chemwolf6922]

Do we know which processes are still alive and preventing that unit from terminating ? That feels like the right place to start investigating.

Systemd uses the cgroup.events / cgroup.procs in the unit's cgroup to tell if the unit fully exited. Those are indeed populated as shown above. And the process is from the other distro since they use the same cgroup.

[chemwolf6922]

For example, when both distros are running:
In Ubuntu

feng@feng-workstation:/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/init.scope$ cat cgroup.events
populated 1
frozen 0

feng@feng-workstation:/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/init.scope$ cat cgroup.procs
696
697
0
0

feng@feng-workstation:/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/init.scope$ ps 696
    PID TTY      STAT   TIME COMMAND
    696 ?        Ss     0:00 /lib/systemd/systemd --user

feng@feng-workstation:/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/init.scope$ ps 697
    PID TTY      STAT   TIME COMMAND
    697 ?        S      0:00 (sd-pam)

In Debian

feng@feng-workstation:/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/init.scope$ cat cgroup.events
populated 1
frozen 0

feng@feng-workstation:/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/init.scope$ cat cgroup.procs
0
0
236
238

feng@feng-workstation:/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/init.scope$ ps 236
    PID TTY      STAT   TIME COMMAND
    236 ?        Ss     0:00 /usr/lib/systemd/systemd --user
feng@feng-workstation:/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/init.scope$ ps 238
    PID TTY      STAT   TIME COMMAND
    238 ?        S      0:00 (sd-pam)

The Debian distro cannot kill the processes from Ubuntu, but they will mark that cgroup as populated.