Skip to content

upsmon fails to start if the old pid happens to exist for an unrelated process #2463

Closed
#2464
Closed
upsmon fails to start if the old pid happens to exist for an unrelated process#2463
#2464
Labels
impacts-release-2.8.1Issues reported against NUT release 2.8.1 (maybe vanilla or with minor packaging tweaks)portabilityWe want NUT to build and run everywhere possiblequestionservice/daemon start/stopGeneral subject for starting and stopping NUT daemons (drivers, server, monitor); also BG/FG/Debugupsmon

Description

@yoyoma2
yoyoma2
opened on Jun 5, 2024

If a previous upsmon.pid file is present at startup and an unrelated process by chance happens to exist with that pid, upsmon will fail to start with the following message:

Fatal error: A previous upsmon instance is already running!
Either stop the previous instance first, or use the 'reload' command.

Workaround:
My cron job that that checks running services using pidof will now delete upsmon.pid before attempting to restart upsmon.

Expected behavior:
If the name of the current process with the pid contained in upsmon.pid is unrelated to upsmon, then upsmon should start up normally instead of exiting with a failure.

Platform: router running linux

Activity

  1. jimklimov commented on Jun 5, 2024

    @jimklimov
    jimklimov
    on Jun 5, 2024
    Member

    Thanks, that's an interesting case, although the hard part can be looking up the process name for a PID across many platforms that all do it differently.

    Just in case: is the problem happening at router start-up? Does your build of NUT there store the PID files in a persistent storage location, or in a tmpfs that should be empty after a reboot?

    Also, which version of NUT is involved? I think with recent ones, a graceful exit of the daemon should have it delete the PID file.

  2. yoyoma2 commented on Jun 6, 2024

    @yoyoma2
    yoyoma2
    on Jun 6, 2024
    Author

    These are the entware packages installed on the router so a pretty recent version.

    nut - 2.8.1-1
nut-common - 2.8.1-1
nut-upsmon - 2.8.1-1
nut-upssched - 2.8.1-1

    The pid file is in /opt/var/run which is persistent. The /opt/etc/init.d/S15upsmon startup script isn't specifying a variable with the directory for the pid file so not sure how to override this to a non-persistent directory.

    This problem only happened once after a non-graceful reboot of the router. I kept getting emails about upsmon not running so all subsequent attempts to start upsmon were also failing.

    Since the entware startup scripts already perform a check of an already running process (below), perhaps a upsmon command-line argument to skip the redundant test at upsmon startup could be a workaround.

    if [ -n "`pidof $PROC`" ]; then
    echo -e "            $ansi_yellow already running. $ansi_std"
    return 0
fi

    This is admittedly a fringe case that requires bad luck to occur.

  3. jimklimov commented on Jun 6, 2024

    @jimklimov
    jimklimov
    on Jun 6, 2024
    Member

    For temporary files (PID, Unix socket, etc.), NUT uses generally several locations which are built into the binaries by configre script settings, but can be customized by environment variables such as NUT_STATEPATH, NUT_PIDPATH and NUT_ALTPIDPATH. And NUT_CONFPATH for config files, to mention them all. Try exporting these first 3 from the init scripts (consistently for all NUT components you run).

  4. yoyoma2 commented on Jun 6, 2024

    @yoyoma2
    yoyoma2
    on Jun 6, 2024
    Author

    Thanks, those variables are very useful. Tweaking the entware scripts makes installing future entware package updates a pain. If there was a portable fix I would log an issue with entware so all entware NUT users would avoid the risk. Even a non-persistent upsmon.pid isn't a perfect solution.

    For now I'll just keep my checkservices cron job hack which deletes upsmon.pid the next time it runs after the problem happens and notifies me. I might get lucky and never hit this issue again.

    This is a apparently a bigger topic than this NUT issue.

  5. jimklimov commented on Jun 7, 2024

    @jimklimov
    jimklimov
    on Jun 7, 2024
    Member

    Well, in case of NUT the PID files are also important for inter-process communications of sorts, such as sending signals to already-running daemons (e.g. to upsmon -c reload live). If a random process lives at that PID, for a majority of NUT programs that run as an unprivileged user like ups or nut it should end up as just inability to send the signal due to permissions. In this, upsmon may be special as it typically starts as root (so there's a piece with permission to shut down the host) and privileges are dropped for the majority of the program.

  6. jimklimov
    added
    service/daemon start/stopGeneral subject for starting and stopping NUT daemons (drivers, server, monitor); also BG/FG/Debug
    upsmon
    portabilityWe want NUT to build and run everywhere possible
    impacts-release-2.8.1Issues reported against NUT release 2.8.1 (maybe vanilla or with minor packaging tweaks)
    question
    on Jun 7, 2024

  7. yoyoma2 commented on Jun 7, 2024

    @yoyoma2
    yoyoma2
    on Jun 7, 2024
    Author

    In the entware distribution of upsmon the -p argument is the only one used so everything is "all root all the time" so sending signals to a a random daemon will succeed. My router only has root so that's a rare case where -p makes sense.

    Could an argument tell upsmon to run pidof from the OS to double check the PID rather than only consulting the PID file? The entware startup scripts use pidof before launching anything so they already assume all the platforms they support have pidof so entware might consider using such an option especially since using -p.

    Just brainstorming crazy ideas on how to never confuse a random process for a NUT program. I'm no expert...

  8. jimklimov commented on Jun 7, 2024

    @jimklimov
    jimklimov
    on Jun 7, 2024
    Member

    Adding a dependency on a random program is not a likely way forward. I'm tinkering to check with how ps programs on differrnt known OSes do it - so if we can get a name, check it...

  9. 37 remaining items

  10. jimklimov
    added 6 commits that reference this issue on Jul 22, 2024

    include/common.h, common/common.c: refactor apart compareprocname() f…

    c7d5027

    include/common.h, common/common.c: refactor apart checkprocname_ignor…

    d20cce4

    common/common.c: refactor sendsignalpid() to check checkprocname_igno…

    f46ea1a

    common/common.c: compareprocname(): fix basename comparison (copy-pas…

    0f6aff1

    common/common.c: compareprocname(): comment about libtool-wrapped dev…

    0c87a5e
  11. Challado
    added 9 commits that reference this issue on Oct 14, 2026

    Introduce getprocname() and checkprocname() for suspicious PID values…

    e8b355f

    Pass "progname" through sendsignal*() methods to verify that we sends…

    d4099e4

    common/common.c: getprocname(): implement support for /proc/NNN/cmdli…

    12e3af1

    common/common.c: getprocname(): implement support for /proc/NNN/stat …

    0889ed3

    common/common.c: getprocname(): implement support for (Open)BSD parsi…

    dc3a2de

    common/common.c: getprocname(): revise printed messages [networkupsto…

    0694063

    common/common.c: getprocname(): implement support for /proc/NNN/psinf…

    a3a3b22

    common/common.c: getprocname(): implement support for WIN32 parsing w…

    3e6e399

    common/common.c: checkprocname(): case networkupstools#5 is not "exac…

    eb41227
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    impacts-release-2.8.1Issues reported against NUT release 2.8.1 (maybe vanilla or with minor packaging tweaks)portabilityWe want NUT to build and run everywhere possiblequestionservice/daemon start/stopGeneral subject for starting and stopping NUT daemons (drivers, server, monitor); also BG/FG/Debugupsmon

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions