05 - Process & Service Management¶

Navigation: ← File Permissions | Index | Next → Package Management

1. Executive Summary¶

Topic Scope: Process monitoring, control, systemd service management, and system targets in RHEL 10

RHCSA Relevance: Critical operational skill - process and service management is essential for system administration

Exam Weight: High - Service management and process control appear in multiple exam scenarios

Prerequisites: Basic understanding of Linux command line and system concepts

Related Topics: Boot Process, Logging, Troubleshooting

2. Conceptual Foundation¶

Core Theory¶

RHEL 10 uses systemd as the init system and service manager, which fundamentally changed how processes and services are managed:

Process hierarchy: All processes descend from PID 1 (systemd)
Service units: Standardized configuration for system services
Targets: Groups of services that define system states
Dependencies: Services can depend on other services or system states
Process control: Signal-based communication for process management

Real-World Applications¶

Web server management: Starting, stopping, and monitoring Apache/Nginx
Database operations: Managing MySQL/PostgreSQL service lifecycle
System maintenance: Controlling system services during maintenance windows
Performance troubleshooting: Identifying resource-intensive processes
Service reliability: Ensuring critical services restart automatically

Common Misconceptions¶

systemctl vs service: Old service command still works but systemctl is preferred
Enable vs start: Services must be both enabled (auto-start) and started (running now)
Process vs service: Not all processes are services; services are managed processes
Kill vs terminate: Different signals have different effects on processes
Targets vs runlevels: systemd targets are more flexible than traditional runlevels

Key Terminology¶

Process: Running instance of a program with unique PID
Service: Long-running process managed by systemd
Unit: systemd configuration object (service, target, socket, etc.)
Target: Group of units that define a system state
Signal: Inter-process communication mechanism
Daemon: Background process providing system services
Job: systemd task for starting/stopping units
Slice: Resource management group for processes

3. Command Mastery¶

Process Monitoring Commands¶

# Basic process listing
ps aux                      # All processes, detailed format
ps -ef                      # All processes, full format
ps -eLf                     # Include threads
ps -o pid,ppid,user,comm    # Custom output format

# Dynamic process monitoring
top                         # Interactive process viewer
htop                        # Enhanced interactive viewer (if installed)
top -u username             # Filter by user
top -p PID                  # Monitor specific process

# Process hierarchy
pstree                      # Process tree view
pstree -p                   # Include PIDs
pstree username             # User's process tree

# Process information
pgrep processname           # Find process IDs by name
pidof processname           # Alternative to pgrep
ps -C processname           # Show processes by command name
lsof -p PID                 # Files opened by process

Process Control Commands¶

# Process signals
kill PID                    # Send TERM signal (graceful termination)
kill -9 PID                 # Send KILL signal (force termination)
kill -HUP PID               # Send HUP signal (often reload config)
kill -USR1 PID              # Send USR1 signal (user-defined)
killall processname         # Kill all processes by name
pkill -u username           # Kill processes by user

# Job control
nohup command &             # Run command immune to hangups
command &                   # Run in background
jobs                        # List active jobs
fg %jobnumber              # Bring job to foreground
bg %jobnumber              # Send job to background
disown %jobnumber          # Remove job from shell's job table

# Process priority
nice -n 10 command          # Start with adjusted priority
renice 5 PID               # Change priority of running process
ionice -c 2 -n 7 PID       # Set I/O priority

systemd Service Management¶

# Service status and control
systemctl status servicename        # Show service status
systemctl start servicename         # Start service
systemctl stop servicename          # Stop service
systemctl restart servicename       # Restart service
systemctl reload servicename        # Reload configuration
systemctl enable servicename        # Enable auto-start at boot
systemctl disable servicename       # Disable auto-start
systemctl mask servicename          # Prevent service from starting
systemctl unmask servicename        # Remove mask

# Service information
systemctl is-active servicename     # Check if running
systemctl is-enabled servicename    # Check if enabled
systemctl is-failed servicename     # Check if failed
systemctl list-units --type=service # List all services
systemctl list-unit-files --type=service # List service files

# Service dependencies
systemctl list-dependencies servicename    # Show dependencies
systemctl list-dependencies --reverse servicename # Reverse dependencies

System Targets¶

# Target management
systemctl get-default               # Show default target
systemctl set-default target       # Set default target
systemctl isolate target           # Switch to target immediately
systemctl list-units --type=target # List active targets

# Common targets
systemctl isolate rescue.target    # Single-user mode
systemctl isolate multi-user.target # Multi-user (no GUI)
systemctl isolate graphical.target  # Multi-user with GUI
systemctl isolate poweroff.target   # Shutdown system
systemctl isolate reboot.target     # Restart system

Advanced systemd Commands¶

# Unit file management
systemctl daemon-reload             # Reload systemd configuration
systemctl edit servicename          # Edit service override
systemctl cat servicename           # Show service unit file
systemctl show servicename          # Show all service properties

# System control
systemctl poweroff                  # Shutdown system
systemctl reboot                    # Restart system
systemctl suspend                   # Suspend to RAM
systemctl hibernate                 # Suspend to disk
systemctl hybrid-sleep              # Suspend to both RAM and disk

Command Reference Table¶

Command	Purpose	Key Options	Example
`ps`	List processes	`aux`, `-ef`, `-C`	`ps aux \\| grep httpd`
`top`	Monitor processes	`-u`, `-p`	`top -u apache`
`kill`	Send signals	`-9`, `-HUP`, `-USR1`	`kill -HUP 1234`
`systemctl`	Manage services	`start`, `stop`, `enable`, `status`	`systemctl enable httpd`
`pgrep`	Find processes	`-u`, `-f`	`pgrep -u root sshd`
`nohup`	Run immune to hangups	`&`	`nohup ./script.sh &`

4. Procedural Workflows¶

Standard Procedure: Service Installation and Configuration¶

Install and enable service

dnf install servicename
systemctl enable servicename
systemctl start servicename

Verify service status

systemctl status servicename
systemctl is-active servicename
systemctl is-enabled servicename

Configure service

# Edit main configuration
vim /etc/servicename/config.conf

# Or create systemd override
systemctl edit servicename

Apply changes

systemctl daemon-reload
systemctl restart servicename
systemctl status servicename

Standard Procedure: Process Troubleshooting¶

Identify problematic process

top                    # Look for high CPU/memory usage
ps aux --sort=-%cpu    # Sort by CPU usage
ps aux --sort=-%mem    # Sort by memory usage

Gather process information

ps -p PID -o pid,ppid,user,cmd
lsof -p PID           # Files opened by process
strace -p PID         # System calls (use carefully)

Take appropriate action

# Graceful termination
kill PID

# Force termination if needed
kill -9 PID

# Or restart associated service
systemctl restart servicename

Decision Tree: Process Management Strategy¶

Process Issue
├── High CPU usage?
│   ├── Expected behavior? → Monitor and document
│   └── Unexpected? → Investigate cause, consider restart
├── High memory usage?
│   ├── Memory leak suspected? → Restart service
│   └── Normal operation? → Check system capacity
├── Process not responding?
│   ├── Try graceful termination → kill PID
│   └── If unsuccessful → kill -9 PID
└── Service keeps failing?
    ├── Check logs → journalctl -u servicename
    ├── Verify configuration
    └── Check dependencies

Standard Procedure: System Target Management¶

Check current target

systemctl get-default
who -r                # Alternative method

Change target temporarily
```
systemctl isolate multi-user.target
```
Change default target
```
systemctl set-default graphical.target
```

Verify target change

systemctl get-default
systemctl list-units --type=target

5. Configuration Deep Dive¶

systemd Unit Files¶

Service Unit Structure¶

# /etc/systemd/system/myservice.service
[Unit]
Description=My Custom Service
After=network.target
Requires=network.target

[Service]
Type=simple
User=myuser
Group=mygroup
ExecStart=/usr/local/bin/myservice
ExecStop=/bin/kill -TERM $MAINPID
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

Common Unit File Sections¶

[Unit]
Description=Service description
Documentation=man:service(8)
After=dependency.service
Before=dependent.service
Requires=hard.dependency
Wants=soft.dependency
Conflicts=conflicting.service

[Service]
Type=simple|forking|oneshot|notify|idle
ExecStart=/path/to/executable
ExecStop=/path/to/stop/command
ExecReload=/path/to/reload/command
User=username
Group=groupname
Restart=no|on-success|on-failure|on-abnormal|on-watchdog|always
RestartSec=seconds

[Install]
WantedBy=target.target
RequiredBy=target.target
Also=other.service

Process Priority and Nice Values¶

Understanding Priority¶

# Nice values range from -20 (highest priority) to 19 (lowest priority)
# Default nice value is 0
# Only root can set negative nice values

# Examples:
nice -n 10 ./cpu-intensive-task     # Lower priority
nice -n -10 ./critical-task         # Higher priority (root only)
renice 5 1234                       # Change running process priority

Resource Control with systemd¶

Systemd Slices and Resource Limits¶

# Create custom slice for resource management
# /etc/systemd/system/myapp.slice
[Unit]
Description=My Application Slice
Before=slices.target

[Slice]
CPUQuota=50%
MemoryLimit=1G
TasksMax=100

Service Resource Limits¶

# In service unit file [Service] section:
CPUQuota=50%                # Limit CPU usage to 50%
MemoryLimit=512M            # Limit memory to 512MB
TasksMax=50                 # Limit number of tasks
IOWeight=100                # I/O priority weight

6. Hands-On Labs¶

Lab 6.1: Process Monitoring and Control (Asghar Ghori Style)¶

Objective: Master process identification, monitoring, and control techniques

Steps:

Start background processes for testing

# Create some test processes
sleep 300 &
PID1=$!
dd if=/dev/zero of=/dev/null &
PID2=$!
find / -name "*.log" > /dev/null 2>&1 &
PID3=$!

echo "Started processes: $PID1 $PID2 $PID3"

Practice process monitoring

# View all processes
ps aux | head -20

# Find specific processes
ps aux | grep sleep
pgrep sleep
pidof sleep

# Monitor resource usage
top -p $PID1,$PID2,$PID3
# Press 'q' to quit top

# View process tree
pstree $$    # Show tree from current shell

Practice process control

# Send different signals
kill -USR1 $PID1        # User signal (sleep will ignore)
kill -STOP $PID2        # Suspend process
kill -CONT $PID2        # Resume process

# Check process status
ps -o pid,state,comm -p $PID1,$PID2,$PID3

# Terminate processes
kill $PID1              # Graceful termination
kill -9 $PID2           # Force termination
killall find            # Kill by name

Practice job control

# Start job in foreground
sleep 100
# Press Ctrl+Z to suspend

# Manage jobs
jobs                    # List jobs
bg %1                   # Send to background
fg %1                   # Bring to foreground
# Press Ctrl+C to terminate

Verification:

# Verify no test processes remain
ps aux | grep -E "(sleep|dd|find)" | grep -v grep
jobs                     # Should show no jobs

Lab 6.2: systemd Service Management (Sander van Vugt Style)¶

Objective: Master systemd service lifecycle and configuration

Steps:

Explore existing services

# List all services
systemctl list-units --type=service --all

# Check specific service status
systemctl status sshd
systemctl is-active sshd
systemctl is-enabled sshd

# View service dependencies
systemctl list-dependencies sshd
systemctl list-dependencies --reverse sshd

Practice service control

# Work with chronyd (time synchronization)
systemctl status chronyd
systemctl stop chronyd
systemctl status chronyd
systemctl start chronyd
systemctl reload chronyd    # If reload is supported
systemctl restart chronyd

Configure service startup

# Check and modify service enablement
systemctl is-enabled chronyd
systemctl disable chronyd
systemctl is-enabled chronyd
systemctl enable chronyd
systemctl is-enabled chronyd

Explore service configuration

# View service unit file
systemctl cat chronyd

# Show all service properties
systemctl show chronyd | head -20

# Create service override (don't actually modify)
systemctl edit chronyd --drop-in=custom
# This would open an editor, but let's skip actual changes

Verification:

# Verify service is properly configured
systemctl status chronyd
systemctl is-active chronyd
systemctl is-enabled chronyd

Lab 6.3: Custom Service Creation (Synthesis Challenge)¶

Objective: Create and manage a custom systemd service

Scenario: Create a custom web log monitor service that watches for specific patterns in web server logs

Requirements:

Service runs as non-root user
Automatically restarts if it fails
Starts after network is available
Can be stopped and started with systemctl

Solution Steps:

Create the monitoring script

sudo mkdir -p /opt/logmonitor
sudo tee /opt/logmonitor/weblog-monitor.sh << 'EOF'
#!/bin/bash
# Simple web log monitor

LOGFILE="/var/log/httpd/access_log"
MONITOR_LOG="/var/log/logmonitor.log"

# Create log file if it doesn't exist
touch "$MONITOR_LOG"

echo "$(date): Web log monitor started" >> "$MONITOR_LOG"

while true; do
    # Monitor for 404 errors (customize as needed)
    if tail -n 1 "$LOGFILE" 2>/dev/null | grep -q " 404 "; then
        echo "$(date): 404 error detected" >> "$MONITOR_LOG"
    fi
    sleep 5
done
EOF

sudo chmod +x /opt/logmonitor/weblog-monitor.sh

Create service user

sudo useradd -r -s /sbin/nologin -d /opt/logmonitor logmonitor
sudo chown -R logmonitor:logmonitor /opt/logmonitor
sudo touch /var/log/logmonitor.log
sudo chown logmonitor:logmonitor /var/log/logmonitor.log

Create systemd service unit

sudo tee /etc/systemd/system/weblog-monitor.service << 'EOF'
[Unit]
Description=Web Log Monitor Service
Documentation=man:tail(1)
After=network.target httpd.service
Wants=network.target

[Service]
Type=simple
User=logmonitor
Group=logmonitor
ExecStart=/opt/logmonitor/weblog-monitor.sh
ExecStop=/bin/kill -TERM $MAINPID
Restart=always
RestartSec=30
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

Enable and test the service

# Reload systemd configuration
sudo systemctl daemon-reload

# Enable and start the service
sudo systemctl enable weblog-monitor
sudo systemctl start weblog-monitor

# Check service status
sudo systemctl status weblog-monitor

# Test service functionality
sudo systemctl stop weblog-monitor
sudo systemctl start weblog-monitor
sudo systemctl restart weblog-monitor

Monitor and verify service

# Check service logs
sudo journalctl -u weblog-monitor -f --no-pager
# Press Ctrl+C to stop following

# Verify service is running as correct user
ps aux | grep weblog-monitor

# Check if service auto-restarts (simulate crash)
sudo pkill -f weblog-monitor.sh
sleep 35  # Wait for restart
sudo systemctl status weblog-monitor

Verification:

# Complete service verification
sudo systemctl is-active weblog-monitor
sudo systemctl is-enabled weblog-monitor
sudo journalctl -u weblog-monitor --no-pager -n 10
ls -l /var/log/logmonitor.log

7. Troubleshooting Playbook¶

Common Issues¶

Issue 1: Service Won't Start¶

Symptoms:

systemctl start fails with error
Service shows "failed" status
Application not responding

Diagnosis:

# Check service status and errors
systemctl status servicename
systemctl --failed

# View detailed logs
journalctl -u servicename --no-pager
journalctl -xe

# Check service dependencies
systemctl list-dependencies servicename

Resolution:

# Fix common issues
systemctl daemon-reload      # Reload if unit file changed
systemctl reset-failed      # Clear failed status

# Check and fix configuration
systemctl cat servicename   # Review unit file
vim /etc/servicename/config  # Fix application config

# Restart dependencies if needed
systemctl restart dependency.service
systemctl start servicename

Prevention: Always test configuration changes before applying to production

Issue 2: High System Load¶

Symptoms:

System responds slowly
High load average
Applications timing out

Diagnosis:

# Check system load
uptime
top
htop (if available)

# Identify resource-intensive processes
ps aux --sort=-%cpu | head -10
ps aux --sort=-%mem | head -10

# Check I/O activity
iotop (if available)
iostat 1 5

Resolution:

# Adjust process priorities
renice 10 PID              # Lower priority
ionice -c 3 PID           # Idle I/O class

# Restart problematic services
systemctl restart servicename

# Kill runaway processes if necessary
kill PID
kill -9 PID  # If graceful doesn't work

Issue 3: Process Won't Terminate¶

Symptoms:

Process ignores TERM signal
kill command has no effect
Process shows as zombie

Diagnosis:

# Check process state
ps -o pid,ppid,state,comm -p PID

# Check what process is doing
lsof -p PID
strace -p PID (use carefully)

# Check for zombie processes
ps aux | grep -E '<defunct>|<zombie>'

Resolution:

# Try escalating signals
kill PID           # TERM signal
sleep 5
kill -QUIT PID     # QUIT signal
sleep 5  
kill -9 PID        # KILL signal (last resort)

# For zombie processes, kill parent
ps -o pid,ppid -p PID
kill PPID

Diagnostic Command Sequence¶

# Service troubleshooting workflow
systemctl status servicename    # Check service status
journalctl -u servicename       # Check service logs
systemctl cat servicename       # Review unit file
ps aux | grep servicename       # Check if process running
lsof -i :port                   # Check port usage

Log File Analysis¶

journalctl: Primary systemd log viewer
/var/log/messages: General system messages
/var/log/secure: Authentication and security events
Service-specific logs: Application logs in /var/log/

8. Quick Reference Card¶

Essential Commands At-a-Glance¶

# Process monitoring
ps aux                          # List all processes
top                            # Interactive process monitor
pgrep processname              # Find process by name
kill PID                       # Terminate process

# Service management
systemctl start servicename    # Start service
systemctl stop servicename     # Stop service
systemctl enable servicename   # Enable auto-start
systemctl status servicename   # Check service status

# System targets
systemctl get-default          # Show default target
systemctl isolate target       # Switch to target

Important Signals¶

TERM (15): Graceful termination (default for kill)
KILL (9): Force termination (cannot be caught)
HUP (1): Hangup (often used to reload config)
STOP (19): Suspend process
CONT (18): Resume suspended process

Common systemd Targets¶

poweroff.target: Shutdown system
rescue.target: Single-user mode
multi-user.target: Multi-user, no GUI
graphical.target: Multi-user with GUI
reboot.target: Restart system

Process States¶

R: Running or runnable
S: Sleeping (waiting for event)
D: Uninterruptible sleep (usually I/O)
T: Stopped (by signal)
Z: Zombie (terminated but not cleaned up)

9. Knowledge Check¶

Conceptual Questions¶

Question: What's the difference between systemctl start and systemctl enable? Answer: start immediately begins running the service, while enable configures the service to start automatically at boot. A service can be enabled but not running, or running but not enabled. For complete setup, you typically need both commands.
Question: Why might a process become a zombie? Answer: A zombie process occurs when a child process has finished executing but its parent hasn't read its exit status yet. The process entry remains in the process table until the parent calls wait(). If the parent never calls wait() or terminates, the zombie persists.
Question: What happens when you send SIGKILL (-9) to a process? Answer: SIGKILL cannot be caught or ignored by the process - the kernel immediately terminates it. This bypasses any cleanup code, potentially leaving files open, shared memory segments allocated, or other resources in an inconsistent state. Use only as a last resort.

Practical Scenarios¶

Scenario: A web service keeps crashing and needs to restart automatically. Solution: Configure the systemd service with Restart=always and RestartSec=30 in the [Service] section of the unit file.
Scenario: You need to temporarily stop a service without preventing future automatic starts. Solution: Use systemctl stop servicename. This stops the service but leaves it enabled, so it will still start automatically at next boot.

Command Challenges¶

Challenge: Find all processes owned by the user "apache" and show their CPU usage. Answer: ps -u apache -o pid,user,%cpu,comm Explanation: -u apache filters by user, -o specifies custom output format
Challenge: Create a command to monitor the top 5 CPU-consuming processes, updating every 2 seconds. Answer: top -n 0 -d 2 | head -12 | tail -5 or use watch -n 2 "ps aux --sort=-%cpu | head -5"

10. Exam Strategy¶

Topic-Specific Tips¶

Always use systemctl status to verify service operations
Remember to both enable and start services for complete setup
Practice signal usage - know when to use graceful vs force termination
Understand systemd dependencies - services may fail if dependencies aren't met

Common Exam Scenarios¶

Scenario: Configure a service to start automatically at boot Approach: Use systemctl enable servicename then systemctl start servicename
Scenario: Troubleshoot a service that won't start Approach: Check systemctl status, review journalctl -u servicename, verify dependencies
Scenario: Find and terminate a runaway process Approach: Use top or ps to identify, then kill with appropriate signal

Time Management¶

Basic service operations: 2-3 minutes including verification
Process troubleshooting: 5-7 minutes depending on complexity
Custom service creation: 8-10 minutes for complete setup
Always verify: Check service status after changes

Pitfalls to Avoid¶

Don't forget to systemctl daemon-reload after editing unit files
Remember that stopping a service doesn't disable it (still starts at boot)
Avoid kill -9 unless absolutely necessary - try graceful termination first
Check service dependencies - some services require others to be running
Verify both that service is running AND enabled for boot

Summary¶

Key Takeaways¶

systemd is the modern service manager - master its commands and concepts
Process management requires understanding signals - different signals have different effects
Service dependencies matter - services may fail if dependencies aren't met
Always verify changes - check status after making service modifications

Critical Commands to Remember¶

systemctl start servicename                 # Start service now
systemctl enable servicename                # Start service at boot  
systemctl status servicename                # Check service status
ps aux                                       # List all processes
kill PID                                     # Graceful process termination
journalctl -u servicename                   # View service logs

Next Steps¶

Continue to Module 06: Package Management
Practice service management in the Vagrant environment
Review related topics: Boot Process, Logging

Navigation: ← File Permissions | Index | Next → Package Management