Skip to main content

Super Technician and little scripts : Server Health & Storage Management

 


Server Health & Storage Management: Automated Cleanup, Disk Offloading, and MySQL Backups When running high-traffic web applications, database services, and build environments, server disks can rapidly fill up due to log aggregation, build caches, and upload directories. Reaching 100% disk usage on the root partition (/) often results in service crashes, database corruption, and deployment failures.

This guide details a multi-tiered strategy for managing server storage using secondary disks, directory symlinking, log truncation, and scheduled database backups.

Key Architecture Strategies

  1. Offloading Storage via Symlinks Instead of reconfiguring internal application paths, heavy folders (such as file upload directories, media storage, or virtual environments) can be moved to an external or secondary mount point (e.g., /data). By creating a symbolic link (ln -s) from the original directory path pointing to the secondary disk, applications continue reading and writing seamlessly without code modification.

  2. Log Truncation & Temporary Storage Systemd journal logs (journalctl) and package caches (apt) can quietly consume gigabytes of storage over time. Setting hard upper limits on journal log size and regularly cleaning /tmp prevents unannounced storage spikes on the primary OS drive.

  3. Automated Database Backups & Retention Automating compressed database dumps (mysqldump piped through gzip) ensures point-in-time recovery capabilities. Storing these backups on a secondary partition prevents database operations from filling up the operating system disk. Incorporating an automated cleanup rule keeps only a rolling retention window (e.g., 14 days).

Unified Maintenance Script & Cron Automation Below is the complete, single-block setup containing both the unified Bash maintenance script and the exact cron automation commands to schedule execution nightly at 2:00 AM.

#!/bin/bash

# ==============================================================================

# Unified Server Maintenance, Storage Migration & Database Backup Script

# Target Mount Point: /data

# User Directory Structure: /home/abcd

# ==============================================================================


set -e # Exit immediately if a command fails


# Configuration Variables

USER_HOME="/home/abcd"

DATA_MOUNT="/data"

BACKUP_DIR="${DATA_MOUNT}/backups/db"

SCRIPT_DIR="${DATA_MOUNT}/scripts"

TIMESTAMP=$(date +"%Y%m%d_%H%M%S")

DB_USER="root"

DB_PASS="" # Specify MySQL password if required


echo "=========================================="

echo "Starting Unified Maintenance Task: $(date)"

echo "=========================================="


# ------------------------------------------------------------------------------

# STEP 1: Directory Setup on Secondary Mount

# ------------------------------------------------------------------------------

echo "[1/6] Setting up secondary disk directories at ${DATA_MOUNT}..."

sudo mkdir -p "${DATA_MOUNT}/uploads/zip_inv1"

sudo mkdir -p "${DATA_MOUNT}/stg"

sudo mkdir -p "${DATA_MOUNT}/ptl"

sudo mkdir -p "${BACKUP_DIR}"

sudo mkdir -p "${SCRIPT_DIR}"


sudo chown -R abcd:abcd "${DATA_MOUNT}"


# ------------------------------------------------------------------------------

# STEP 2: Offload Uploads & Staging Directories via Symlinks

# ------------------------------------------------------------------------------

echo "[2/6] Offloading heavy storage folders to secondary drive..."


# Offload Uploads Directory

ORIGINAL_UPLOADS="${USER_HOME}/stg/zip_inv1/uploads"

TARGET_UPLOADS="${DATA_MOUNT}/uploads/zip_inv1"


if [ -d "${ORIGINAL_UPLOADS}" ] && [ ! -L "${ORIGINAL_UPLOADS}" ]; then

    echo "  -> Copying existing uploads..."

    rsync -avP "${ORIGINAL_UPLOADS}/" "${TARGET_UPLOADS}/"

    rm -rf "${ORIGINAL_UPLOADS}"

    ln -s "${TARGET_UPLOADS}" "${ORIGINAL_UPLOADS}"

    echo "  -> Symlink created for uploads."

fi


# Clean up stale backup uploads if present

if [ -d "${USER_HOME}/stg/zip_inv1/uploads_old" ]; then

    echo "  -> Removing legacy uploads backup..."

    rm -rf "${USER_HOME}/stg/zip_inv1/uploads_old"

fi


# Offload Staging Projects (e.g., app1)

if [ -d "${USER_HOME}/stg/app1" ] && [ ! -L "${USER_HOME}/stg/app1" ]; then

    echo "  -> Moving app1 staging folder..."

    mv "${USER_HOME}/stg/app1" "${DATA_MOUNT}/stg/"

    ln -s "${DATA_MOUNT}/stg/app1" "${USER_HOME}/stg/app1"

fi


# Offload Portal Projects (e.g., app1)

if [ -d "${USER_HOME}/ptl/app1" ] && [ ! -L "${USER_HOME}/ptl/app1" ]; then

    echo "  -> Moving app1 portal folder..."

    mv "${USER_HOME}/ptl/app1" "${DATA_MOUNT}/ptl/"

    ln -s "${DATA_MOUNT}/ptl/app1" "${USER_HOME}/ptl/app1"

fi


# ------------------------------------------------------------------------------

# STEP 3: MySQL Database Backup

# ------------------------------------------------------------------------------

echo "[3/6] Performing compressed MySQL database dump..."

BACKUP_FILE="${BACKUP_DIR}/db_all_${TIMESTAMP}.sql.gz"


if [ -n "${DB_PASS}" ]; then

    mysqldump -u "${DB_USER}" -p"${DB_PASS}" --all-databases | gzip > "${BACKUP_FILE}"

else

    sudo mysqldump --all-databases | gzip > "${BACKUP_FILE}"

fi


if [ -f "${BACKUP_FILE}" ]; then

    echo "  -> Backup created successfully: ${BACKUP_FILE}"

else

    echo "  -> ERROR: Failed to create database backup!"

fi


# ------------------------------------------------------------------------------

# STEP 4: Retention Management (Purge Old Backups)

# ------------------------------------------------------------------------------

echo "[4/6] Cleaning database backups older than 14 days..."

find "${BACKUP_DIR}" -type f -name "*.sql.gz" -mtime +14 -delete


# ------------------------------------------------------------------------------

# STEP 5: System Log & Package Cache Truncation

# ------------------------------------------------------------------------------

echo "[5/6] Truncating system logs and clearing caches..."

sudo journalctl --vacuum-size=100M

sudo apt-get clean

sudo apt-get autoremove -y


# Clear system temp files older than 2 days

sudo find /tmp /var/tmp -type f -atime +2 -delete 2>/dev/null || true


# ------------------------------------------------------------------------------

# STEP 6: Service Status Verification

# ------------------------------------------------------------------------------

echo "[6/6] Verifying system storage state..."

df -h / "${DATA_MOUNT}"


echo "=========================================="

echo "Maintenance Complete: $(date)"

echo "=========================================="



# ==============================================================================

# CRON AUTOMATION SETUP

# Save above script to: /data/scripts/maintenance.sh

# Execute commands below to schedule run at 2:00 AM daily

# ==============================================================================


# 1. Grant execution permissions:

# sudo chmod +x /data/scripts/maintenance.sh


# 2. Add to root crontab (/etc/crontab or sudo crontab -e):

# 0 2 * * * /data/scripts/maintenance.sh >> /data/backups/maintenance.log 2>&1


AI is so amazing , one hour scripting  work I did in 5 minutes ...

Comments

Post a Comment

Popular posts from this blog

Telecom OSS and BSS: A Comprehensive Guide

  Telecom OSS and BSS: A Comprehensive Guide Table of Contents Part I: Foundations of Telecom Operations Chapter 1: Introduction to Telecommunications Networks A Brief History of Telecommunications Network Architectures: From PSTN to 5G Key Network Elements and Protocols Chapter 2: Understanding OSS and BSS Defining OSS and BSS The Role of OSS in Network Management The Role of BSS in Business Operations The Interdependence of OSS and BSS Chapter 3: The Telecom Business Landscape Service Providers and Their Business Models The Evolving Customer Experience Regulatory and Compliance Considerations The Impact of Digital Transformation Part II: Operations Support Systems (OSS) Chapter 4: Network Inventory Management (NIM) The Importance of Accurate Inventory NIM Systems and Their Functionality Data Modeling and Management Automation and Reconciliation Chapter 5: Fault Management (FM) Detecting and Isolating Network Faults FM Systems and Alerting Mecha...
1 comment
Read more

"Depth-Guard" – 3D Spatial Occupancy monitor Challenge -2

  Project Title: "Depth-Guard" – 3D Spatial Occupancy Monitor 1. The Problem In a smart warehouse, a robot needs to know if a loading zone is clear or occupied. A 2D camera alone can’t tell the difference between a "flat picture of a box" on the floor and an "actual 3D box." The Goal: Build a Python-based system that uses Computer Vision and Depth Perception (AI 3D) to identify objects and determine their 3D volume (Size) and Distance from the camera. 2. Intern Tasks Object Detection: Use a pre-trained model (like YOLOv8) to draw 2D boxes around objects. Depth Mapping: Use a depth estimation model (like MiDaS or a simulated Stereo-depth feed) to calculate how far each object is. Occupancy Logic: If an object is closer than 1 meter and larger than a specific volume, mark the zone as "BLOCKED." Alert System: Print a warning if the 3D space is too crowded. 3. Sample Datasets (Simulation) Since interns may not have 3D cameras (LiDAR/RGB-D), pr...
Post a Comment
Read more

Simple Virtual Waiting Room -Challenge 1

   Simple Virtual Waiting Room (VWR) 1. The Problem Our website can only handle 10 users per minute . If more than 10 people try to access it at once, the server will crash. We need a system that: Counts incoming users. Redirects "overflow" users to a waiting page. Admits them back to the main site one by one as space becomes available. 2. Intern Tasks Create a Gateway: A simple script that checks: if (active_users < 10) { allow } else { send to queue } . Build the Queue: Use a simple list (FIFO) to store user IDs. The Wait Page: A basic HTML page that says: "You are number X in line. Estimated wait: Y minutes." Admission Logic: Every 30 seconds, pull the next user from the queue and "admit" them. 3. Sample Datasets (Simulation) Provide these two datasets to the interns. They should write a script to "read" these files and simulate how their system reacts. Dataset A: The Traffic Surge (Input) This file simulates users arriving at the ...
Post a Comment
Read more