Skip to content

Commit 0db56fc

Browse files
author
Oliver Cervera
authored
Merge dev from auanasgheps/dev
Push dev to main
2 parents 9ee9e58 + dc529dd commit 0db56fc

File tree

3 files changed

+302
-128
lines changed

3 files changed

+302
-128
lines changed

README.md

Lines changed: 165 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -4,37 +4,188 @@ The definitive all-in-one [SnapRAID](https://github.com/amadvance/snapraid) scri
44
There are many SnapRAID scripts out there, but none could fit my needs. So I took the best of them to start a new one.
55

66
It is meant to be run periodically (e.g. daily) and do the heavy lifting, then send an email you will actually read.
7-
It is highly customizable.
8-
It has been tested with Debian 10 and OpenMediaVault 5.
7+
8+
Supports single and dual parity configurations.
9+
10+
It is customizable and has been tested with Debian 10 and OpenMediaVault 5.
911

1012
Contributions are welcome: there's always room for improvement!
1113

12-
This readme has some rough edges which will be smoothened over time.
14+
_This readme has some rough edges which will be smoothened over time._
15+
16+
# Highlights
17+
18+
## How it works
19+
- After some preliminary checks, the script will execute `snapraid diff` to figure out if parity info is out of date, which means checking for changes since the last execution.
20+
- One of the following will happen:
21+
- If parity info is out of sync **and** the number of deleted or changed files exceed the threshold you have configured it **stops**. You may want to take a look to the output log.
22+
- If parity info is out of sync **and** the number of deleted or changed files exceed the threshold, you can still **force a sync** after a number of warnings. It's useful If you often get a false alarm but you're confident enough.
23+
- If parity info is out of sync **but** the number of deleted or changed files did not exceed the treshold, it **executes a sync** to update the parity info.
24+
- When the parity info is in sync, either because nothing has changed or after a successfully sync, it runs the `snapraid scrub` command to validate the integrity of the data, both the files and the parity info. _Note that each run of the scrub command will validate only a configurable portion of parity info to avoid having a long running job and affecting the performance of the server._
25+
- When the script is done sends an email with the results, both in case of error or success.
26+
27+
Pre-hashing is enabled by default to avoid silent read errors. It mitigates the lack of ECC memory.
28+
29+
## A nice email report
30+
This report produces emails that don't contain a list of changed files to improve clarity.
31+
32+
You can re-enable full output in the email by switching the option `VERBOSITY` but the full report will always be available in `/tmp/snapRAID.out` and will be replaced after each run or deleted when the system is shut down if kept there.
33+
34+
SMART drive report from SnapRAID is also included by default.
35+
36+
Here's a sneak peek of the email report.
37+
38+
```markdown
39+
## [COMPLETED] DIFF + SYNC + SCRUB Jobs (SnapRAID on omv-test.local)
40+
41+
SnapRAID Script Job started [Sat Jan 9 02:07:46 CET 2021]
42+
Running SnapRAID version 11.5
43+
SnapRAID Script version 2.7.0
44+
45+
----------
46+
47+
## Preprocessing
48+
49+
Configuration file found! Proceeding.
50+
Testing that all parity files are present.
51+
All parity files found. Continuing...
52+
53+
----------
54+
55+
## Processing
56+
57+
### SnapRAID TOUCH [Sat Jan 9 02:07:46 CET 2021]
58+
59+
Checking for zero sub-second files.
60+
No zero sub-second timestamp files found.
61+
TOUCH finished [Sat Jan 9 02:07:46 CET 2021]
62+
63+
### SnapRAID DIFF [Sat Jan 9 02:07:46 CET 2021]
64+
65+
DIFF finished [Sat Jan 9 02:07:46 CET 2021]
66+
67+
**SUMMARY of changes - Added [2] - Deleted [0] - Moved [0] - Copied [0] - Updated [0]**
68+
69+
There are deleted files. The number of deleted files, (0), is below the threshold of (2). SYNC Authorized.
70+
There are updated files. The number of updated files, (0), is below the threshold of (2). SYNC Authorized.
71+
72+
### SnapRAID SYNC [Sat Jan 9 02:07:46 CET 2021]
73+
74+
Self test...
75+
Loading state from /srv/dev-disk-by-label-DISK1/snapraid.content...
76+
Scanning disk DATA1...
77+
Scanning disk DATA2...
78+
Using 0 MiB of memory for the file-system.
79+
Initializing...
80+
Hashing...
81+
SYNC_JOB--Everything OK
82+
Resizing...
83+
Saving state to /srv/dev-disk-by-label-DISK1/snapraid.content...
84+
Saving state to /srv/dev-disk-by-label-DISK2/snapraid.content...
85+
Saving state to /srv/dev-disk-by-label-DISK3/snapraid.content...
86+
Saving state to /srv/dev-disk-by-label-DISK4/snapraid.content...
87+
Verifying /srv/dev-disk-by-label-DISK1/snapraid.content...
88+
Verifying /srv/dev-disk-by-label-DISK2/snapraid.content...
89+
Verifying /srv/dev-disk-by-label-DISK3/snapraid.content...
90+
Verifying /srv/dev-disk-by-label-DISK4/snapraid.content...
91+
Verified /srv/dev-disk-by-label-DISK4/snapraid.content in 0 seconds
92+
Verified /srv/dev-disk-by-label-DISK3/snapraid.content in 0 seconds
93+
Verified /srv/dev-disk-by-label-DISK2/snapraid.content in 0 seconds
94+
Verified /srv/dev-disk-by-label-DISK1/snapraid.content in 0 seconds
95+
Syncing...
96+
Using 32 MiB of memory for 32 cached blocks.
97+
98+
DATA1 59% | ***********************************
99+
DATA2 55% | ********************************
100+
parity 0% |
101+
2-parity 0% |
102+
raid 6% |
103+
hash 5% |
104+
sched 7% |
105+
misc 17% |
106+
|______________
107+
wait time (total, less is better)
108+
109+
SYNC_JOB--Everything OK
110+
Saving state to /srv/dev-disk-by-label-DISK1/snapraid.content...
111+
Saving state to /srv/dev-disk-by-label-DISK2/snapraid.content...
112+
Saving state to /srv/dev-disk-by-label-DISK3/snapraid.content...
113+
Saving state to /srv/dev-disk-by-label-DISK4/snapraid.content...
114+
Verifying /srv/dev-disk-by-label-DISK1/snapraid.content...
115+
Verifying /srv/dev-disk-by-label-DISK2/snapraid.content...
116+
Verifying /srv/dev-disk-by-label-DISK3/snapraid.content...
117+
Verifying /srv/dev-disk-by-label-DISK4/snapraid.content...
118+
Verified /srv/dev-disk-by-label-DISK4/snapraid.content in 0 seconds
119+
Verified /srv/dev-disk-by-label-DISK3/snapraid.content in 0 seconds
120+
Verified /srv/dev-disk-by-label-DISK2/snapraid.content in 0 seconds
121+
Verified /srv/dev-disk-by-label-DISK1/snapraid.content in 0 seconds
122+
SYNC finished [Sat Jan 9 02:07:49 CET 2021]
123+
124+
### SnapRAID SCRUB [Sat Jan 9 02:07:49 CET 2021]
125+
126+
Self test...
127+
Loading state from /srv/dev-disk-by-label-DISK1/snapraid.content...
128+
Using 0 MiB of memory for the file-system.
129+
Initializing...
130+
Scrubbing...
131+
Using 48 MiB of memory for 32 cached blocks.
132+
SCRUB_JOB--Nothing to do
133+
SCRUB finished [Sat Jan 9 02:07:49 CET 2021]
134+
135+
----------
136+
137+
## Postprocessing
138+
139+
SnapRAID SMART report:
140+
141+
Temp Power Error FP Size
142+
C OnDays Count TB Serial Device Disk
143+
144+
----------
145+
146+
- - - SSD 0.0 00000000000000000001 /dev/sdb DATA1
147+
- - - - 0.0 01000000000000000001 /dev/sdc DATA2
148+
- - - SSD 0.0 02000000000000000001 /dev/sdd parity
149+
- - - SSD 0.0 03000000000000000001 /dev/sde 2-parity
150+
0 - - - 0.0 - /dev/sda -
151+
152+
The FP column is the estimated probability (in percentage) that the disk
153+
is going to fail in the next year.
154+
155+
Probability that at least one disk is going to fail in the next year is 0%.
156+
All jobs ended. [Sat Jan 9 02:07:49 CET 2021]
157+
Email address is set. Sending email report to [email protected] [Sat Jan 9 02:07:49 CET 2021]
158+
```
159+
160+
## Customization
161+
Many options can be changed to your taste, their behaviour is documented in the script config file.
162+
163+
If you don't know what to do, I recommend using the default values and see how it performs.
164+
165+
You can also change more advanced options such as mail binary (by default uses `mailx`), SnapRAID binary location, log file location.
13166

14-
# Features
15-
[WIP]
16167

17168
# Requirements
18169
- Markdown to have nice emails
19-
- Hd-idle to spin down disks - [Link TBD]
170+
- ~~Hd-idle to spin down disks - [Link TBD] - currently not required since spin down does not work properly.~~
20171

21172
# Installation
22-
[WIP]
23-
1. Install markdown `apt install python-markdown`
24-
2. Place the script wherever you prefer e.g. `/usr/sbin/snapraid`
25-
3. Give executable rights - `chmod +x snapraid-aio-script.sh`
26-
4. Open the script and add your email address at line 43
27-
5. Tweak the script if needed
173+
1. Install markdown `apt install python-markdown`. You can skip this step since the script will check and install it for you.
174+
2. Download config file and script, then place wherever you prefer e.g. `/usr/sbin/snapraid`
175+
3. Give executable rights to the main script - `chmod +x snapraid-aio-script.sh`
176+
4. Edit the config file and add your email address at line 9
177+
5. Tweak the config file if needed
28178
6. Schedule the script execution time
29179

30180
# Known Issues
31-
Hard disk spin down does not work: they are immediately woken up. The script probably does not handle this correctly while running.
181+
- Hard disk spin down does not work: they are immediately woken up. The script probably does not handle this correctly while running.
182+
- The report is not perfect, we can't be solve this because SnapRAID does not natively support Markdown.
32183

33184
# Credits
34185
All rights belong to the respective creators.
35186
Thanks to:
36187
- [Zack Reed](https://zackreed.me/snapraid-split-parity-sync-script/) for most of the original script
37188
- [mtompkins](https://gist.github.com/mtompkins/91cf0b8be36064c237da3f39ff5cc49d) for most of the original script
38189
- [sburke](https://zackreed.me/snapraid-split-parity-sync-script/#comment-300) for the Debian 10 fix
39-
- metagliatore (I don't think he's on Github) for removing the DIFF output from the email
190+
- metagliatore (a friend, not on Github) for removing the DIFF output from the email
40191
- [ozboss](https://forum.openmediavault.org/wsc/index.php?user/27331-ozboss/)

script-config.sh

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
#!/bin/bash
2+
######################
3+
# USER VARIABLES #
4+
######################
5+
6+
####################### USER CONFIGURATION START #######################
7+
8+
# address where the output of the jobs will be emailed to.
9+
EMAIL_ADDRESS="youremailgoeshere"
10+
11+
# Set the threshold of deleted files to stop the sync job from running.
12+
# NOTE that depending on how active your filesystem is being used, a low
13+
# number here may result in your parity info being out of sync often and/or
14+
# you having to do lots of manual syncing.
15+
DEL_THRESHOLD=500
16+
UP_THRESHOLD=500
17+
18+
# Set number of warnings before we force a sync job.
19+
# This option comes in handy when you cannot be bothered to manually
20+
# start a sync job when DEL_THRESHOLD is breached due to false alarm.
21+
# Set to 0 to ALWAYS force a sync (i.e. ignore the delete threshold above)
22+
# Set to -1 to NEVER force a sync (i.e. need to manual sync if delete threshold is breached)
23+
SYNC_WARN_THRESHOLD=-1
24+
25+
# Set percentage of array to scrub if it is in sync.
26+
# i.e. 0 to disable and 100 to scrub the full array in one go
27+
# WARNING - depending on size of your array, setting to 100 will take a very long time!
28+
SCRUB_PERCENT=5
29+
SCRUB_AGE=10
30+
31+
# Prehash Data To avoid the risk of a latent hardware issue, you can enable the "pre-hash" mode and have all the
32+
# data read two times to ensure its integrity. This option also verifies the files moved inside the array, to ensure
33+
# that the move operation went successfully, and in case to block the sync and to allow to run a fix operation.
34+
# 1 to enable, any other values to disable
35+
PREHASH=1
36+
37+
# Set the option to log SMART info. 1 to enable, any other value to disable
38+
SMART_LOG=1
39+
40+
# Set verbosity of the email output. TOUCH and DIFF outputs will be kept in the email, producing a potentially huge email. Keep this disabled for optimal reading
41+
# You can always check TOUCH and DIFF outputs using the TMP file.
42+
# 1 to enable, any other values to disable
43+
VERBOSITY=0
44+
45+
# Set if disk spindown should be performed. Depending on your system, this may not work. 1 to enable, any other values to disable
46+
SPINDOWN=0
47+
48+
# Run snapraid status command to show array general information.
49+
# Be aware the HTML output is pretty broken.
50+
SNAP_STATUS=0
51+
52+
# location of the snapraid binary
53+
SNAPRAID_BIN="/usr/bin/snapraid"
54+
# location of the mail program binary
55+
MAIL_BIN="/usr/bin/mailx"
56+
57+
####################### USER CONFIGURATION END #######################
58+
59+
####################### SYSTEM CONFIGURATION #######################
60+
# Make changes only if you know what you're doing
61+
######################
62+
63+
# Init variables
64+
CHK_FAIL=0
65+
DO_SYNC=0
66+
EMAIL_SUBJECT_PREFIX="(SnapRAID on `hostname`)"
67+
GRACEFUL=0
68+
SYNC_WARN_FILE="$CURRENT_DIR/snapRAID.warnCount"
69+
SYNC_WARN_COUNT=""
70+
TMP_OUTPUT="/tmp/snapRAID.out"
71+
SNAPRAID_LOG="/var/log/snapraid.log"
72+
SECONDS=0 #Capture time
73+
74+
# Expand PATH for smartctl
75+
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
76+
77+
# Determine names of first content file...
78+
CONTENT_FILE=`grep -v '^$\|^\s*\#' /etc/snapraid.conf | grep snapraid.content | head -n 1 | cut -d " " -f2`
79+
80+
# Build an array of parity all files...
81+
PARITY_FILES[0]=`grep -v '^$\|^\s*\#' /etc/snapraid.conf | grep snapraid.parity | head -n 1 | cut -d " " -f2`
82+
IFS=$'\n' PARITY_FILES=(`cat /etc/snapraid.conf | grep "^[^#;]" | grep "^\([2-6z]-\)*parity" | cut -d " " -f 2 | tr ',' '\n'`)

0 commit comments

Comments
 (0)