Merge pull request #47 from auanasgheps/dev

Merge dev for 3.2 release

Author: Oliver Cervera

README.md

@@ -32,7 +32,8 @@ Contributions are welcome!
 - After some preliminary checks, the script will execute `snapraid diff` to figure out if parity info is out of date, which means checking for changes since the last execution. During this step, the script will ensure drives are fine by reading parity and content files. 
 - One of the following will happen:     
     - If parity info is out of sync **and** the number of deleted or changed files exceed the threshold you have configured it **stops**. You may want to take a look to the output log.
-    - If parity info is out of sync **and** the number of deleted or changed files exceed the threshold, you can still **force a sync** after a number of warnings. It's useful If  you often get a false alarm but you're confident enough. This is called "Sync with threshold warnings"
+    - If parity info is out of sync **and** the number of deleted or changed files exceed the threshold, you can still **force a sync** after a number of warnings. It's useful If you often get a false alarm but you're confident enough. This is called "Sync with threshold warnings"
+    	- Instead of forcing a sync based on the number of deleted files, you may consider the `ADD_DEL_THRESHOLD` feature, by allowing a sync that would otherwise violate the delete threshold, if the ratio of added to deleted files is greater than the value set. 
     - If parity info is out of sync **but** the number of deleted or changed files did not exceed the threshold, it **executes a sync** to update the parity info.
 - When the parity info is in sync, either because nothing has changed or after a successfully sync, it runs the `snapraid scrub` command to validate the integrity of the data, both the files and the parity info. If sync was cancelled or other issues were found, scrub will not be run. 
     - Note that each run of the scrub command will validate only a configurable portion of parity info to avoid having a long running job and affecting the performance of the server. 
@@ -64,15 +65,21 @@ If you don't know what to do, I recommend using the default values and see how i
 	- Sync always (forced sync).
 	- Sync after a number of breached threshold warnings. 
 	- Sync only if thresholds warnings are not breached (enabled by default).
+	- Sync even if the delete threshold has been breached, but the ratio of added to deleted files is greater than the value set. 
 	- User definable thresholds for deleted and updated files.
 - Scrub options 
 	- Enable or disable scrub job.
 	- Delayed option, disabled by default. Run scrub only after a number of script executions, e.g. every 7 times. If you don't want to scrub your array every time, this one is for you.
 	- Data to be scrubbed - by default 5% older than 10 days.
-- Pre-hashing - enabled by default. Mitigate the lack of ECC memory, reading data twice to avoid silent read errors. 
+	- Scrub new data - scrub the data that was just added by the sync.
+- Pre-hashing - enabled by default. Mitigate the lack of ECC memory, reading data twice to avoid silent read errors.
+- Force zero size sync -  disabled by default. Forces the operation of syncing a file with zero size that before was not. Use with caution!
 - Snapraid Status - shows the status of the array, disabled by default.
 - SMART Log - enabled by default. A SnapRAID report for disks health status.
 - Verbosity option - disabled by default. When enabled, includes the TOUCH and DIFF commands output. Please note email will be huge and mostly unreadable.
+- SnapRAID Output (log) retention - disabled by default (log is overriden every run)
+	- Detailed output retention for each run
+	- You can choose the amount of days and the path, by default set to the user home 
 - Healthchecks.io, Telegram and Discord integration
 	- If you don't read your emails every day, this is a great one for you, since you can be quickly informed if things go wrong. 
   	- The script will report to Healthchecks.io, Telegram and Discord when is started and when is completed. If there's a failure it's included as well.
@@ -82,7 +89,7 @@ If you don't know what to do, I recommend using the default values and see how i
 	- Made for external services or mail binaries with different commands than `mailx`.
 	- Configure the path of the script or the mail binary to be invoked.
 	- You can still use native services since it only replaces the standard email.
-- Container management - disabled by default. 
+- Docker Container management
 	- A list of containers you want to be interrupted before running actions and restored when completed.
    	- Docker mode - choose to pause/unpause or to stop/restart your containers
    	- Docker remote - if docker is running on a remote machine
@@ -98,7 +105,7 @@ You can also change more advanced options such SnapRAID binary location, log fil
 ## A nice email report
 This script produces emails that don't contain a list of changed files to improve clarity.
 
-You can re-enable full output in the email by switching the option `VERBOSITY`. The full report will always be available in `/tmp/snapRAID.out` but will be replaced after each run, or deleted when the system is shut down. You can change the location of the file if you need to keep it.
+You can re-enable full output in the email by switching the option `VERBOSITY`. The full report is available in `/tmp/snapRAID.out` but will be replaced after each run, or deleted when the system is shut down. You can enable the retention policy to keep logs for some days and customize the folder location.
 
 Here's an example email report. 
 
@@ -279,7 +286,7 @@ If you start with empty disks, you cannot use (yet) this script, since it expect
 First run `snapraid sync`. Once completed, the array will be ready to be used with this script.
 
 ## OMV5/6 and SnapRAID plugin
-Ignore what you see at _OMV GUI > Services > SnapRAID > Diff Script Settings_, since it only applies to the plugin's built-in script. Also don't forget to remove the built-in `omv-snapraid-diff` script from _OMV GUI > System > Scheduled Tasks_, either by deleting the job, or simply disabling it.
+Ignore what you see at _OMV GUI > Services > SnapRAID > Diff Script Settings_, since it only applies to the plugin's built-in script. Also don't forget to remove the built-in `omv-snapraid-diff` job from _OMV GUI > System > Scheduled Tasks_, either by deleting the job, or simply disabling it.
 
 ## Installing `hd-idle` for Automatic Disk Spindown
 If you would like to enable automatic disk spindown after the script job runs, then you will need to install `hd-idle`. The version included in default Debian and Ubuntu repositories is buggy and out of date - fortunately developer [adelolmo](https://github.com/adelolmo/hd-idle) has improved the project and released an updated version.
@@ -318,3 +325,4 @@ This script would not exist without:
 - [nzlov](https://github.com/nzlov)
 - [Caedis](https://github.com/Caedis)
 - [Pushpender](https://github.com/ranapushpender)
+- [Phidauex](https://github.com/phidauex)

script-config.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-CONFIG_VERSION=3.1
+CONFIG_VERSION=3.2
 ######################
 #   USER VARIABLES   #
 ######################
@@ -8,15 +8,20 @@ CONFIG_VERSION=3.1
 
 ### NOTIFICATION SETTINGS ###
 
-# address where the output of the jobs will be emailed to.
-EMAIL_ADDRESS="youremailgoeshere"
-FROM_EMAIL_ADDRESS="fromemailgoeshere"
+# Address where the output will be emailed to.
+# If you do not want to receive emails and rely on other notification 
+# methods, leave these fields empty.
+EMAIL_ADDRESS="destination-email-goes-here"
+FROM_EMAIL_ADDRESS="sender-email-goes-here"
 
 # Use Healthchecks.io to report script errors. Set to 1 to enable.
 # Please note that every "WARNING" will be reported as failure.
 # When enabled, enter your Healthchecks UUID (not the full URL).
+# If using a self-hosted instance, change the URL to your endpoint                                                                                                      
+# including the trailing slash.
 HEALTHCHECKS=0
 HEALTHCHECKS_ID="your-uuid-here"
+HEALTHCHECKS_URL="https://hc-ping.com/"
 
 # Use Telegram to report script execution summary (not the whole report)
 # Set 1 to enable. Create a bot using @botfather, then copy the API token.
@@ -45,26 +50,37 @@ HOOK_NOTIFICATION=""
 
 ### SCRIPT AND SNAPRAID SETTINGS ###
 
-# Set the threshold of deleted files to stop the sync job from running. Note
-# that depending on how active your filesystem is being used, a low number here
-# may result in your parity info being out of sync often and/or you having to
-# do lots of manual syncing.
+# Set the threshold of deleted and updated files to stop the sync job from running. 
+# Note that depending on how active your filesystem is being used, a low number
+# here may result in your parity info being out of sync often and/or you having
+# to do lots of manual syncing.
 DEL_THRESHOLD=500
 UP_THRESHOLD=500
 
+# Allow a sync that would otherwise violate the delete threshold, but only
+# if the ratio of added to deleted files is greater than the value set. 
+# Set to 0 to disable this option.
+# Example: A senario with 5000 deleted files and 3800 added files would 
+# result in an ADD_DEL_THRESHOLD of 0.76 (3800/5000) 
+ADD_DEL_THRESHOLD=0
+
 # Set number of warnings before we force a sync job. This option comes in handy
 # when you cannot be bothered to manually start a sync job when DEL_THRESHOLD
 # is breached due to false alarm. Set to 0 to ALWAYS force a sync (i.e. ignore
 # the delete threshold above) Set to -1 to NEVER force a sync (i.e. need to
 # manual sync if delete threshold is breached).
 SYNC_WARN_THRESHOLD=-1
 
-# Set percentage of array to scrub if it is in sync. i.e. 0 to disable and 100
-# to scrub the full array in one go WARNING - depending on size of your array,
-# setting to 100 will take a very long time!
+# Set percentage and age, in days, of blocks in array to scrub if it is in sync.
+# i.e. 0 to disable and 100 to scrub the full array in one go.
+# WARNING - depending on size of your array, setting to 100 can take a long time!
 SCRUB_PERCENT=5
 SCRUB_AGE=10
 
+# Scrub new blocks after sync that have yet to be scrubbed. 1 to enable and any
+# other value to disable.
+SCRUB_NEW=0
+
 # Set number of script runs before running a scrub. Use this option if you
 # don't want to scrub the array every time.
 # Set to 0 to disable this option and run scrub every time.
@@ -77,6 +93,13 @@ SCRUB_DELAYED_RUN=0
 # allow to run a fix operation. 1 to enable, any other value to disable.
 PREHASH=1
 
+# Forces the operation of syncing a file with zero size that before was not. 
+# If SnapRAID detects a such condition, it stops proceeding unless you enable
+# this option. Useful when syncing system files which can genuinely get 
+# changed to zero.
+# Disabled by default, 1 to enable.
+FORCE_ZERO=0
+
 # Set if disk spindown should be performed. Depending on your system, this may
 # not work. 1 to enable, any other value to disable.
 # hd-idle is required and must be already configured.
@@ -86,12 +109,22 @@ SPINDOWN=0
 # other value to disable.
 SMART_LOG=1
 
-# Increase verbosity of the email output. If set to 1, TOUCH and DIFF outputs
-# will be kept in the email, producing a mostly unreadable email. Keep this
-# disabled for optimal results. You can always check TOUCH and DIFF outputs
-# using the TMP file. 1 to enable, any other value to disable.
+# Increase verbosity of the email output. NOT RECOMMENDED!
+# If set to 1, TOUCH and DIFF outputs will be kept in the email, producing 
+# a mostly unreadable email. You can always check TOUCH and DIFF outputs
+# using the TMP file or use the feature KEEP_LOG.
+# 1 to enable, any other value to disable.
 VERBOSITY=0
 
+# SnapRAID detailed output retention for each run.
+# Default behaviour is RETENTION_DAYS=0: every time your run SnapRAID, the 
+# output is saved to "/tmp" and is overridden during every run.
+# To enable retention, set RETENTION_DAYS to the days of output you want to 
+# keep in your home folder. Files will have timestamps. 
+# SNAPRAID_LOG_DIR can be changed to any folder you like.
+RETENTION_DAYS=0
+SNAPRAID_LOG_DIR="$HOME"
+
 # Run 'snapraid status' command to show array general information.
 # 1 to enable, any other value to disable.
 SNAP_STATUS=0
@@ -132,7 +165,7 @@ DOCKER_DELAY=10
 
 # Hooks are shell commands that the scripts executes for you.
 # You can specify 'before_hook' to perform preparation steps before SnapRAID
-# actions and specify 'after_hook' to perform steps afterwards.
+# actions and 'after_hook' to perform steps afterwards.
 
 # Set to 1 to enable custom hooks
 CUSTOM_HOOK=0