From 228ea14222db5f7ccbe1a7a4b5ef79a1a9ce2e24 Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Thu, 24 Apr 2025 16:07:45 -0400 Subject: [PATCH 01/11] WIP Heredoc concept --- concepts/README.md | 10 +- concepts/heredocs/.meta/config.json | 11 ++ concepts/heredocs/about.md | 169 ++++++++++++++++++++++++++++ concepts/heredocs/introduction.md | 3 + concepts/heredocs/links.json | 7 ++ config.json | 5 + 6 files changed, 202 insertions(+), 3 deletions(-) create mode 100644 concepts/heredocs/.meta/config.json create mode 100644 concepts/heredocs/about.md create mode 100644 concepts/heredocs/introduction.md create mode 100644 concepts/heredocs/links.json diff --git a/concepts/README.md b/concepts/README.md index 6b9c92e4..cbad123e 100644 --- a/concepts/README.md +++ b/concepts/README.md @@ -95,18 +95,22 @@ The suggested concept flow: ``` - sublist syntax `${ary[@]:offset:length}` -11. I/O +11. Redirection - file descriptors, stdin, stdout, stderr - redirection + +12. Here Documents - here-docs and here-strings + +## More Concepts + +x. I/O - command substitution - capturing stdout and stderr - capturing stdout and stderr **into separate variables** - `exec` and redirections - process substitutions -## More Concepts - - brace expansions and how it's different from patterns `/path/to/{foo,bar,baz}.txt` x. option parsing with getopts diff --git a/concepts/heredocs/.meta/config.json b/concepts/heredocs/.meta/config.json new file mode 100644 index 00000000..8d62b522 --- /dev/null +++ b/concepts/heredocs/.meta/config.json @@ -0,0 +1,11 @@ +{ + "authors": [ + "glennj" + ], + "contributors": [ + "IsaacG", + "kotp" + ], + "blurb": "Here Documents provide a redirection that input from an inline document." +} + \ No newline at end of file diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md new file mode 100644 index 00000000..f6215578 --- /dev/null +++ b/concepts/heredocs/about.md @@ -0,0 +1,169 @@ +# About Here Documents + +In Bash scripting, a "here document" (or "heredoc") is a way to redirect multiple lines of input to a command or program, as if you were typing them directly into the terminal. +It's a powerful tool for embedding multi-line text within your scripts without needing external files or complex string manipulation. + +## Key Features and Syntax + +1. Delimiter: A heredoc starts with the `<<` operator followed by a delimiter word (often called the "marker" or "terminator"). + This delimiter can be any word you choose, but it's common to use something like `EOF`, `EOF`, `END`, or `TEXT` for clarity. + +1. Content: After the initial `<< DELIMITER`, you write the content you want to redirect. This can be multiple lines of text, code, or anything else. + +1. Termination: The heredoc ends when the delimiter word appears again on a line by itself, with no leading or trailing spaces. + +## Basic Syntax + +```bash +command << DELIMITER + Content line 1 + Content line 2 + ... + Content line N +DELIMITER +``` + +## How it Works + +* Bash reads everything between the `<< DELIMITER` and the `DELIMITER` on its own line. +* This content is then treated as standard input (`stdin`) for the command. +* The command processes this input as if it were coming from the keyboard. + +## Example 1: Simple Text Output + +```bash +cat << EOF +This is the first line. +This is the second line. +This is the third line. +EOF +``` + +Output: + +```plaintext +This is the first line. +This is the second line. +This is the third line. +``` + +In this example: + +* `cat` is the command. +* `<< EOF` starts the heredoc with `EOF` as the delimiter. +* The three lines of text are the content. +* `EOF` on its own line ends the heredoc. +* `cat` then outputs the content it received. + +## Example 2: Using with wc (Word Count) + +```bash +wc -l << END +Line 1 +Line 2 +Line 3 +END +``` + +Output: + +```plaintext +3 +``` + +Here, `wc -l` counts the number of lines. +The heredoc provides the three lines as input. + +## Example 3: Passing to a Script + +The script: + +```bash +#!/usr/bin/env bash + +# Script to process input +while IFS= read -r line; do + echo "Processing: $line" +done +``` + + +Call the script from an interactive bash prompt with a heredoc: + +```bash +./your_script.sh << MY_DATA +Item 1 +Item 2 +Item 3 +MY_DATA +``` + +This example shows how you can pass data to a script. + +## Variations and Advanced Features + +* Stripping Leading Tabs: If you use `<<-` instead of `<<`, Bash will strip leading _tab characters_ from each line of the heredoc. +This is useful for indenting the heredoc content within your script without affecting the output. + +```bash +# Note, the leading whitespace is tabs only! +cat <<- END + This line has a leading tab. + This line also has a leading tab. +END +``` + +The output is printed without the leading tabs: + +```plaintext +This line has a leading tab. +This line also has a leading tab. +``` + +* Literal Content: Quoting the delimiter (single or double quotes) prevents parameter expansion, command substitution, and arithmetic expansion within the heredoc. +The content is taken literally. + +```bash +cat <<'EOF' +The value of $HOME is not expanded here. +The result of $(date) is not executed. +EOF +``` + +Output: + +```plaintext +The value of $HOME is not expanded here. +The result of $(date) is not executed. +``` + +If the delimiter is not quoted, variable expansion, command substitution, and arithmetic expansion will occur. + +```bash +my_var="Hello" +cat < Date: Fri, 25 Apr 2025 11:11:13 -0400 Subject: [PATCH 02/11] add here strings --- concepts/heredocs/about.md | 88 ++++++++++++++++++++++++++++++++++---- 1 file changed, 80 insertions(+), 8 deletions(-) diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md index f6215578..e53ddfc1 100644 --- a/concepts/heredocs/about.md +++ b/concepts/heredocs/about.md @@ -7,6 +7,7 @@ It's a powerful tool for embedding multi-line text within your scripts without n 1. Delimiter: A heredoc starts with the `<<` operator followed by a delimiter word (often called the "marker" or "terminator"). This delimiter can be any word you choose, but it's common to use something like `EOF`, `EOF`, `END`, or `TEXT` for clarity. + For more readable code, you can add the purpose of the document to the delimiter, for example `END_INSTALLATION_INSTRUCTIONS`. 1. Content: After the initial `<< DELIMITER`, you write the content you want to redirect. This can be multiple lines of text, code, or anything else. @@ -91,22 +92,22 @@ done Call the script from an interactive bash prompt with a heredoc: ```bash -./your_script.sh << MY_DATA +./your_script << MY_DATA Item 1 Item 2 Item 3 MY_DATA ``` -This example shows how you can pass data to a script. - ## Variations and Advanced Features -* Stripping Leading Tabs: If you use `<<-` instead of `<<`, Bash will strip leading _tab characters_ from each line of the heredoc. +### Stripping Leading Tabs + +If you use `<<-` instead of `<<`, Bash will strip leading _tab characters_ from each line of the heredoc. This is useful for indenting the heredoc content within your script without affecting the output. ```bash -# Note, the leading whitespace is tabs only! +# Note, the leading whitespace is tab characters only, not spaces! cat <<- END This line has a leading tab. This line also has a leading tab. @@ -120,7 +121,12 @@ This line has a leading tab. This line also has a leading tab. ``` -* Literal Content: Quoting the delimiter (single or double quotes) prevents parameter expansion, command substitution, and arithmetic expansion within the heredoc. +The author doesn't recommend this usage. +While it does improve the readability of the script, it's too easy to accidentally replace the tab characters with spaces and it's too hard to spot the difference. + +### Literal Content + +Quoting the delimiter (single or double quotes) prevents parameter expansion, command substitution, and arithmetic expansion within the heredoc. The content is taken literally. ```bash @@ -140,7 +146,6 @@ The result of $(date) is not executed. If the delimiter is not quoted, variable expansion, command substitution, and arithmetic expansion will occur. ```bash -my_var="Hello" cat < data.csv + + url='https//example.com/api/query?page=1' + looping=true + + while $looping; do + json=$( curl "$url" ) + # handle non-success response here ... + + jq -r '.results[] | [.id, .value] | @csv' <<< "$json" + + url=$( jq -r '.next_url // ""' <<< "$json" ) + if [[ "$url" == "" ]]; then + looping=false + fi + done >> data.csv + ``` + ## In Summary Here documents are a flexible and convenient way to manage multi-line input in Bash scripts. They simplify the process of embedding text and data directly within your scripts, making them more self-contained and easier to read. -They are a very useful tool for any bash scripter. + +Here strings are like here documents, but offer a simpler, more dynamic syntax. \ No newline at end of file From 06f98f4cb5992a758d5bf51bb2d4840f8072ca3e Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Fri, 25 Apr 2025 11:20:06 -0400 Subject: [PATCH 03/11] add links --- concepts/heredocs/about.md | 17 ++++++++++++++++- concepts/heredocs/links.json | 12 ++++++++++-- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md index e53ddfc1..ce872f1c 100644 --- a/concepts/heredocs/about.md +++ b/concepts/heredocs/about.md @@ -233,9 +233,24 @@ Here's a real-world application of that example: done >> data.csv ``` + Note the position of the output redirection. + All output from the while loop will be appended to the file `data.csv`. + +## Heredocs and Herestrings are Redirections + +Because they are just redirections, they can be combined with other redirections: + +```bash +cat <<< END_OF_TEXT > output.txt +This is my important text. +END_OF_TEXT + +awk '...' <<< "$my_var" >> result.csv +``` + ## In Summary Here documents are a flexible and convenient way to manage multi-line input in Bash scripts. They simplify the process of embedding text and data directly within your scripts, making them more self-contained and easier to read. -Here strings are like here documents, but offer a simpler, more dynamic syntax. \ No newline at end of file +Here strings are like here documents, but offer a simpler, more dynamic syntax. diff --git a/concepts/heredocs/links.json b/concepts/heredocs/links.json index 583a7fe8..8a417bd6 100644 --- a/concepts/heredocs/links.json +++ b/concepts/heredocs/links.json @@ -1,7 +1,15 @@ [ { - "url": "https://www.gnu.org/software/bash/manual/bash.html#Redirections", - "description": "Redirection in the manual" + "url": "https://www.gnu.org/software/bash/manual/bash.html#Here-Documents", + "description": "Here Documents in the manual" + }, + { + "url": "https://www.gnu.org/software/bash/manual/bash.html#Here-Strings", + "description": "Here Strings in the manual" + }, + { + "url": "https://mywiki.wooledge.org/BashGuide/InputAndOutput#Heredocs_And_Herestrings", + "description": "Heredocs in the Bash Guide" } ] \ No newline at end of file From f8cade644815a25f356497ab5610cc47645e444c Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Fri, 25 Apr 2025 11:22:52 -0400 Subject: [PATCH 04/11] updates to concept readme --- concepts/README.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/concepts/README.md b/concepts/README.md index cbad123e..e97eedd2 100644 --- a/concepts/README.md +++ b/concepts/README.md @@ -102,20 +102,19 @@ The suggested concept flow: 12. Here Documents - here-docs and here-strings -## More Concepts +## More Concepts to Add -x. I/O +- I/O continued - command substitution - capturing stdout and stderr - capturing stdout and stderr **into separate variables** - - `exec` and redirections - process substitutions - brace expansions and how it's different from patterns `/path/to/{foo,bar,baz}.txt` -x. option parsing with getopts +- option parsing with getopts -x. `set` command and "strict mode" +- `set` command and "strict mode" - pros and cons of - `set -e` From 03d53eeac24d617870088b1a598a8ccd1bf27d69 Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Fri, 25 Apr 2025 11:25:34 -0400 Subject: [PATCH 05/11] add heredocs to syllabus document --- docs/SYLLABUS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/SYLLABUS.md b/docs/SYLLABUS.md index 570c765c..3474ec1c 100644 --- a/docs/SYLLABUS.md +++ b/docs/SYLLABUS.md @@ -16,6 +16,7 @@ While the learning exercises are still incomplete, most of the concept documenta * [Pipelines and Command Lists](https://exercism.org/tracks/bash/concepts/pipelines) * [Functions](https://exercism.org/tracks/bash/concepts/functions) * [Redirection](https://exercism.org/tracks/bash/concepts/redirection) + * [Here Documents](https://exercism.org/tracks/bash/concepts/heredocs) There will be more. Check the "**What's going on with Bash**" section on the [Bash track page](https://exercism.org/tracks/bash) periodically. From f2247d07d389932cf8d5ae39963c0fb41ff35e92 Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Sat, 26 Apr 2025 10:45:01 -0400 Subject: [PATCH 06/11] review suggestions --- concepts/heredocs/.meta/config.json | 3 +- concepts/heredocs/about.md | 114 ++++++++++++++++------------ concepts/heredocs/links.json | 1 - 3 files changed, 67 insertions(+), 51 deletions(-) diff --git a/concepts/heredocs/.meta/config.json b/concepts/heredocs/.meta/config.json index 8d62b522..af23605b 100644 --- a/concepts/heredocs/.meta/config.json +++ b/concepts/heredocs/.meta/config.json @@ -6,6 +6,5 @@ "IsaacG", "kotp" ], - "blurb": "Here Documents provide a redirection that input from an inline document." + "blurb": "Here Documents redirect an inline document to the standard input of a command." } - \ No newline at end of file diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md index ce872f1c..16163362 100644 --- a/concepts/heredocs/about.md +++ b/concepts/heredocs/about.md @@ -1,17 +1,18 @@ # About Here Documents -In Bash scripting, a "here document" (or "heredoc") is a way to redirect multiple lines of input to a command or program, as if you were typing them directly into the terminal. +In Bash scripting, a "here document" (or "heredoc") redirects multiple lines of input to a command or program, as if you were typing them directly into the terminal. It's a powerful tool for embedding multi-line text within your scripts without needing external files or complex string manipulation. ## Key Features and Syntax -1. Delimiter: A heredoc starts with the `<<` operator followed by a delimiter word (often called the "marker" or "terminator"). - This delimiter can be any word you choose, but it's common to use something like `EOF`, `EOF`, `END`, or `TEXT` for clarity. - For more readable code, you can add the purpose of the document to the delimiter, for example `END_INSTALLATION_INSTRUCTIONS`. +1. Delimiter -- A heredoc starts with the `<<` operator followed by a delimiter word (often called the "marker" or "terminator"). + This delimiter can be any word you choose, but it's common to use something like `EOF`, `END`, or `TEXT` for clarity. + For more readable code, you can use something descriptive as the delimiter, for example `END_INSTALLATION_INSTRUCTIONS`. -1. Content: After the initial `<< DELIMITER`, you write the content you want to redirect. This can be multiple lines of text, code, or anything else. +1. Content -- After the initial `<< DELIMITER`, you write the content you want to redirect. + This can be multiple lines of text, code, or anything else. -1. Termination: The heredoc ends when the delimiter word appears again on a line by itself, with no leading or trailing spaces. +1. Termination -- The heredoc ends when the delimiter word appears again on a line by itself, with no leading or trailing whitespace. ## Basic Syntax @@ -26,11 +27,11 @@ DELIMITER ## How it Works -* Bash reads everything between the `<< DELIMITER` and the `DELIMITER` on its own line. -* This content is then treated as standard input (`stdin`) for the command. +* Bash reads all the lines between the starting `<< DELIMITER` and the ending `DELIMITER`. +* This content is then treated as standard input (`stdin`) to the command. * The command processes this input as if it were coming from the keyboard. -## Example 1: Simple Text Output +### Example 1: Simple Text Output ```bash cat << EOF @@ -56,7 +57,7 @@ In this example: * `EOF` on its own line ends the heredoc. * `cat` then outputs the content it received. -## Example 2: Using with wc (Word Count) +### Example 2: Using with wc (Word Count) ```bash wc -l << END @@ -72,12 +73,12 @@ Output: 3 ``` -Here, `wc -l` counts the number of lines. +Here, `wc -l` counts the number of lines. The heredoc provides the three lines as input. -## Example 3: Passing to a Script +### Example 3: Passing data to a script -The script: +The script: ```bash #!/usr/bin/env bash @@ -88,7 +89,6 @@ while IFS= read -r line; do done ``` - Call the script from an interactive bash prompt with a heredoc: ```bash @@ -99,40 +99,46 @@ Item 3 MY_DATA ``` +Output: + +```plaintext +Processing: Item 1 +Processing: Item 2 +Processing: Item 3 +``` + ## Variations and Advanced Features -### Stripping Leading Tabs +### Literal Content -If you use `<<-` instead of `<<`, Bash will strip leading _tab characters_ from each line of the heredoc. -This is useful for indenting the heredoc content within your script without affecting the output. +Quoting the delimiter (single or double quotes) _prevents_ parameter expansion, command substitution, and arithmetic expansion within the heredoc. +The content is taken literally. + +If the delimiter is not quoted (this is the typical usage), variable expansion, command substitution, and arithmetic expansion will occur. ```bash -# Note, the leading whitespace is tab characters only, not spaces! -cat <<- END - This line has a leading tab. - This line also has a leading tab. -END +cat <> result.csv ## In Summary -Here documents are a flexible and convenient way to manage multi-line input in Bash scripts. -They simplify the process of embedding text and data directly within your scripts, making them more self-contained and easier to read. +Here documents are a flexible and convenient way to manage multi-line input in Bash scripts. +They simplify the process of embedding text and data directly within your scripts, making them more self-contained and easier to read. Here strings are like here documents, but offer a simpler, more dynamic syntax. diff --git a/concepts/heredocs/links.json b/concepts/heredocs/links.json index 8a417bd6..2a4d4742 100644 --- a/concepts/heredocs/links.json +++ b/concepts/heredocs/links.json @@ -12,4 +12,3 @@ "description": "Heredocs in the Bash Guide" } ] - \ No newline at end of file From efc991ff765b3407b5b3ceb24b04bb0a4f61e080 Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Sat, 26 Apr 2025 13:12:27 -0400 Subject: [PATCH 07/11] review suggestions --- concepts/heredocs/about.md | 54 +++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md index 16163362..5ffd41c4 100644 --- a/concepts/heredocs/about.md +++ b/concepts/heredocs/about.md @@ -5,14 +5,14 @@ It's a powerful tool for embedding multi-line text within your scripts without n ## Key Features and Syntax -1. Delimiter -- A heredoc starts with the `<<` operator followed by a delimiter word (often called the "marker" or "terminator"). +1. Delimiter: a heredoc starts with the `<<` operator followed by a delimiter word (often called the "marker" or "terminator"). This delimiter can be any word you choose, but it's common to use something like `EOF`, `END`, or `TEXT` for clarity. For more readable code, you can use something descriptive as the delimiter, for example `END_INSTALLATION_INSTRUCTIONS`. -1. Content -- After the initial `<< DELIMITER`, you write the content you want to redirect. +1. Content: after the initial `<< DELIMITER`, you write the content you want to redirect. This can be multiple lines of text, code, or anything else. -1. Termination -- The heredoc ends when the delimiter word appears again on a line by itself, with no leading or trailing whitespace. +1. Termination: the heredoc ends when the delimiter word appears again on a line by itself, with no leading or trailing whitespace. ## Basic Syntax @@ -111,13 +111,11 @@ Processing: Item 3 ### Literal Content -Quoting the delimiter (single or double quotes) _prevents_ parameter expansion, command substitution, and arithmetic expansion within the heredoc. -The content is taken literally. - -If the delimiter is not quoted (this is the typical usage), variable expansion, command substitution, and arithmetic expansion will occur. +Bash performs variable expansion, command substitution, and arithmetic expansion within a heredoc. +In this sense, heredocs act like double quoted strings. ```bash -cat < data.csv +```bash +# initialize the output CSV file +echo "ID,VALUE" > data.csv - url='https//example.com/api/query?page=1' - looping=true +url='https//example.com/api/query?page=1' - while $looping; do - json=$( curl "$url" ) - # handle non-success response here ... +while true; do + json=$( curl "$url" ) - jq -r '.results[] | [.id, .value] | @csv' <<< "$json" + # convert the results part of the response into CSV + jq -r '.results[] | [.id, .value] | @csv' <<< "$json" - url=$( jq -r '.next_url // ""' <<< "$json" ) - if [[ "$url" == "" ]]; then - looping=false - fi - done >> data.csv - ``` + # get the URL for the next page + url=$( jq -r '.next_url // ""' <<< "$json" ) + if [[ "$url" == "" ]]; then + break + fi +done >> data.csv +``` - Note the position of the output redirection. - All output from the while loop will be appended to the file `data.csv`. +Note the position of the output redirection. +All output from the while loop will be appended to the file `data.csv`. ## Heredocs and Herestrings are Redirections From 1debd1460042f11b4fbaa961718fa0704a9f4fc4 Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Sat, 26 Apr 2025 13:24:55 -0400 Subject: [PATCH 08/11] a more active voice --- concepts/heredocs/about.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md index 5ffd41c4..43f34020 100644 --- a/concepts/heredocs/about.md +++ b/concepts/heredocs/about.md @@ -28,7 +28,7 @@ DELIMITER ## How it Works * Bash reads all the lines between the starting `<< DELIMITER` and the ending `DELIMITER`. -* This content is then treated as standard input (`stdin`) to the command. +* Bash connects this content to the command's standard input. * The command processes this input as if it were coming from the keyboard. ### Example 1: Simple Text Output @@ -130,7 +130,7 @@ The current date is Thu Apr 24 13:47:32 EDT 2025 Two plus two is 4 ``` -Quoting the delimiter (single or double quotes) prevents these expansions. +When the delimiter is quoting (using single or double quotes), these expansions are prevented. The content is taken literally. This is like single quoted strings. @@ -157,10 +157,11 @@ This is useful for indenting the heredoc content within your script without affe ```bash # Note, the leading whitespace is tab characters only, not spaces! +# and the ending delimiter can have leading tabs as well cat <<- END - This line has a leading tab. - This line also has a leading tab. -END + This line has a leading tab. + This line also has a leading tab. + END ``` The output is printed without the leading tabs: @@ -171,7 +172,7 @@ This line also has a leading tab. ``` The author doesn't recommend this usage. -While it does improve the readability of the script, it's too easy to accidentally replace the tab characters with spaces and it's too hard to spot the difference. +While it can improve the readability of the script, it's too easy to accidentally replace the tab characters with spaces and it's too hard to spot the difference. ## When to Use Here Documents From 3b19e82ef86d55b729db76fcb68bd0037d1a48fb Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Mon, 28 Apr 2025 16:05:09 -0400 Subject: [PATCH 09/11] Adding a real-world example and possible drawbacks. Also replacing the image in the concept readme with a mermaid diagram. --- concepts/README.md | 21 +++++++++++-- concepts/heredocs/about.md | 60 +++++++++++++++++++++++++++++++------- 2 files changed, 67 insertions(+), 14 deletions(-) diff --git a/concepts/README.md b/concepts/README.md index e97eedd2..5c92c4bd 100644 --- a/concepts/README.md +++ b/concepts/README.md @@ -2,9 +2,24 @@ The [plan](http://forum.exercism.org/t/bash-syllabus-planning/11952) -The suggested concept flow: - -[![bash syllabus concept flowchart](https://glennj.github.io/img/bash.syllabus.flow.png)](http://forum.exercism.org/t/bash-syllabus-flow/15038) +## Concept Flow: + +```mermaid +erDiagram +"Commands and Arguments" ||--|| Variables : "" +Variables ||--|| "The Importance of Quoting" : "" +"The Importance of Quoting" ||--|| Conditionals : "" +"The Importance of Quoting" ||--|| Arrays : "" +"The Importance of Quoting" ||--|| "Pipelines and Command Lists" : "" +Conditionals ||--|| Arithmetic : "" +Conditionals ||--|| Looping : "" +Arrays ||--|| "More About Arrays" : "" +"Pipelines and Command Lists" ||--|| Functions : "" +Functions ||--|| Redirection : "" +Redirection ||..|| "Command Substitution" : TODO +Redirection ||--|| "Here Documents" : "" +"Command Substitution" ||..|| "Process Substitution" : TODO +``` 1. Basic syntax: commands and arguments diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md index 43f34020..9471866a 100644 --- a/concepts/heredocs/about.md +++ b/concepts/heredocs/about.md @@ -152,27 +152,34 @@ Two plus two is calculated by $((2 + 2)) ### Stripping Leading Tabs -If you use `<<-` (with a trailing hyphen) instead of `<<`, Bash will strip leading _tab characters_ from each line of the heredoc. +If you use `<<-` (with a trailing hyphen) instead of `<<`, Bash will strip any leading _tab characters_ from each line of the heredoc. This is useful for indenting the heredoc content within your script without affecting the output. ```bash # Note, the leading whitespace is tab characters only, not spaces! -# and the ending delimiter can have leading tabs as well +# The ending delimiter can have leading tabs as well. cat <<- END - This line has a leading tab. - This line also has a leading tab. - END + This line has 1 leading tab. + This line has a leading tab and some spaces. + This line 2 leading tabs. + END ``` -The output is printed without the leading tabs: +The output is printed with all the leading tabs removed: ```plaintext -This line has a leading tab. -This line also has a leading tab. +This line has 1 leading tab. + This line has a leading tab and some spaces. +This line 2 leading tabs. ``` +~~~~exercism/caution The author doesn't recommend this usage. -While it can improve the readability of the script, it's too easy to accidentally replace the tab characters with spaces and it's too hard to spot the difference. +While it can improve the readability of the script, + +1. it's too easy to accidentally replace the tab characters with spaces (your editor may do this automatically), and +1. it's too hard to spot the difference between spaces and tabs. +~~~~ ## When to Use Here Documents @@ -182,6 +189,37 @@ While it can improve the readability of the script, it's too easy to accidentall * Scripting interactions: simulating user input for interactive programs. * Avoiding external files: when you want to avoid creating temporary files. +A typical usage might be to provide some help text: + +```bash +#!/usr/bin/env bash + +usage() { + cat << END_USAGE +Refresh database tables. + +usage: $(basename "$0") [-h|--help] [-A|--no-archive] + +where: --no-archive flag will _skip_ the archive jobs +END_USAGE +} + +# ... parsing command line options here ... + +if [[ $flag_help == true" ]]; then + usage + exit 0 +fi +``` + +## Possible Drawbacks + +* Large embedded documents can make your code harder to read. + It can be better to deploy your script with separate documentation files. +* Here documents can break the flow of the code. + You might be in a deeply nested section of code, and you want to pass some text to a program. + The heredoc's indentation can look jarring compared to the surrounding code. + ## Here Strings Like heredocs, _here strings_ provide input to a command. @@ -255,9 +293,9 @@ done >> data.csv Note the position of the output redirection. All output from the while loop will be appended to the file `data.csv`. -## Heredocs and Herestrings are Redirections +## Heredocs and Herestrings as Redirection -Because they are just redirections, they can be combined with other redirections: +Because these are just forms of redirection, they can be combined with other redirection operations: ```bash cat <<< END_OF_TEXT > output.txt From 07902e0d14277fb5c87c03d11427455ff1510ab5 Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Mon, 28 Apr 2025 17:02:16 -0400 Subject: [PATCH 10/11] introduce "heresstring" word --- concepts/heredocs/about.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md index 9471866a..16252c89 100644 --- a/concepts/heredocs/about.md +++ b/concepts/heredocs/about.md @@ -222,8 +222,8 @@ fi ## Here Strings -Like heredocs, _here strings_ provide input to a command. -However, while heredocs are given as a block of text, here strings are given as a single string of text. +Like here documents, _here strings_ (or "herestrings") provide input to a command. +However, while heredocs are given as a block of text, herestrings are given as a single string of text. Here strings use the `<<< "text"` syntax. ```bash @@ -307,7 +307,7 @@ awk '...' <<< "$my_var" >> result.csv ## In Summary -Here documents are a flexible and convenient way to manage multi-line input in Bash scripts. +Here documents (or "heredocs") are a flexible and convenient way to manage multi-line input in Bash scripts. They simplify the process of embedding text and data directly within your scripts, making them more self-contained and easier to read. -Here strings are like here documents, but offer a simpler, more dynamic syntax. +Here strings (or "herestrings") are like here documents, but offer a simpler, more dynamic syntax. From 8e1a6714b6ef352ab1456236e61c6f942069f25b Mon Sep 17 00:00:00 2001 From: Glenn Jackman Date: Tue, 29 Apr 2025 09:28:27 -0400 Subject: [PATCH 11/11] review comments, and populating intro doc --- concepts/heredocs/about.md | 12 +- concepts/heredocs/introduction.md | 312 +++++++++++++++++++++++++++++- 2 files changed, 317 insertions(+), 7 deletions(-) diff --git a/concepts/heredocs/about.md b/concepts/heredocs/about.md index 16252c89..b64141bb 100644 --- a/concepts/heredocs/about.md +++ b/concepts/heredocs/about.md @@ -177,8 +177,8 @@ This line 2 leading tabs. The author doesn't recommend this usage. While it can improve the readability of the script, -1. it's too easy to accidentally replace the tab characters with spaces (your editor may do this automatically), and -1. it's too hard to spot the difference between spaces and tabs. +1. it's easy to accidentally replace the tab characters with spaces (your editor may do this automatically), and +1. it's hard to spot the difference between spaces and tabs. ~~~~ ## When to Use Here Documents @@ -198,7 +198,7 @@ usage() { cat << END_USAGE Refresh database tables. -usage: $(basename "$0") [-h|--help] [-A|--no-archive] +usage: ${0##*/} [-h|--help] [-A|--no-archive] where: --no-archive flag will _skip_ the archive jobs END_USAGE @@ -206,7 +206,7 @@ END_USAGE # ... parsing command line options here ... -if [[ $flag_help == true" ]]; then +if [[ $flag_help == "true" ]]; then usage exit 0 fi @@ -215,9 +215,9 @@ fi ## Possible Drawbacks * Large embedded documents can make your code harder to read. - It can be better to deploy your script with separate documentation files. + It can be better to deploy your script with documentation in separate files. * Here documents can break the flow of the code. - You might be in a deeply nested section of code, and you want to pass some text to a program. + You might be in a deeply nested section of code when you want to pass some text to a program. The heredoc's indentation can look jarring compared to the surrounding code. ## Here Strings diff --git a/concepts/heredocs/introduction.md b/concepts/heredocs/introduction.md index d856344e..5a2446a8 100644 --- a/concepts/heredocs/introduction.md +++ b/concepts/heredocs/introduction.md @@ -1,3 +1,313 @@ # Introduction to Here Documents -TODO +In Bash scripting, a "here document" (or "heredoc") redirects multiple lines of input to a command or program, as if you were typing them directly into the terminal. +It's a powerful tool for embedding multi-line text within your scripts without needing external files or complex string manipulation. + +## Key Features and Syntax + +1. Delimiter: a heredoc starts with the `<<` operator followed by a delimiter word (often called the "marker" or "terminator"). + This delimiter can be any word you choose, but it's common to use something like `EOF`, `END`, or `TEXT` for clarity. + For more readable code, you can use something descriptive as the delimiter, for example `END_INSTALLATION_INSTRUCTIONS`. + +1. Content: after the initial `<< DELIMITER`, you write the content you want to redirect. + This can be multiple lines of text, code, or anything else. + +1. Termination: the heredoc ends when the delimiter word appears again on a line by itself, with no leading or trailing whitespace. + +## Basic Syntax + +```bash +command << DELIMITER + Content line 1 + Content line 2 + ... + Content line N +DELIMITER +``` + +## How it Works + +* Bash reads all the lines between the starting `<< DELIMITER` and the ending `DELIMITER`. +* Bash connects this content to the command's standard input. +* The command processes this input as if it were coming from the keyboard. + +### Example 1: Simple Text Output + +```bash +cat << EOF +This is the first line. +This is the second line. +This is the third line. +EOF +``` + +Output: + +```plaintext +This is the first line. +This is the second line. +This is the third line. +``` + +In this example: + +* `cat` is the command. +* `<< EOF` starts the heredoc with `EOF` as the delimiter. +* The three lines of text are the content. +* `EOF` on its own line ends the heredoc. +* `cat` then outputs the content it received. + +### Example 2: Using with wc (Word Count) + +```bash +wc -l << END +Line 1 +Line 2 +Line 3 +END +``` + +Output: + +```plaintext +3 +``` + +Here, `wc -l` counts the number of lines. +The heredoc provides the three lines as input. + +### Example 3: Passing data to a script + +The script: + +```bash +#!/usr/bin/env bash + +# Script to process input +while IFS= read -r line; do + echo "Processing: $line" +done +``` + +Call the script from an interactive bash prompt with a heredoc: + +```bash +./your_script << MY_DATA +Item 1 +Item 2 +Item 3 +MY_DATA +``` + +Output: + +```plaintext +Processing: Item 1 +Processing: Item 2 +Processing: Item 3 +``` + +## Variations and Advanced Features + +### Literal Content + +Bash performs variable expansion, command substitution, and arithmetic expansion within a heredoc. +In this sense, heredocs act like double quoted strings. + +```bash +cat << EOF +The value of HOME is $HOME +The current date is $(date) +Two plus two is $((2 + 2)) +EOF +``` + +Output: + +```plaintext +The value of HOME is /home/glennj +The current date is Thu Apr 24 13:47:32 EDT 2025 +Two plus two is 4 +``` + +When the delimiter is quoting (using single or double quotes), these expansions are prevented. +The content is taken literally. +This is like single quoted strings. + +```bash +cat << 'EOF' +The value of $HOME is not expanded here. +The result of $(date) is not executed. +Two plus two is calculated by $((2 + 2)) +EOF +``` + +Output: + +```plaintext +The value of $HOME is not expanded here. +The result of $(date) is not executed. +Two plus two is calculated by $((2 + 2)) +``` + +### Stripping Leading Tabs + +If you use `<<-` (with a trailing hyphen) instead of `<<`, Bash will strip any leading _tab characters_ from each line of the heredoc. +This is useful for indenting the heredoc content within your script without affecting the output. + +```bash +# Note, the leading whitespace is tab characters only, not spaces! +# The ending delimiter can have leading tabs as well. +cat <<- END + This line has 1 leading tab. + This line has a leading tab and some spaces. + This line 2 leading tabs. + END +``` + +The output is printed with all the leading tabs removed: + +```plaintext +This line has 1 leading tab. + This line has a leading tab and some spaces. +This line 2 leading tabs. +``` + +~~~~exercism/caution +The author doesn't recommend this usage. +While it can improve the readability of the script, + +1. it's easy to accidentally replace the tab characters with spaces (your editor may do this automatically), and +1. it's hard to spot the difference between spaces and tabs. +~~~~ + +## When to Use Here Documents + +* Multi-line input: when you need to provide multiple lines of text to a command. +* Configuration files: embedding small configuration snippets within a script. +* Generating code: creating code on the fly within a script. +* Scripting interactions: simulating user input for interactive programs. +* Avoiding external files: when you want to avoid creating temporary files. + +A typical usage might be to provide some help text: + +```bash +#!/usr/bin/env bash + +usage() { + cat << END_USAGE +Refresh database tables. + +usage: ${0##*/} [-h|--help] [-A|--no-archive] + +where: --no-archive flag will _skip_ the archive jobs +END_USAGE +} + +# ... parsing command line options here ... + +if [[ $flag_help == "true" ]]; then + usage + exit 0 +fi +``` + +## Possible Drawbacks + +* Large embedded documents can make your code harder to read. + It can be better to deploy your script with documentation in separate files. +* Here documents can break the flow of the code. + You might be in a deeply nested section of code when you want to pass some text to a program. + The heredoc's indentation can look jarring compared to the surrounding code. + +## Here Strings + +Like here documents, _here strings_ (or "herestrings") provide input to a command. +However, while heredocs are given as a block of text, herestrings are given as a single string of text. +Here strings use the `<<< "text"` syntax. + +```bash +tr 'a-z' 'A-Z' <<< "upper case this string" +``` + +Output: + +```plaintext +UPPER CASE THIS STRING +``` + +Unlike heredocs, no ending delimiter is required. + +### Why Use Here Strings? + +A pipeline can be used instead of a here string: + +```bash +echo "upper case this string" | tr 'a-z' 'A-Z' +``` + +So why use a here string? + +Consider the case where you get the string as output from a long-running computation, and you want to feed the result to two separate commands. +Using pipelines, you have to execute the computation twice: + +```bash +some_long_running_calculation | first_command +some_long_running_calculation | second_command +``` + +A more efficient approach is to capture the output of the computation (using command substutition), and use here strings to provide input to the two subsequent commands: + +```bash +result=$( some_long_running_calculation ) +first_command <<< "$result" +second_command <<< "$result" +``` + +Here's a real-world application of that example: + +* capture the JSON response to a REST API query (that is paginated), +* provide the JSON data to a jq program to parse the results and output that to a file, and then +* provide the JSON data to another jq program to determine the URL of the next query. + +```bash +# initialize the output CSV file +echo "ID,VALUE" > data.csv + +url='https//example.com/api/query?page=1' + +while true; do + json=$( curl "$url" ) + + # convert the results part of the response into CSV + jq -r '.results[] | [.id, .value] | @csv' <<< "$json" + + # get the URL for the next page + url=$( jq -r '.next_url // ""' <<< "$json" ) + if [[ "$url" == "" ]]; then + break + fi +done >> data.csv +``` + +Note the position of the output redirection. +All output from the while loop will be appended to the file `data.csv`. + +## Heredocs and Herestrings as Redirection + +Because these are just forms of redirection, they can be combined with other redirection operations: + +```bash +cat <<< END_OF_TEXT > output.txt +This is my important text. +END_OF_TEXT + +awk '...' <<< "$my_var" >> result.csv +``` + +## In Summary + +Here documents (or "heredocs") are a flexible and convenient way to manage multi-line input in Bash scripts. +They simplify the process of embedding text and data directly within your scripts, making them more self-contained and easier to read. + +Here strings (or "herestrings") are like here documents, but offer a simpler, more dynamic syntax.