Handle Bash config file variables like a pro

A good practice when writing large Bash scripts, is to separate the config data from the actual processor, hence the humble config file.

The Config File

After playing around with various config file types (ini, json, yaml, txt), I found the YAML config file to be most flexible due to its ability to insert comments and nested structure. The only downside of Yaml is its fanatical 2-space requirement, meaning that you need to run ‘yamllint’ or some other linter to make sure the Yaml has no errors, otherwise you will get runtime errors

Another thing to consider when writing Bash or any shell, is that shells donot handle nested data structure well, hence the need for higher languages like Python or Ruby to parse YAML or JSON files.

If you want to keep things relatively sane, all-Bash and dependency-free then here is a good method to handle your config parameters

Lets use this sample config file ‘config.yaml’

---
#Company ABC Config
company:
  address: 123 new city
  datacenters:
    CA:
      North: 'San Francisco'
      South: 'San Diego'
      East:
    OH:
      West: 'Cincinati'
      East: 'Cleveland'
  phone:
    # memo: need to change US number to new area code
    US: '1-800-222-3333'
    EU: '2-1234-3433-33444'

This looks painful but its actually very simple, the data is arranged logically with ‘Company’ being the top node, with Address, Datacenters, Phone being the 3 subnodes, and each subnode with its own subnodes, all separated by 2 spaces. You can also add comments to the config unlike JSON.

Parsing YAML data with pure Bash

To get all these values as parameters into Bash without using Python or Ruby and without explicitly declaring each parameter, use the following Bash function,

function parse_yaml {
  local prefix=$2
  local s=’[[:space:]]*’ w=’[a-zA-Z0–9_]*’ fs=$(echo @|tr @ ‘\034’)
  sed -ne “s|^\($s\)\($w\)$s:$s\”\(.*\)\”$s\$|\1$fs\2$fs\3|p” \
  -e “s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p” $1 |
  awk -F$fs ‘{
    indent = length($1)/2;
    vname[indent] = $2;
    for (i in vname) {if (i > indent) {delete vname[i]}}
    if (length($3) > 0) {
      vn=””; for (i=0; i<indent; i++) {vn=(vn)(vname[i])(“_”)}
      printf(“%s%s%s=\”%s\”\n”, “‘$prefix’”,vn, $2, $3);
    }
   }’
}

Add this function to your Bash script

To get all your parameters from the config.yaml in 1 shot, add this line the script (add this after the function declaration),

eval $(parse_yaml config.yaml)

This will read your config file, and add each data node as a parameter to your script, so for example, you will end up with following parameters (it will add an underscore to separate each YAML level)

echo $company_address # outputs “123 new city”
echo $company_phone_EU #outputs “2–1234–3433–33444”

The beauty of this approach is that all your parameters are inside your Bash script without explicit individual declaration or messy iteration loops. You now have all your variables and are ready to work with them.

The only caveat with the “parse_yaml” function is that it cant handle YAML lists, for example

company:
  employees:
    - joe
    - mary
    - bob

This is a limitation of Bash and a workaround is to put all your list items into an string like this,

company:
  employees: 'joe, mary, bob'

Ok, now we are ready..

But wait.. what if the data is missing or someone removed a value from config.yaml? Heres a simple method to check for null or empty values:

Verify your Parameter Values

add a new function to your Bash script to verify each param,

function verify_param() {
  [[ -z “${!1}” ]] && echo $1 value not present in config.yaml, exiting.
}

To verify a param:

verify_param company_datacenters_CA_East

Since this param is null in our config file, it will give you an error when your run your script

“company_datacenters_CA_East value is not present in config.yaml”

Thats it, you can now handle all your params with 2 simple functions, there is no need to initialize each parameter individually and checking for null values is a breeze.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s