Bash Notes

String Processing

I prefer to process in strings rather than files. It has been commented that Bash lacks string processing facilities. There is a constant interplay between file and string processing. Awk, Sed and Perl can be used here.

Sed can be used to process strings directly. For example to find if a substring exists in a string
local i="system abc/123 def.ghj"
e3=`echo $i | sed -n '/def/p'`
if [ $e3 == "" ]; then echo "not found"...


Converting between files and strings.
# Write the file to a string.
declare filetostringval
function filetostring()
{
    local filetostringi
    filetostringval="";
    for filetostringi in `cat $1`; do
        filetostringval=$filetostringval" "$filetostringi;
    done;
}
...
filetostring temp.txt
local s=$filetostringvar
Converting a string to a file.
`cat s` > temp2.txt

This uses a global variable to pass out the resulting data. I prefix the function name to give an object orientated environment, especially to avoid variable name collisions.

Here is function to trim leading and trailing whitespace.
# Trim leading and trailing spaces from a variable. # 1: variable name.
function stringtrim
{
    local stringtrimarg="$1=\`echo \$$1 | sed -e 's/^ *//' -e 's/ *$//'\`"
    eval "$stringtrimarg"
}
...
local a=" happy "
stringtrim a
#a="happy"

This uses a reference like calling convention giving bash functions the ability to pass data out in variables.

function readfilelinebyline
{
    local aline
    while read aline; do
        echo "$aline"
    done < testscript01.txt
}
Reads lines from a file into variable aline with each iteration.

Local Variables

I have just spent a few days tracking down a bug in a bash script. The bug was only mild but no matter what I did it would not go away. It was one of those irritating bugs that worked for the majority of cases so seemed harmless and easily solved.

It would be fair to say that Bash is not a strongly typed language. So when I finally tracked down the bug it was because I used i to iterate in two functions, one which called the other and the collision only occurred in a particular case.

So while this may seem a lengthy discussion it is here to save me further grief.

Define all variables inside functions as local or with declare. By default the scope is the whole shell!

Another problem was when I had i as a local variable and called a function which had i as a global variable. It rewrote over my local variable. So be aware that functions called see the variables and can modify them.

For Loop

The for loop evaluates in the bash environment.
local i
for i in *.h *.cpp; do
    cp ${i} ${i}.txt;
done;

Finds all the files beginning with a period.
for i in `find . -name .\*`; do
   echo ${i};
done

Iterate over a list of strings.
local filetypes="png html"
local i
for i in ${filetypes}; do
    cp *.${i} temp
done;

Metacharacters can be a problem in loops. Here the output of the echo command is different from iterating over the output. The commented out code produces the desired output.
echo cat mat abc\\/
for i in `echo cat mat abc\\/`; do
# for i in `echo cat mat "abc\/"`; do
    echo ${i}
done;
produces

cat mat abc\/
cat
mat
abc/

In this case it is easier to use strings.
local i
local k="cat mat abc\/"
for i in ${k}; do
echo ${i}
done;

Using variables in loops is expected. But how variables interact is part of the bash shell. Command substitution is tricky. Here is an example where the metacharacter / must be back slashed in the sed command. Since variables are used we also need the eval command.

local arg1
local arg2
local k="cat mat abc\/"
for arg1 in ${k}; do
    arg2=${arg1}crap
    eval sed 's/${arg1}/${arg2}/g' q01.txt
done;

eval

Command substitution and command line processing use quotes, double quotes, ... to do some job but this can be complex as the parsing rules are difficult to understand.

For my sanity and yours here is a technique to program and have it work. The motivation comes from using variables with metacharacters which get interpreted differently and so expressions can not be substituted into other expressions without the meaning changing.

For example the following failed because the space character interfered with eval.
local k="zpr"
for arg1 in ${k}; do
    arg2="<a\\ href=\"${arg1}.h\">\\ ${arg1}.h<\/a>"
    eval sed 's/${arg1}.h/${arg2}/g' Makefile
done;

The issues lie in quote interpretation,... which is difficult. Instead I threw away this and got a command line instance working. Then the trick is to get the same text string as the working command line. So we are really doing the work for the interpreter by doing the substitutions ourselves in strings. This is much more sane than the standard approach.

arg3 is where the substitution occurs. Further the variable can be printed so it is known what is going to be evaluated, unlike the situation where substitution accrues and this can not be easily seen.
local k="zpr"
for arg1 in ${k}; do
    arg2="<a href=\"${arg1}.h\">${arg1}.h<\/a>"
    arg3="sed 's/${arg1}.h/${arg2}/g' Makefile"
    eval ${arg3}
done;

Another useful use for eval is with indirect references to variables. Here is a general purpose function to add a string to the end of a list of strings. Because of a possible name collision with the local variables I have prepended the functions name to the variable.
# 1: list name
# 2: value added to list
function stringlistadd
{
    local stringlistaddtemp="$1=\"\$$1 $2\""
    eval "$stringlistaddtemp"
}

Files as Variables

Unix can be fucking irritating with respect to knowing how to implement and use command options. A technology that requires a huge memory to remember every case is bad, despite its stability it should be fixed (progressively made easier through public standardization ).

As I am a simple person I have been looking at using files to store intermittent data because they seem to have advantages over pipes: what you do is clear and easy.

Consider the following with pipes if you must. Essentially the output of two commands is added together.
    local filetmp=/tmp/temp5.txt
    find . -name \*.xml > $filetmp
    find . -name \*.html >> $filetmp

Iterating over the file.
    local filetmp=/tmp/temp5.txt
    for k in `cat $filetmp`; do
        echo $k
    done

There is probably someone who knows Bash better and can do all this stuff with variables allown.

Here is another problem solved with files as variables. Find the files modified in the last k days and sort by time.

#Remove carriage returns and replace with space.
function filetostring()
{
  s="";
  local i
  for i in `cat $1`; do
    s=$s" "$i;
  done;
  echo $s;
}

function filesortbytime()
{
  echo "ls -al -t " > /tmp/t1.txt;
  find . -mtime -$1 -type f >> /tmp/t1.txt;
  filetostring /tmp/t1.txt > /tmp/t2.txt;
  bash /tmp/t2.txt;
}

The strategy is to form a string with all the arguments. Unfortunately Unix makes nested function calls extremely difficult. Something that you would think should be solved by now. Consider a better solution posted on the net.

function filesortbytime2()
{
  find . -mtime -$1 -type f | xargs ls -al -t
}

The pipe along with xargs is using the output of the first command as an input into the second command. Well I just learned something new. Thanks Unix.

Handle Too-long Command Lines
Example
ls *jpg fails
bash: /bin/ls: Argument list too long
This succeeds
find . -type f -name "*.jpg" | more
xargs takes a command and makes dynamically makes commands.
find . -mindepth 1 -maxdepth 1 -type f -name "*.jpg" | xargs -l56 -i mv {} ./d018/
I believe {} substitues the filename/s. xargs controls overflowing command line buffer, -l56 to pass maximum 56 files to mv command so xargs services do not overflow.
find . -name "*.jpg" -print0 | xargs -0 ls
With -exec this took over 20 times longer.
find . -name "*.jpg" -exec ls {} \;

for ((a=INIT; a <= LIMIT ; a++))  # Double parentheses, and "LIMIT" with no "$".
do
  echo "$bs$a"
  echo -n "$bs"      # no return
  if [ $a -lt 10 ]   # less than 10
    then echo -n 0
  fi
  echo "$a"
done     

Generate a temporary file filename.
local tempfile=/tmp/tempfile`date +%s``date +%N`.txt

Arithmetic

Surrounding expressions in double parentheses and execute in a C style. For example to set a variable i to the value of one.
(( i = 1 ))

This syntax replaced the older syntax and is much easier and sane to use. Also the square bracket test can be replaced. Heres a while loop. Notice that the if then statement has ; separators. If these are not there then the commands must be on their own lines.

  while (( $i <= 117 )) 
  do
    if (( $i == 114 )); then echo fku; fi
    echo ${i}
    (( i++ ))
  done

109
110
111
112
113
fku
114
115
116
117

The main thing is that this code is easy to change. I do not have to look up the -ne -lt or whatever command to find the right comparison operator.

Functions

function name
{
    ...
}


In bash functions are called with string arguments. Bash can return a value but it is closely associated in meaning with a programs exit status, therefore never use bash to return values in the traditional sense of a function.

Instead values are passed out of functions through global variables. I combine this with a convention on the variable names such that the function name is prefixed to the variables name. Using this convention you can turn a functional programming language into an OOP language by following the convention.

Consider the following example of a function to convert a number less than 1000 into three digits. So 20 is represented as 020 .

declare digit3number
function digit3
{
  local x
  (( x = $1 ))
  if (( x < 10 )); then digit3number=00${x}
    return
  fi

  if (( x < 100 )); then digit3number=0${x}
    return
  fi

  digit3number=${x}
}

# Example of using digit3 function.
function f2
{
  local i
  (( i = 7 ))
  while (( $i < 10 ))
  do
    digit3 ${i} 

    echo ${digit3number}

    (( i++ ))
  done

}

f2

Produced

007
008
009

Functions can be used to to tasks with a variable number of arguments. Here is the technique where the arguments to a function are iterated over.

function demo_arg_iteration
{
    local k=1
    local kmax=$#
    while (( k <= kmax ))
    do
        echo $1
        (( k++ ))
        shift
    done
}

demo_arg_iteration have a good day

Calling the script with the demo_arg_iteration
$ bash t1.sh
have
a
good
day