I found some ways to pass external shell variables to an awk script, but I'm confused about ' and ".

First, I tried with a shell script:

$ v=123test

$ echo $v

123test

$ echo "$v"

123test

Then tried awk:

$ awk 'BEGIN{print "'$v'"}'

$ 123test

$ awk 'BEGIN{print '"$v"'}'

$ 123

Why is the difference?

Lastly I tried this:

$ awk 'BEGIN{print " '$v' "}'

$  123test

$ awk 'BEGIN{print ' "$v" '}'

awk: cmd. line:1: BEGIN{print

awk: cmd. line:1:             ^ unexpected newline or end of string 

I'm confused about this.

Getting shell variables into awk

may be done in several ways. Some are better than others. This should cover most of them. If you have a comment, please leave below.

Using -v (The best way, most portable)

Use the -v option: (P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar=)

variable="line onenline two"

awk -v var="$variable" 'BEGIN {print var}'

line one

line two

This should be compatible with most awk, and the variable is available in the BEGIN block as well:

If you have multiple variables:

awk -v a="$var1" -v b="$var2" 'BEGIN {print a,b}'

Warning. As Ed Morton writes, escape sequences will be interpreted so t becomes a real tab and not t if that is what you search for. Can be solved by using ENVIRON or access it via ARGV

Variable after code block

Here we get the variable after the awk code. This will work fine as long as you do not need the variable in the BEGIN block:

variable="line onenline two"

echo "input data" | awk '{print var}' var="${variable}"

or

awk '{print var}' var="${variable}" file

This also works with multiple variables
awk '{print a,b,$0}' a="$var1" b="$var2" file

Using variable this way does not work in BEGIN block:

echo "input data" | awk 'BEGIN {print var}' var="${variable}"

Here-string

Variable can also be added to awk using a here-string from shells that support them (including Bash):

awk '{print $0}' <<< "$variable"

test

This is the same as:

printf '%s' "$variable" | awk '{print $0}'

P.S. this treats the variable as a file input.

ENVIRON input

As TrueY writes, you can use the ENVIRON to print Environment Variables.
Setting a variable before running AWK, you can print it out like this:

X=MyVar awk 'BEGIN{print ENVIRON["X"],ENVIRON["SHELL"]}'

MyVar /bin/bash

ARGV input

As Steven Penny writes, you can use ARGV to get the data into awk:

v="my data"

awk 'BEGIN {print ARGV[1]}' "$v"

my data

To get the data into the code itself, not just the BEGIN:

v="my data"

echo "test" | awk 'BEGIN{var=ARGV[1];ARGV[1]=""} {print var, $0}' "$v"

my data test

Variable within the code: USE WITH CAUTION

You can use a variable within the awk code, but it's messy and hard to read, and as Charles Duffy points out, this version may also be a victim of code injection. If someone adds bad stuff to the variable, it will be executed as part of the awk code.

This works by extracting the variable within the code, so it becomes a part of it.

If you want to make an awk that changes dynamically with use of variables, you can do it this way, but DO NOT use it for normal variables.

variable="line onenline two"

awk 'BEGIN {print "'"$variable"'"}'

line one

line two

Here is an example of code injection:

variable='line onenline two" ; for (i=1;i<=1000;++i) print i"'

awk 'BEGIN {print "'"$variable"'"}'

line one

line two

1

2

3

.

.

1000

You can add lots of commands to awk this way. Even make it crash with non valid commands.

Extra info:

Use of double quote

It's always good to double quote variable "$variable"

If not, multiple lines will be added as a long single line.

Example:

var="Line one

This is line two"



echo $var

Line one This is line two



echo "$var"

Line one

This is line two

Other errors you can get without double quote:

variable="line onenline two"

awk -v var=$variable 'BEGIN {print var}'

awk: cmd. line:1: onenline

awk: cmd. line:1:    ^ backslash not last character on line

awk: cmd. line:1: onenline

awk: cmd. line:1:    ^ syntax error

And with single quote, it does not expand the value of the variable:

awk -v var='$variable' 'BEGIN {print var}'

$variable

More info about AWK and variables

Read this faq.

It seems that the good-old ENVIRON awk built-in hash is not mentioned at all. An example of its usage:

$ X=Solaris awk 'BEGIN{print ENVIRON["X"], ENVIRON["TERM"]}'

Solaris rxvt

Use either of these depending how you want backslashes in the shell variables handled (avar is an awk variable, svar is a shell variable):

awk -v avar="$svar" '... avar ...' file

awk 'BEGIN{avar=ARGV[1];ARGV[1]=""}... avar ...' "$svar" file

See http://cfajohnson.com/shell/cus-faq-2.html#Q24 for details and other options. The first method above is almost always your best option and has the most obvious semantics.

You could pass in the command-line option -v with a variable name (v) and a value (=) of the environment variable ("${v}"):

% awk -vv="${v}" 'BEGIN { print v }'

123test

Or to make it clearer (with far fewer vs):

% environment_variable=123test

% awk -vawk_variable="${environment_variable}" 'BEGIN { print awk_variable }'

123test