HANDY ONE-LINERS FOR AWK 22 July 2003
compiled by Eric Pement version 0.22
Latest version of this file is usually at:

http://www.student.northpark.edu/pemente/awk/awk1line.txt

USAGE:

Unix: awk ‘/pattern/ {print “$1″}’ # standard Unix shells
DOS/Win: awk ‘/pattern/ {print “$1″}’ # okay for DJGPP compiled
awk “/pattern/ {print \”$1\”}” # required for Mingw32

Most of my experience comes from version of GNU awk (gawk) compiled for
Win32. Note in particular that DJGPP compilations permit the awk script
to follow Unix quoting syntax ‘/like/ {“this”}’. However, the user must
know that single quotes under DOS/Windows do not protect the redirection
arrows (< , >) nor do they protect pipes (|). Both are special symbols
for the DOS/CMD command shell and their special meaning is ignored only
if they are placed within “double quotes.” Likewise, DOS/Win users must
remember that the percent sign (%) is used to mark DOS/Win environment
variables, so it must be doubled (%%) to yield a single percent sign
visible to awk.

If I am sure that a script will NOT need to be quoted in Unix, DOS, or
CMD, then I normally omit the quote marks. If an example is peculiar to
GNU awk, the command ‘gawk’ will be used. Please notify me if you find
errors or new commands to add to this list (total length under 65
characters). I usually try to put the shortest script first.

FILE SPACING:

 # double space a file
 awk '1;{print ""}'
 awk 'BEGIN{ORS="\n\n"};1'

 # double space a file which already has blank lines in it. Output file
 # should contain no more than one blank line between lines of text.
 # NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are
 # often treated as non-blank, and thus 'NF' alone will return TRUE.
 awk 'NF{print $0 "\n"}'

 # triple space a file
 awk '1;{print "\n"}'

NUMBERING AND CALCULATIONS:

 # precede each line by its line number FOR THAT FILE (left alignment).
 # Using a tab (\t) instead of space will preserve margins.
 awk '{print FNR "\t" $0}' files*

 # precede each line by its line number FOR ALL FILES TOGETHER, with tab.
 awk '{print NR "\t" $0}' files*

 # number each line of a file (number on left, right-aligned)
 # Double the percent signs if typing from the DOS command prompt.
 awk '{printf("%5d : %s\n", NR,$0)}'

 # number each line of file, but only print numbers if line is not blank
 # Remember caveats about Unix treatment of \r (mentioned above)
 awk 'NF{$0=++a " :" $0};{print}'
 awk '{print (NF? ++a " :" :"") $0}'

 # count lines (emulates "wc -l")
 awk 'END{print NR}'

 # print the sums of the fields of every line
 awk '{s=0; for (i=1; i< =NF; i++) s=s+$i; print s}'

 # add all fields in all lines and print the sum
 awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'

 # print every line after replacing each field with its absolute value
 awk '{for (i=1; i<=NF; i++) if ($i < 0) $i = -$i; print }'
 awk '{for (i=1; i<=NF; i++) $i = ($i < 0) ? -$i : $i; print }'

 # print the total number of fields ("words") in all lines
 awk '{ total = total + NF }; END {print total}' file

 # print the total number of lines that contain "Beth"
 awk '/Beth/{n++}; END {print n+0}' file

 # print the largest first field and the line that contains it
 # Intended for finding the longest string in field #1
 awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'

 # print the number of fields in each line, followed by the line
 awk '{ print NF ":" $0 } '

 # print the last field of each line
 awk '{ print $NF }'

 # print the last field of the last line
 awk '{ field = $NF }; END{ print field }'

 # print every line with more than 4 fields
 awk 'NF > 4'

 # print every line where the value of the last field is > 4
 awk '$NF > 4'

TEXT CONVERSION AND SUBSTITUTION:

 # IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
 awk '{sub(/\r$/,"");print}'   # assumes EACH line ends with Ctrl-M

 # IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
 awk '{sub(/$/,"\r");print}

 # IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
 awk 1

 # IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
 # Cannot be done with DOS versions of awk, other than gawk:
 gawk -v BINMODE="w" '1' infile >outfile

 # Use "tr" instead.
 tr -d \r outfile            # GNU tr version 1.22 or higher

 # delete leading whitespace (spaces, tabs) from front of each line
 # aligns all text flush left
 awk '{sub(/^[ \t]+/, ""); print}'

 # delete trailing whitespace (spaces, tabs) from end of each line
 awk '{sub(/[ \t]+$/, "");print}'

 # delete BOTH leading and trailing whitespace from each line
 awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}'
 awk '{$1=$1;print}'           # also removes extra space between fields

 # insert 5 blank spaces at beginning of each line (make page offset)
 awk '{sub(/^/, "     ");print}'

 # align all text flush right on a 79-column width
 awk '{printf "%79s\n", $0}' file*

 # center all text on a 79-character width
 awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*

 # substitute (find and replace) "foo" with "bar" on each line
 awk '{sub(/foo/,"bar");print}'           # replaces only 1st instance
 gawk '{$0=gensub(/foo/,"bar",4);print}'  # replaces only 4th instance
 awk '{gsub(/foo/,"bar");print}'          # replaces ALL instances in a line

 # substitute "foo" with "bar" ONLY for lines which contain "baz"
 awk '/baz/{gsub(/foo/, "bar")};{print}'

 # substitute "foo" with "bar" EXCEPT for lines which contain "baz"
 awk '!/baz/{gsub(/foo/, "bar")};{print}'

 # change "scarlet" or "ruby" or "puce" to "red"
 awk '{gsub(/scarlet|ruby|puce/, "red"); print}'

 # reverse order of lines (emulates "tac")
 awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*

 # if a line ends with a backslash, append the next line to it
 # (fails if there are multiple lines ending with backslash...)
 awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*

 # print and sort the login names of all users
 awk -F ":" '{ print $1 | "sort" }' /etc/passwd

 # print the first 2 fields, in opposite order, of every line
 awk '{print $2, $1}' file

 # switch the first 2 fields of every line
 awk '{temp = $1; $1 = $2; $2 = temp}' file

 # print every line, deleting the second field of that line
 awk '{ $2 = ""; print }'

 # print in reverse order the fields of every line
 awk '{for (i=NF; i>0; i--) printf("%s ",i);printf ("\n")}' file

 # remove duplicate, consecutive lines (emulates "uniq")
 awk 'a !~ $0; {a=$0}'

 # remove duplicate, nonconsecutive lines
 awk '! a[$0]++'                     # most concise script
 awk '!($0 in a) {a[$0];print}'      # most efficient script

 # concatenate every 5 lines of input, using a comma separator
 # between fields
 awk 'ORS=%NR%5?",":"\n"' file

SELECTIVE PRINTING OF CERTAIN LINES:

 # print first 10 lines of file (emulates behavior of "head")
 awk 'NR < 11'

 # print first line of file (emulates "head -1")
 awk 'NR>1{exit};1'

  # print the last 2 lines of a file (emulates "tail -2")
 awk '{y=x "\n" $0; x=$0};END{print y}'

 # print the last line of a file (emulates "tail -1")
 awk 'END{print}'

 # print only lines which match regular expression (emulates "grep")
 awk '/regex/'

 # print only lines which do NOT match regex (emulates "grep -v")
 awk '!/regex/'

 # print the line immediately before a regex, but not the line
 # containing the regex
 awk '/regex/{print x};{x=$0}'
 awk '/regex/{print (x=="" ? "match on line 1" : x)};{x=$0}'

 # print the line immediately after a regex, but not the line
 # containing the regex
 awk '/regex/{getline;print}'

 # grep for AAA and BBB and CCC (in any order)
 awk '/AAA/; /BBB/; /CCC/'

 # grep for AAA and BBB and CCC (in that order)
 awk '/AAA.*BBB.*CCC/'

 # print only lines of 65 characters or longer
 awk 'length > 64'

 # print only lines of less than 65 characters
 awk 'length < 64'

 # print section of file from regular expression to end of file
 awk '/regex/,0'
 awk '/regex/,EOF'

 # print section of file based on line numbers (lines 8-12, inclusive)
 awk 'NR==8,NR==12'

 # print line number 52
 awk 'NR==52'
 awk 'NR==52 {print;exit}'          # more efficient on large files

 # print section of file between two regular expressions (inclusive)
 awk '/Iowa/,/Montana/'             # case sensitive

SELECTIVE DELETION OF CERTAIN LINES:

 # delete ALL blank lines from a file (same as "grep '.' ")
 awk NF
 awk '/./'

CREDITS AND THANKS:

Special thanks to Peter S. Tillier for helping me with the first release
of this FAQ file.

For additional syntax instructions, including the way to apply editing
commands from a disk file instead of the command line, consult:

"sed & awk, 2nd Edition," by Dale Dougherty and Arnold Robbins
O'Reilly, 1997
"UNIX Text Processing," by Dale Dougherty and Tim O'Reilly
Hayden Books, 1987
"Effective awk Programming, 3rd Edition." by Arnold Robbins
O'Reilly, 2001

To fully exploit the power of awk, one must understand "regular
expressions." For detailed discussion of regular expressions, see
"Mastering Regular Expressions, 2d edition" by Jeffrey Friedl
(O'Reilly, 2002).

The manual ("man") pages on Unix systems may be helpful (try "man awk",
"man nawk", "man regexp", or the section on regular expressions in "man
ed"), but man pages are notoriously difficult. They are not written to
teach awk use or regexps to first-time users, but as a reference text
for those already acquainted with these tools.

USE OF '\t' IN awk SCRIPTS: For clarity in documentation, we have used
the expression '\t' to indicate a tab character (0x09) in the scripts.
All versions of awk, even the UNIX System 7 version should recognize
the '\t' abbreviation.

#---end of file---

Shell中的[],没太搞明白，请高手指点。

ricky@ricky-desktop:~$ a=0
ricky@ricky-desktop:~$ if [ $a ]; then
> echo "Gooooood"
> else
> echo "Baaaaaaad"
> fi
Gooooood
ricky@ricky-desktop:~$ a=1
ricky@ricky-desktop:~$ if [ $a ]; then echo "Gooooood"; else echo "Baaaaaaad"; fi
Gooooood
ricky@ricky-desktop:~$ a="string"
ricky@ricky-desktop:~$ if [ $a ]; then echo "Gooooood"; else echo "Baaaaaaad"; fi
Gooooood
ricky@ricky-desktop:~$ a=
ricky@ricky-desktop:~$ if [ $a ]; then echo "Gooooood"; else echo "Baaaaaaad"; fi
Baaaaaaad
ricky@ricky-desktop:~$ a=100
ricky@ricky-desktop:~$ if [ $a ]; then echo "Gooooood"; else echo "Baaaaaaad"; fi
Gooooood
ricky@ricky-desktop:~$ unset a
ricky@ricky-desktop:~$ if [ $a ]; then echo "Gooooood"; else echo "Baaaaaaad"; fi
Baaaaaaad
ricky@ricky-desktop:~$ 

多些几位大虾的指点，上面如果用string存在不存在来解释，那么下面怎么解释：

ricky@ricky-desktop:~$ a=1
ricky@ricky-desktop:~$ if [ ! $a ] ; then echo "Goooooood"; else echo "Baaaaaad"; fi
Baaaaaad
ricky@ricky-desktop:~$ a=0
ricky@ricky-desktop:~$ if [ ! $a ] ; then echo "Goooooood"; else echo "Baaaaaad"; fi
Baaaaaad
ricky@ricky-desktop:~$ a=true
ricky@ricky-desktop:~$ if [ ! $a ] ; then echo "Goooooood"; else echo "Baaaaaad"; fi
Baaaaaad
ricky@ricky-desktop:~$ a=false
ricky@ricky-desktop:~$ if [ ! $a ] ; then echo "Goooooood"; else echo "Baaaaaad"; fi
Baaaaaad
ricky@ricky-desktop:~$ 

在Shell中读取文件，一次一行进行处理。用for处理起来比较简单，但是如果一行中有空格分隔的话，那么处理起来如果把一行作为一个参数，用for就不行了，变通一下的做法有很多中，比如用awk。昨天在处理这个问题花了一点时间研究了一下shell读取文件的方法，如下：

#!/bin/ksh

cat file1 | while read line
do
  line2=`echo $line | sed 's/\*/\\\*/g' `
# echo "$line2"
  grep "$line2" file2 > /dev/null
  a=$?
  grep "$line2" file3 >/dev/null
  b=$?
# echo "a=$a,b=$b"
  if [[ $a == 0 ]] && [[ $b == 0 ]]
  then
    echo $line
  fi
done
ricky@ricky-desktop:~$

这个脚本是用来实现如下的需求：
1）比较三个文件中相同的部分，输出
2）每个文件中的内容中有些特殊字符，在grep的时候需要进行特殊处理，用sed进行了替换
3）因为文件内容中有空格，所以读取的时候不能用传统的for x in a b c的方法

文件内容大致如下：

ricky@ricky-desktop:~$ more file1
line1 32*256 megabyte(s)
line2 32*256 megabyte(s)
line3 32*256 megabyte(s)

今天跟老铁在Solaris上调试一个小的脚本，最初的需求是希望用Shell来实现修改系统时间，随机提前N秒或者推迟N秒。这个在Linux中是非常容易做到的，因为Linux的date函数提供了一个–date选项，可以支持秒，分钟，小时，甚至天或者月的修改，用法如下。

1) 取之前的时间：
date -d “a day ago” 取出前1天的系统时间
date -d “2 days ago” 取出前2天的系统时间

2) 取之后的时间：
date -d “a day ” 取出1天后的系统时间
date -d “2 days” 取出2天后的系统时间

因为Linux下的date是基于GNU的，但是Solaris基于POSIX的时间函数并没有提供类似的功能，甚至连秒一下的单位都无法直接获得，而是要自己写函数获得微秒和毫秒。

那么如何利用Shell在Solaris中实现之前的需求呢。有这么几种方法来实现：

利用其它语言，比如Perl、Tcl或者python。还有就是利用时区，但是这个仅仅局限于小时的修改，精确到分钟或者秒就不行了：

1)取之前的时间：yesterday=`TZ=$TZ+3; date +%Y%m%d`; echo $yesterday 取出前1天的系统时间

yesterday=`TZ=$TZ+27; date +%Y%m%d`; echo $yesterday 取出前2天的系统时间

2)取之后的时间：yesterday=`TZ=$TZ-22; date +%Y%m%d`; echo $yesterday 取出后1天的系统时间

yesterday=`TZ=$TZ-46; date +%Y%m%d`; echo $yesterday 取出后1天的系统时间

这里需要几点：
1）不能在命令行中直接执行TZ=$TZ-100; date +%Y%m%d，这样会修改当前terminal中的TZ和系统时间。
2）TZ=$TZ+3 需要加减多少小时才是前/后一天，和系统的$TZ有关。以上运算是在$TZ=PRC下进行的。

在Perl中，可以取epoch时间，并加上一定的秒数然后再转回来：
#perl -e ‘print int(time)’

相对于Solaris，在Linux中，取epoch时间就容易的多，只需要一条Shell命令。

关于GNU，POSIX和epoch，可以参考维基百科

。

大家都知道著名的Unix高手的十个习惯，殊不知Unix高手还有另外十个习惯，呵呵，偶不是Unix高手，不过有几个习惯还是挺常用的，转载一下，IBM的网站上还是有很多好文章的。

您应当采纳的其他 10 个好习惯包括：

* 使用文件名自动完成功能 (file name completion)。
* 使用历史扩展。
* 重用以前的参数。
* 使用 pushd 和 popd 管理目录导航。
* 查找大型文件。
* 不使用编辑器创建临时文件。
* 使用 curl 命令行实用工具。
* 最有效地利用正则表达式。
* 确定当前用户。
* 使用 awk 处理数据。

使用文件名完成

如果不需要在命令提示符处键入长的、令人费解的文件名，这是不是很棒呢？的确，您不需要这样做。相反，您可以配置最流行的 UNIX Shell 以使用文件名完成。该功能在各个 Shell 中的工作方式略有不同，因此我将向您展示如何在最流行的 Shell 中使用文件名完成。文件名完成使您可以更快地输入并避免错误。懒惰？也许吧。效率更高？当然！

我正在运行哪种 Shell？

如果您不知道目前使用的是哪一种 Shell，会怎么样？虽然这个诀窍不是另外 10 个好习惯的正式组成部分，但它仍然很有用。如清单 1 所示，您可以使用 echo $0 或 ps -p $$ 命令显示您正在使用的 Shell。对于我来说，运行的是 Bash Shell。
Read more »

最近写了一个系统资源监控的脚本，想让它每次机器重启后也可以自动启动，决定放到inittab中去，于是写了一个脚本run.sh,执行的时候启动脚本，并且加入inittab,可是调试的时候发现sed对shell的变量引用一直无法替换，百思不得其解，后来得到几位高手指点，终于解决，还有两种解决方法。

问题描述：

命令是这样的

work_dir=/home/ractest

sed ‘/start_mon.sh/d
/init.crsd/ a\
h4:3:respawn:${work_dir}/scripts/start_mon.sh >/dev/null 2>&1
‘ /etc/inittab

想把work_dir替换，但是没成功。

sed “/start_mon.sh/d
/init.crsd/ a\
h4:3:respawn:${work_dir}/scripts/start_mon.sh >/dev/null 2>&1
” /etc/inittab
试过，报错
sed: command garbled: /init.crsd/ ah4:3:respawn:/home/ractest/ricky/atm_monitor/scripts/start_mon.sh >/dev/null 2>&1

在Solaris上不行，但是在Linux上是可以的，所以初步确定是sed版本的问题，但是在Solaris上怎么解决这个我呢？有如下两种方法：

方法一：使用单引号
单引号：

sed ‘/start_mon.sh/d
> /init.crsd/ a\
> h4:3:respawn:’${work_dir}’/scripts/start_mon.sh >/dev/null 2>&1
> ‘ /etc/inittab

方法二：使用双引号

sed “/start_mon.sh/d
> /init.crsd/ a\\
> h4:3:respawn:${work_dir}/scripts/start_mon.sh >/dev/null 2>&1
> ” /etc/inittab

经过测试，两种方法都可以成功。
记录之！

一直以为自己很喜欢shell，看了ChinaUnix上的这个水平测试，才知道原来自己什么都不是

[ OVERVIEW 篇 ]

1. 有很多种 shell, 你熟悉几种？ 各个 shell 的 home page 在那里？

2. 为什么说 zsh 是目前为止功能最为强大的 shell.

3. 为什么说 pdksh 功能较弱？

4. ksh88 与 ksh93 有何区别？

5. 为什么 shell 编程最好用 ksh?

6. 你的系统都有哪些 shell? 版本是多少？

7. 你知道 POSIX 吗？最新版本是多少？和你的 shell 有什么关系？

8. /sbin/sh 和 /bin/sh 有何区别？

9. 你分析过 1000　行以上的 shell 程序吗？

10. 各种 shell 的变量名长度有何限制？
Read more »