Blog

Howto – bash audit / command logger

Introduction

Having a complete history of all typed commands can be very helpful in many scenarios:

  • when several administrators work together on the same server and need to know what was done previously
  • when someone need to redo an older sequence of commands or to understand an undocumented maintenance process
  • for troubleshooting or forensic analysis, by crosschecking the date of an event or of a file with the commands executed at that date

The standard ‘.bash_history’ file of the shell is unfortunately not written on disk in the case of a crash and it may be deleted by the user.
Another problem is that when many shell sessions are running concurrently, their logging will only occur when they are closed, therefore the commands of the history will not appear in their chronological order.
Furthermore, ‘.bash_history’ will not include essential information like the ‘working directory’ of the command; and by default the repetition or re-edition of commands will not be logged, too.

Some solutions exist to improve this, either by patching or installing binaries:

  • ‘bash-BOFH’ patching and recompiling: works well but need a new patch for each release of the bash
  • ‘snoopy’: is logging all commands except shell builtins
  • ‘rootsh / sniffy / ttyrpld / ttysnoop’: logs everything, also output of commands, it may be useful but it generates very verbose logs
  • ‘grsecurity’ patched kernels: powerful but it may be a not suitable solution if an official kernel is required (e.g. for Oracle DB)
  • there is also an old ‘sshd’ patch (‘http://www.kdvelectronics.eu/ssh-logging/ssh-logging.html‘)
  • ‘screen -x’ can also be useful for cooperation work, but it is not a command logger

In contrast to that, the presented method is very easy to deploy; it is just a shellscript that is running in bash (standard shell on most systems) and therefore it is architecture independent.
It will allow a complete audit of all commands/builtins executed interactively in the bash.
Note that a user can avoid calling this file by starting a shell with options like ‘–norc’; he also can unset or overwrite variables like ‘PROMPT_COMMAND’.
Therefore this script is useful for audit but an alternative solution with bash patching should be considered if the security requirements are the priority.

The result of the shellscript will be sent via syslog and will look like the following example:

2012-05-31T15:40:52.530657+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: id
2012-05-31T15:41:10.533963+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: id #again, up-key
2012-05-31T15:41:26.421925+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: id #again, ctrl-r
2012-05-31T15:41:44.223986+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: (echo a; echo b); { echo c; echo d; }; echo e | ( cat && echo f)
2012-05-31T15:41:49.687712+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22] /root: exit
2012-05-31T15:41:49.690234+02:00 debian1 [audit root/31240 as root/31240 on pts/0/192.168.1.50:40067->192.168.1.120:22]: #=== bash session ended. ===
2012-05-31T15:51:01.399710+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22]: #=== New bash session started. ===
2012-05-31T15:51:02.980606+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: id
2012-05-31T15:51:09.327614+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: ssh localhost
2012-05-31T15:51:14.768578+02:00 debian1 [audit root/. as root/31874 on pts/1/127.0.0.1:47702->127.0.0.1:22]: #=== New bash session started. ===
2012-05-31T15:51:48.031612+02:00 debian1 [audit root/. as root/31874 on pts/1/127.0.0.1:47702->127.0.0.1:22] /root: echo hi from ssh
2012-05-31T15:51:49.875367+02:00 debian1 [audit root/. as root/31874 on pts/1/127.0.0.1:47702->127.0.0.1:22] /root: exit
2012-05-31T15:51:49.877675+02:00 debian1 [audit root/. as root/31874 on pts/1/127.0.0.1:47702->127.0.0.1:22]: #=== bash session ended. ===
2012-05-31T15:51:51.075034+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: id
2012-05-31T15:51:53.027380+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: less /var/log/user.log
2012-05-31T16:00:06.676849+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22]: #=== New bash session started. ===
2012-05-31T16:00:12.690457+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22] /home/pointsoftware: ls
2012-05-31T16:00:16.114366+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22] /home/pointsoftware: cd /
2012-05-31T16:00:21.002682+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22] /: ps aux
2012-05-31T16:00:30.714988+02:00 debian1 [audit root/31843 as root/31843 on pts/0/192.168.1.50:48710->192.168.1.120:22] /root: less /var/log/user.log
2012-05-31T16:00:37.754175+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22] /: ps aux
2012-05-31T16:00:37.758383+02:00 debian1 [audit pointsoftware/31950 as pointsoftware/31977 on pts/2/192.168.1.50:48804->192.168.1.120:22]: #=== bash session ended. ===

 

Explanation of the method

First of all we tune up some bash settings related to history:

#'history' options
declare -rx HISTFILE="$HOME/.bash_history"
declare -rx HISTSIZE=500000                                 #nbr of cmds in memory
declare -rx HISTFILESIZE=500000                             #nbr of cmds on file
declare -rx HISTCONTROL=""                                  #does not ignore spaces or duplicates
declare -rx HISTIGNORE=""                                   #does not ignore patterns
declare -rx HISTCMD                                         #history line number
#history -r                                                  #to reload history from file if a prior HISTSIZE has truncated it
if [ "${OSTYPE:0:7}" != "solaris" ] #following not working in solaris
then
if groups | grep -q root
then
  declare -x TMOUT=3600                                     #timeout for root's sessions
  chattr +a "$HISTFILE"                                     #set append-only
fi
fi
shopt -s histappend
shopt -s cmdhist

#history substitution ask for a confirmation
shopt -s histverify

 

The bash history could also include timestamps with following options, but we will not need it, as our solution will create another log file (‘/var/log/userlog.info’) with all details like timestamp, working directory, PID, userid, etc..

#add timestamps in history - obsoleted with logger/syslog
#'http://www.thegeekstuff.com/2008/08/15-examples-to-master-linux-command-line-history/#more-130'
#declare -rx HISTTIMEFORMAT='%F %T '

 

A nicer colored shell prompt is also very handy when we scroll back the terminal. BTW, it can be very useful to set your terminal to have a least 10’000 lines of scrollback.

#prompt & color
#'http://www.pixelbeat.org/docs/terminal_colours/#256'
#'http://www.frexx.de/xterm-256-notes/'
_backnone="\e[00m"
_backblack="\e[40m"
_backblue="\e[44m"
_frontred_b="\e[01;31m"
_frontgreen_b="\e[01;32m"
_frontgrey_b="\e[01;37m"
_frontgrey="\e[00;37m"
_frontblue_b="\e[01;34m"
PS1="\[${_backblue}${_frontgreen_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontgreen_b}\] "

 

By default ‘ctrl-s’ will block the shell until you press ‘ctrl-q’ (flow control). We disable this in order to be able to search forward and backward for commands (‘ctrl-s’/'ctrl-r’):

#enable forward search ('ctrl-s')
#'http://ruslanspivak.com/2010/11/25/bash-history-incremental-search-forward/'
if shopt -q login_shell && [ -t 0 ]
then
  stty -ixon
fi

 

Following shortcut keys are available in ‘bash’:

# bash shortcuts
#'http://www.techrepublic.com/article/master-the-linux-bash-command-line-with-these-10-shortcuts/5827311'
#'http://www.hypexr.org/bash_tutorial.php'
# ctrl-r                reverse search
# ctrl-s                forward search
# alt-. or esc-.        reuse 1st arg
# ctrl-a                Move cursor to beginning of line
# ctrl-e                Move cursor to end of line
# meta-b                Move cursor back one word
# meta-f                Move cursor forward one word
# ctrl-w                Cut the last word
# ctrl-u                Cut everything before the cursor
# ctrl-k                Cut everything after the cursor
# ctrl-y                Paste the last thing to be cut
# ctrl-_                Undo
#bash-history-cheat-sheet.pdf 'http://www.catonmat.net/download/bash-history-cheat-sheet.pdf'
# Emacs Mode Shortcuts:
# CTRL-p                Fetch the previous command from the history list.
# CTRL-n                Fetch the next command from the history list.
# CTRL-r                Search history backward (incremental search).
# CTRL-s                Search history forward (incremental search).
# Meta-p                Search backward using non-incremental search.
# Meta-n                Search forward using non-incremental search.
# Meta-<                Move to the first line in the history.
# Meta->                Move to the end of the history list.
# Vi Mode Shortcuts:
# k                     Fetch the previous command from the history list.
# j                     Fetch the next command from the history list.
# /string or CTRL-r     Search history backward for a command matching string.
# ?string or CTRL-s     Search history forward for a command matching string.
# n                     Repeat search in the same direction as previous.
# N                     Repeat search in the opposite direction as previous.
# G                     Move to the N-th history line (for example, 15G).

 

To retrieve the original username (not the current username that may be changed with ‘su’), we tried this method using a loop to seek the parent-PID in ‘/proc/’:

#seek the oldest parent 'bash' process, to get $AUDIT_LOGINID and $AUDIT_LOGINUSER,
#which may be different from $USER after 'su' or 'sudo' commands
AUDIT_LOGINID=$$
AUDIT_LOGINID2=$AUDIT_LOGINID
while AUDIT_LOGINID2="$(awk '/PPid:/ {print $2}' /proc/$AUDIT_LOGINID2/status)" && [ "$AUDIT_LOGINID2" != "1" ]
do
  if [ "$(awk '/Name:/ {print $2}' /proc/$AUDIT_LOGINID2/status)" == "bash" ]
  then
    AUDIT_LOGINID="$AUDIT_LOGINID2"
  fi
done
AUDIT_LOGINUSER=$(awk "/^Uid:/ {print \$2}" /proc/$AUDIT_LOGINID/status)
AUDIT_LOGINUSER=$(awk -F":" "\$3 ~ /$AUDIT_LOGINUSER/ {print \$1}" /etc/passwd)
#old: AUDIT_LOGINUSER=$(awk -F":" "/^[^:]*:[^:]*:$AUDIT_LOGINUSER:/ {print \$1}" /etc/passwd)
#old: AUDIT_LOGINUSER=$(getent passwd $AUDIT_LOGINUSER | sed -e 's%:.*%%')"

 

We then used a simpler approach, using the ‘who -mu’ command:

declare -rx AUDIT_LOGINUSER="$(who -mu | awk '{print $1}')"
declare -rx AUDIT_LOGINPID="$(who -mu | awk '{print $6}')"
declare -rx AUDIT_USER="$USER"                              #defined by pam during su/sudo
declare -rx AUDIT_PID="$$"
declare -rx AUDIT_TTY="$(who -mu | awk '{print $2}')"
declare -rx AUDIT_SSH="$([ -n "$SSH_CONNECTION" ] && echo "$SSH_CONNECTION" | awk '{print $1":"$2"->"$3":"$4}')"
declare -rx AUDIT_STR="[audit $AUDIT_LOGINUSER/$AUDIT_LOGINPID as $AUDIT_USER/$AUDIT_PID on $AUDIT_TTY/$AUDIT_SSH]"
declare -x AUDIT_LASTHISTLINE=""                            #to avoid logging the same line twice
declare -rx AUDIT_SYSLOG="1"                                #to use a local syslogd

 

Now we need a way to perform the logging at each execution of command. The following solution using ‘PROMPT_COMMAND’ is working but the syslog messages are sent after the command execution,
this causes ‘su’ or ‘sudo’ commands to appear only after logouts, and ‘cd’ commands to display the wrong working directory:

#'http://jablonskis.org/2011/howto-log-bash-history-to-syslog/'
declare -rx PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -p user.info -t "$AUDIT_STR $PWD")' #no subshell is used here, it would else duplicate execution!

 

‘history -a >(tee -a ~/.bash_history | logger -p user.info -t “$AUDIT_STR $PWD”)’ will take the history commands that are still uncommited to disk (‘history -a’) and write them to a process substitution (‘>()’) that will be the standard input for the ‘tee -a’ command, which will append the result to ‘~/.bash_history’ and send it also to the ‘logger’ command via a pipe; logger will send it to the local syslog daemon.

Another solution is to use ‘trap’ DEBUG, which is executed before the command.
See ‘http://superuser.com/questions/175799/does-bash-have-a-hook-that-is-run-before-executing-a-command‘ and ‘http://www.davidpashley.com/articles/xterm-titles-with-bash.html‘.
Short example (does not work with piped command):

set -o functrace; trap 'echo -ne "===$BASH_COMMAND===${_backnone}${_frontgrey}\n"' DEBUG

 

Longer example with a subshell (works with piped command):

set +o functrace                                            #disable trap DEBUG inherited in functions, command substitutions or subshells, normally the default setting already
function AUDIT_DEBUG() {
  echo -ne "${_backnone}${_frontgrey}"                      #disable prompt colors for the command's output
  (history -a >(logger -p user.info -t "$AUDIT_STR $PWD" < <(tee -a ~/.bash_history))) && sync && history -c && history -r
}
set -o functrace                                            #enable trap DEBUG inherited for all subsequent functions; required to audit commands beginning with the char '(' for a subshell
#=> problem: auto-completion in commands avoids logging them
#launches AUDIT_DEBUG() and then stops the trap DEBUG, to avoid a useless rerun of AUDIT_DEBUG() during the execution of $PROMPT_COMMAND
declare -rx PROMPT_COMMAND="trap 'AUDIT_DEBUG; trap DEBUG' DEBUG; $PROMPT_COMMAND"

 

‘history -a >(logger -p user.info -t “$AUDIT_STR $PWD” < <(tee -a ~/.bash_history))' will take the history commands that are still uncommited to disk ('history -a') and write them to a process substitution ('>()’) that will be the standard input for the ‘tee -a’ command, which will append the result to ‘~/.bash_history’ and to another process substition (‘<()'), which finally get read by the 'logger' command.
'history -c && history -r' are here forcing a refresh of the history because 'history -a' was called within a subshell and therefore the new history commands that were just appent to file will keep their "new" status outside of the subshell, causing their logging to re-occur on every function call... (cf. 'http://stackoverflow.com/questions/103944/real-time-history-export-amongst-bash-terminal-windows‘)
Note that without the subshell, piped bash commands would hang… (it seems that the trap + process substitution interfer with stdin redirection)

The final solution is quicker and it is avoiding ‘sync’ and ‘history -r’ that are time consuming.
It turns out that this solution is much simpler and works well with piped commands, subshells, aborted commands with ‘ctrl-c’ and all test-cases that we could try.
The trick is again to use a trap DEBUG function, to have set the required history options (HISTCONTROL, HISTIGNORE) and to disable the trap in functions, command substitutions or subshells.

set +o functrace                                            #disable trap DEBUG inherited in functions, command substitutions or subshells, normally the default setting already
shopt -s extglob                                            #enable extended pattern matching operators
function AUDIT_DEBUG() {
  if [ -z "$AUDIT_LASTHISTLINE" ]                           #initialization
  then
    local AUDIT_CMD="$(fc -l -1 -1)"                        #previous history command
    AUDIT_LASTHISTLINE="${AUDIT_CMD%%+([^ 0-9])*}"
  else
    AUDIT_LASTHISTLINE="$AUDIT_HISTLINE"
  fi
  local AUDIT_CMD="$(history 1)"                            #current history command
  AUDIT_HISTLINE="${AUDIT_CMD%%+([^ 0-9])*}"
  if [ "${AUDIT_HISTLINE:-0}" -ne "${AUDIT_LASTHISTLINE:-0}" ] || [ "${AUDIT_HISTLINE:-0}" -eq "1" ]        #avoid logging unexecuted commands after 'ctrl-c', 'empty+enter', or after 'ctrl-d'
  then
    echo -ne "${_backnone}${_frontgrey}"                    #disable prompt colors for the command's output
    #remove in last history cmd its line number (if any) and send to syslog
    if [ -n "$AUDIT_SYSLOG" ]
    then
      if ! logger -p user.info -t "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}"
      then
        echo error "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}"
      fi
    else
      echo $( date +%F_%H:%M:%S ) "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}" >>/var/log/userlog.info
  fi
  #echo "===cmd:$BASH_COMMAND/subshell:$BASH_SUBSHELL/fc:$(fc -l -1)/history:$(history 1)/histline:${AUDIT_CMD%%+([^ 0-9])*}/last_histline:${AUDIT_LASTHISTLINE}===" #for debugging
    return 0
  else
    return 1
  fi
}

#audit the session closing
function AUDIT_EXIT() {
  local AUDIT_STATUS="$?"
  if [ -n "$AUDIT_SYSLOG" ]
  then
    logger -p user.info -t "$AUDIT_STR" "#=== session closed ==="
  else
    echo $( date +%F_%H:%M:%S ) "$AUDIT_STR" "#=== session closed ===" >>/var/log/userlog.info
  fi
  exit "$AUDIT_STATUS"
}

#make audit trap functions readonly; disable trap DEBUG inherited (normally the default setting already)
declare -frx +t AUDIT_DEBUG
declare -frx +t AUDIT_EXIT
 #audit the session opening
if [ -n "$AUDIT_SYSLOG" ]
then
  logger -p user.info -t "$AUDIT_STR" "#=== session opened ===" #audit the session openning
else
  echo $( date +%F_%H:%M:%S ) "$AUDIT_STR" "#=== session opened ===" >>/var/log/userlog.info
fi

#enable the trap DEBUG (at every call of $PROMPT_COMMAND) and trap EXIT
declare -rx PROMPT_COMMAND="trap 'AUDIT_DEBUG; trap DEBUG' DEBUG"
declare -rx BASH_COMMAND                                    #current command executed by user or a trap
declare -rx SHELLOPT                                        #shell options, like functrace
trap AUDIT_EXIT EXIT                                        #audit the session closing

 

When a bash command is executed it first launches the AUDIT_DEBUG(),
then the trap DEBUG is disabled to avoid a useless rerun of AUDIT_DEBUG() during the execution of pipes-commands;
at the end, when the prompt is displayed, it re-enables the trap DEBUG.

 

Installation

Step 1 – Login as root and to create a file ‘/etc/bash_franzi’ (download bash_franzi) with following content:

#Pointsoftware AG, 2013-11-03
#created by francois scheurer
#filename: '/etc/bash_franzi'
#This file must be sourced by '~/.bashrc', which is the last runned startup script for bash invocation for login interactive, login non-interactive and non-login interactive shells.
#
#Having a complete history of all typed commands can be very helpful in many scenarios:
#  when several administrators work together on the same server and need to know what was done previously
#  when someone need to redo an older sequence of commands or to understand an undocumented maintenance process
#  for troubleshooting or forensic analysis, by crosschecking the date of an event or of a file with the commands executed at that date
#
#The standard '.bash_history' file of the shell is unfortunately not written on disk in the case of a crash and it may be deleted by the user.
#Another problem is that when many shell sessions are running concurrently, their logging will only occur when they are closed, therefore the commands of the history will not appear in their chronological order.
#Furthermore, '.bash_history' will not include essential information like the 'working directory' of the command; and by default the repetition or re-edition of commands will not be logged, too.
#
#Some solutions exist to improve this, either by patching or installing binaries:
#  'bash-BOFH' patching and recompiling: works well but need a new patch for each release of the bash
# 'snoopy': is logging all commands except shell builtins
#  'rootsh / sniffy / ttyrpld / ttysnoop': logs everything, also output of commands, it may be useful but it generates very verbose logs
#  'grsecurity' patched kernels: powerful but it may be a not suitable solution if an official kernel is required (e.g. for Oracle DB)
#  there is also an old 'sshd' patch ('http://www.kdvelectronics.eu/ssh-logging/ssh-logging.html')
#  'screen -x' can also be useful for cooperation work, but it is not a command logger
#
#In contrast to that, the presented method is very easy to deploy; it is just a shellscript that is running in bash (standard shell on most systems) and therefore it is architecture independent.
#It will allow a complete audit of all commands/builtins executed interactively in the bash.
#Note that a user can avoid calling this file by starting a shell with options like '--norc'; he also can unset or overwrite variables like 'PROMPT_COMMAND'.
#Therefore this script is useful for audit but an alternative solution with bash patching should be considered if the security requirements are the priority.
#
#Note on Solaris:
#       In Solaris please use 'grep' without the '-q' option, like this:
#               if groups | grep root &>/dev/null
#       Please also remove the following line (chattr unsupported in Solaris:
#               chattr +a "$HISTFILE"
#       Then modify your /etc/syslog.conf to include this line:
#               user.info /var/adm/userlog.info
#       To assign 'bash' as the login shell in Solaris: passwd -e /bin/bash .
#       Make sure that the audit-script is sourced (=included) correctly during the bash invocation.
#       If your bash version is too old, $HISTCONTROL will not allow you to log duplicated commands correctly.
#       svcadm restart system/system-log
#       svcadm disable ssh
#       svcadm enable ssh


if [ "${SHELL##*/}" != "bash" ]; then
  return
fi

#to avoid sourcing this file more than once
if [ -n "${OSTYPE##solaris*}" ]; then #following not working in solaris
  #do not source this file twice; also do not source it if we are in forcecommand.sh, source it later from "-bash-li"
  #if we would source it from forcecommand.sh, the environment would be lost after the call of 'exec -l bash -li'
  if [ "$AUDIT_INCLUDED" == "$$" ] || { [ -z "$SSH_ORIGINAL_COMMAND" ] && [ "$(cat /proc/$$/cmdline)" == 'bash-c"/etc/forcecommand.sh"' ]; }; then
    return
  else
    declare -rx AUDIT_INCLUDED="$$"
  fi
fi

#prompt & color
#'http://www.pixelbeat.org/docs/terminal_colours/#256'
#'http://www.frexx.de/xterm-256-notes/'
_backnone="\e[00m"
_backblack="\e[40m"
_backblue="\e[44m"
_frontred_b="\e[01;31m"
_frontgreen_b="\e[01;32m"
_frontgrey_b="\e[01;37m"
_frontgrey="\e[00;37m"
_frontblue_b="\e[01;34m"
#PS1="\[${_backblue}${_frontgrey_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontgrey_b}\] " #grey
PS1="\[${_backblue}${_frontgreen_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontgreen_b}\] " #green
#PS1="\[${_backblue}${_frontred_b}\]\u@\h:\[${_backblack}${_frontblue_b}\]\w\\$\[${_backnone}${_frontred_b}\] " #red
declare -rx PS1

#'history' options
declare -rx HISTFILE="$HOME/.bash_history"
declare -rx HISTSIZE=500000                                 #nbr of cmds in memory
declare -rx HISTFILESIZE=500000                             #nbr of cmds on file
declare -rx HISTCONTROL=""                                  #does not ignore spaces or duplicates
declare -rx HISTIGNORE=""                                   #does not ignore patterns
declare -rx HISTCMD                                         #history line number
#following line is commented to avoid following issue: loading the history during the sourcing of this file (non-interactive bash) is also loading history lines that begin with '#', but then during the trap DEBUG calls it reloads the whole history without '#'-lines and produces an double-length history.
#history -r                                                  #to reload history from file if a prior HISTSIZE has truncated it

#following 2 lines commented because 'history -r' was still loading '#'-lines
#shopt -s extglob                                            #enable extended pattern matching operators
#HISTIGNORE="*([ \t])#*"; history -r                         #reload history without commented lines; this force non-interactive bash to behave like interactive bash, without this AUDIT_HISTLINE will get a wrong initial value, leading then to a small issue where empty bash sessions are actually logging the last command of history

if [ -n "${OSTYPE##solaris*}" ]; then #following not working in solaris
  if groups | grep -q root; then
    declare -x TMOUT=43200                                    #timeout for root's sessions
    chattr +a "$HISTFILE"                                     #set append-only
  fi
fi
shopt -s histappend
shopt -s cmdhist

#history substitution ask for a confirmation
shopt -s histverify

#add timestamps in history - obsoleted with logger/syslog
#'http://www.thegeekstuff.com/2008/08/15-examples-to-master-linux-command-line-history/#more-130'
#declare -rx HISTTIMEFORMAT='%F %T '

#enable forward search ('ctrl-s')
#'http://ruslanspivak.com/2010/11/25/bash-history-incremental-search-forward/'
if shopt -q login_shell && [ -t 0 ]; then
  stty -ixon
fi



#bash audit & traceability
#
#
#
declare -rx AUDIT_LOGINUSER="$(who -mu | awk '{print $1}')"
declare -rx AUDIT_LOGINPID="$(who -mu | awk '{print $6}')"
declare -rx AUDIT_USER="$USER"                              #defined by pam during su/sudo
declare -rx AUDIT_PID="$$"
declare -rx AUDIT_TTY="$(who -mu | awk '{print $2}')"
declare -rx AUDIT_SSH="$([ -n "$SSH_CONNECTION" ] && echo "$SSH_CONNECTION" | awk '{print $1":"$2"->"$3":"$4}')"
declare -rx AUDIT_STR="[audit $AUDIT_LOGINUSER/$AUDIT_LOGINPID as $AUDIT_USER/$AUDIT_PID on $AUDIT_TTY/$AUDIT_SSH]"
declare -x AUDIT_LASTHISTLINE=""                            #to avoid logging the same line twice
declare -rx AUDIT_SYSLOG="1"                                #to use a local syslogd
#
#
#
#the logging at each execution of command is performed with a trap DEBUG function
#and having set the required history options (HISTCONTROL, HISTIGNORE)
#and to disable the trap in functions, command substitutions or subshells.
#it turns out that this solution is simple and works well with piped commands, subshells, aborted commands with 'ctrl-c', etc..
set +o functrace                                            #disable trap DEBUG inherited in functions, command substitutions or subshells, normally the default setting already
shopt -s extglob                                            #enable extended pattern matching operators
function AUDIT_DEBUG() {
  if [ -z "$AUDIT_LASTHISTLINE" ]; then                     #initialization
    local AUDIT_CMD="$(fc -l -1 -1)"                        #previous history command
    AUDIT_LASTHISTLINE="${AUDIT_CMD%%+([^ 0-9])*}"
  else
    AUDIT_LASTHISTLINE="$AUDIT_HISTLINE"
  fi
  local AUDIT_CMD="$(history 1)"                            #current history command
  AUDIT_HISTLINE="${AUDIT_CMD%%+([^ 0-9])*}"
  if [ "${AUDIT_HISTLINE:-0}" -ne "${AUDIT_LASTHISTLINE:-0}" ] || [ "${AUDIT_HISTLINE:-0}" -eq "1" ]; then  #avoid logging unexecuted commands after 'ctrl-c', 'empty+enter', or after 'ctrl-d'
    echo -ne "${_backnone}${_frontgrey}"                    #disable prompt colors for the command's output
    #remove in last history cmd its line number (if any) and send to syslog
    if [ -n "$AUDIT_SYSLOG" ]; then
      if ! logger -p user.info -t "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}"; then
        echo error "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}"
      fi
    else
      echo $( date +%F_%H:%M:%S ) "$AUDIT_STR $PWD" "${AUDIT_CMD##*( )?(+([0-9])?(\*)+( ))}" >>/var/log/userlog.info
    fi
    #echo "===cmd:$BASH_COMMAND/subshell:$BASH_SUBSHELL/fc:$(fc -l -1)/history:$(history 1)/histline:${AUDIT_CMD%%+([^ 0-9])*}/last_histline:${AUDIT_LASTHISTLINE}===" #for debugging
    return 0
  else
    return 1
  fi
}
#
#
#
#audit the session closing
function AUDIT_EXIT() {
  local AUDIT_STATUS="$?"
  if [ -n "$AUDIT_SYSLOG" ]; then
    logger -p user.info -t "$AUDIT_STR" "#=== session closed ==="
  else
    echo $( date +%F_%H:%M:%S ) "$AUDIT_STR" "#=== session closed ===" >>/var/log/userlog.info
  fi
  exit "$AUDIT_STATUS"
}
#
#
#
#make audit trap functions readonly; disable trap DEBUG inherited (normally the default setting already)
declare -frx +t AUDIT_DEBUG
declare -frx +t AUDIT_EXIT
#
#
#
#audit the session opening
if [ -n "$AUDIT_SYSLOG" ]; then
  logger -p user.info -t "$AUDIT_STR" "#=== session opened ===" #audit the session openning
else
  echo $( date +%F_%H:%M:%S ) "$AUDIT_STR" "#=== session opened ===" >>/var/log/userlog.info
fi
#
#
#
#when a bash command is executed it launches first the AUDIT_DEBUG(),
#then the trap DEBUG is disabled to avoid a useless rerun of AUDIT_DEBUG() during the execution of pipes-commands;
#at the end, when the prompt is displayed, re-enable the trap DEBUG
        #declare -rx PROMPT_COMMAND="AUDIT_DONE=; trap 'AUDIT_DEBUG && AUDIT_DONE=1; trap DEBUG' DEBUG; [ -n \"\$AUDIT_DONE\" ] && echo '-----------------------------'"
        #NOK: declare -rx PROMPT_COMMAND="echo "-----------------------------"; trap 'AUDIT_DEBUG; trap DEBUG' DEBUG; echo '-----------------------------'"
        #OK:  declare -rx PROMPT_COMMAND="echo "-----------------------------"; trap 'AUDIT_DEBUG; trap DEBUG' DEBUG"
declare -rx PROMPT_COMMAND="[ -n \"\$AUDIT_DONE\" ] && echo '-----------------------------'; AUDIT_DONE=; trap 'AUDIT_DEBUG && AUDIT_DONE=1; trap DEBUG' DEBUG"
declare -rx BASH_COMMAND                                    #current command executed by user or a trap
declare -rx SHELLOPT                                        #shell options, like functrace
trap AUDIT_EXIT EXIT                                        #audit the session closing


#endof

 
Change its ownership/permissions:

chown root:root /etc/bash_franzi
chmod 644 /etc/bash_franzi

 

Step 2 – This file need to be sourced automatically at start by appending “. /etc/bash_franzi” to your init-file ‘/etc/.bashrc’.
This can be done automatically by running following lines (just copy-paste them in your terminal):

for i in /etc/profile /etc/skel/.bashrc /root/.bashrc /home/*/.bashrc; do
  if ! grep -q ". /etc/bash_franzi" "$i"; then
    echo "===updating $i==="
    echo "[ -f /etc/bash_franzi ] && . /etc/bash_franzi #added by francois scheurer" >>"$i"
  fi
done

Note: sourcing this file directly from /etc/profile may cause problems with the display manager’s starting scripts of Ubuntu (gnome or lightdm).
So it is better to only source it from ~/.bashrc , but then there is another issue, with Debian: in Debian each user’s home folder gets by default
a .bash_profile file that has priority over .bashrc . The bash man pages state:
When bash is invoked as an interactive login shell, or as a non-interactive shell with the –login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The –noprofile option may be used when the shell is started to inhibit this behavior.
In other words ~/.bashrc will only be read if you delete or rename ~/.bash_profile .
 

Step 3 – Configure ‘rsyslogd’:

Configure ‘/etc/rsyslog.conf’ by running following lines:

cat >>/etc/rsyslog.conf <<"EOF"
#added by francois scheurer
$ActionFileDefaultTemplate RSYSLOG_FileFormat
#stop avahi if messages are dropped (cf. /var/log/messages with 'net_ratelimit' or 'imuxsock begins to drop')
#update-rc.d -f avahi-daemon remove && service avahi-daemon stop
#https://isc.sans.edu/diary/Are+you+losing+system+logging+information+%28and+don%27t+know+it%29%3F/15106
#$SystemLogRateLimitInterval 10 
#$SystemLogRateLimitBurst 500
$SystemLogRateLimitInterval 0
#endof
EOF

 

Create ‘/etc/rsyslog.d/45-franzi.conf’ by running following lines:

cat >/etc/rsyslog.d/45-franzi.conf <<"EOF"
#added by francois scheurer

# Filter duplicated messages
$RepeatedMsgReduction off

# Enable high precision timestamps
$ActionFileDefaultTemplate RSYSLOG_FileFormat

# Log bash audit generated log messages to file
if $syslogfacility-text == 'user' and $syslogseverity-text == 'info' and $syslogtag startswith '[audit' then /var/log/userlog.info

#then drop them
& ~

#'http://content.hccfl.edu/pollock/aunix2/logging.htm'
#'http://www.rsyslog.com/doc/rsyslog_conf_filter.html'
EOF

 

Restart 'rsyslogd':

/etc/init.d/rsyslog restart

 

Now do a logout and login and you should find the audit in '/var/log/userlog.info'. ^_^

Optional Step 4 - If you also want to audit SCP/SFTP, create also a file '/etc/forcecommand.sh' (download forcecommand.sh) by running following lines:

cat >/etc/forcecommand.sh <<"EOF"
#!/bin/bash
#Pointsoftware AG, 2012-08-24
#filename: '/etc/forcecommand.sh'
#created by francois scheurer
#used for bash audit, see '/etc/bash_franzi'

if [ -n "${SSH_ORIGINAL_COMMAND}" ]; then
  exec bash -c "${SSH_ORIGINAL_COMMAND}"
else
  exec -l bash -li
fi
#endof
EOF

 
Change its ownership/permissions:

chown root:root /etc/forcecommand.sh
chmod 755 /etc/forcecommand.sh

 

Then configure your ‘/etc/ssh/sshd_config’ by running following lines:

cat >>/etc/ssh/sshd_config <<"EOF"
ForceCommand "/etc/forcecommand.sh"
EOF

 
And reload ‘sshd’:

/etc/init.d/ssh reload

 
(hint: test your sshd by starting a second session before closing the current session, just in case ;-) )

Kommentare

  1. Donovan Baarda

    I think ‘history -a’ may have changed behaviour since this was written. It doesn’t seem to produce any output, just writes the new history to the history file.

    • Donovan Baarda

      Doh! you are doing “history -a >(…)”, which does send the recent history to the cmd instead of appending it to .bash_history… I understand now.

      • Francois Scheurer

        Yes exactly ;-)

    • Francois Scheurer

      Hi Donovan,

      Yes, it just append the uncommited commands to the history files, as you write. But this behavior is not new.
      Maybe you are thinking that it should gives some output because of the redirection symbol in ‘history -a >(tee -a ~/.bash_history | logger -p user.info -t “$AUDIT_STR $PWD”)’. But if you read the blog carefuly it explains that ‘>()’ is not a redirection of output but a process substitution, which you can see as a virtual file for writing. So ‘history -a’ will write into this virtual history file, which actually forward all data to ‘tee’, which sends them to .bash_history and to ‘logger’.

      Best Regards

  2. Felipe Ruiz

    Hello Francois. First of all, thanks for this excellent script. I have installed it in debian 6 and it works very good, except for the SCP/SFTP audit.
    I am having some troubles with SCP/SFTP. If you check, the contents of forcecheck.sh in this post is different from the file posted in http://www.pointsoftware.ch/wp-content/uploads/2012/09/forcecommand.sh_.txt
    Nevertheless, none of them works. The first time I connect, I can see the following (also strange) lines in my userlog.info:

    Apr 16 11:15:06 localhost [audit / as asso/13661 on /10.1.1.1:54206->10.1.1.9:22]: #=== session opened ===
    Apr 16 11:15:06 localhost [audit / as asso/13661 on /10.1.1.1:54206->10.1.1.9:22] /opt/_home/user1: echo “WinSCP: this is end-of-file:0″
    Apr 16 11:15:06 localhost [audit / as asso/13661 on /10.1.1.1:54206->10.1.1.9:22] /opt/_home/user1: echo “$status” ; echo “WinSCP: this is end-of-file:0″
    Apr 16 11:15:06 localhost [audit / as asso/13661 on /10.1.1.1:54206->10.1.1.9:22] /opt/_home/user1: echo “$?” ; echo “WinSCP: this is end-of-file:0″
    Apr 16 11:15:06 localhost [audit / as asso/13661 on /10.1.1.1:54206->10.1.1.9:22]: #=== session closed ===

    As you can see, it seems to be an error with the “echo” commands, or with some variable identification.
    The second sftp connection logs the following:

    Apr 16 11:19:34 localhost [audit / as asso/16172 on /10.1.1.1:54234->10.1.1.9:22] /opt/_home/asso: /usr/lib/openssh/sftp-server

    And nothing more. I tried creating directories, copying files, changing cwd, but nothing gets logged.

    Can you help me?
    Thanks.

  3. Amit

    Hi Francois,

    Thank you for the great script.

    I have a question, what is the best way to read the logs? Can the log be written delimited by # or @ or any special character so that we can parse the output in excel?

    If yes, then can you please help me in this?

    regards,
    Amit

    • Francois Scheurer

      Hi Amit,

      I would use ‘sed’ or ‘awk’ to reformat a log.
      Can you mabe write and example of input-output that you need?

      On my side I use a modified version of cronic.sh from a crontab to run the backups, so that I have a full log on the server but only an email when an error occurs.

      for example:
      00 01 * * 2-6 CRONIC_LOG=/backup/snapshot/snapshot.log /etc/opt/cronic.sh /backup/snapshot/rsync/rsync-snapshot-postgres.sh; CRONIC_LOG=/backup/snapshot/snapshot.log /etc/opt/cronic.sh /backup/snapshot/rsync/backup-postgresql.sh; CRONIC_LOG=/backup/snapshot/snapshot.log /etc/opt/cronic.sh /backup/snapshot/rsync/backup-mysql.sh #local backup of filesystem & postgres & mysql database

      cat /etc/opt/cronic.sh
      #!/bin/bash

      # Cronic v2 – cron job report wrapper
      # Copyright 2007 Chuck Houpt. No rights reserved, whatsoever.
      # Public Domain CC0: http://creativecommons.org/publicdomain/zero/1.0/

      #added by francois scheurer
      #see http://habilis.net/cronic/
      #example in crontab:
      #0 1 * * * CRONIC_LOG=/backup/snapshot/snapshot.log /etc/opt/cronic.sh /backup/snapshot/rsync/backup-mysql.sh #local backup of mysql database
      #endof

      set -eu

      OUT=/tmp/cronic.out.$$
      ERR=/tmp/cronic.err.$$
      TRACE=/tmp/cronic.trace.$$

      set +e
      “$@” >$OUT 2>$TRACE
      RESULT=$?
      set -e

      PATTERN=”^${PS4:0:1}\\+${PS4:1}”
      if grep -aq “$PATTERN” $TRACE
      then
      ! grep -av “$PATTERN” $TRACE > $ERR
      else
      ERR=$TRACE
      fi

      if [ $RESULT -ne 0 -o -s "$ERR" ]
      then
      echo “Cronic detected failure or error output for the command:”
      echo “$@”
      echo
      echo “RESULT CODE: $RESULT”
      echo
      echo “ERROR OUTPUT:”
      cat “$ERR”
      echo
      echo “STANDARD OUTPUT:”
      cat “$OUT”
      if [ $TRACE != $ERR ]
      then
      echo
      echo “TRACE-ERROR OUTPUT:”
      cat “$TRACE”
      fi
      fi

      #added by francois scheurer
      CRONIC_LOG=”${CRONIC_LOG:=}” [ -n "$CRONIC_LOG" ] && cat “$OUT” “$ERR” >>”$CRONIC_LOG”
      #endof

      rm -f “$OUT”
      rm -f “$ERR”
      rm -f “$TRACE”

  4. Does this work on Ubuntu?

    • Francois Scheurer

      Yes Phil404, it was tested on ubuntu as well.

  5. Antonio

    Hi,
    I followed the guide step by step but my userlog.info file remains empty.
    My release is RHEL 6.0
    What can I check to fix the problem?
    Thank you.

    Antonio

    • Francois Scheurer

      Hi Antonio

      some ideas:
      i) try to create the file with ‘touch /var/log/userlog.info’ and then do “/etc/init.d/rsyslog restart”.
      ii) try to comment the line ‘declare -rx AUDIT_SYSLOG=”1″‘ in /etc/bash_franzi and open a new terminal.
      iii) try to manually import the script with ‘. /etc/bash_franzi’.

      if ii) solved the issue it means that we need to check the /etc/rsyslog.conf file.

      Can you post back your results?
      Thank you

  6. Mark

    Sorry,
    I’m newbie, where i put the code at point 2) ???
    A paste the code at the end of /etc/bashrc file and after logon i receive : -bash: warning: here-document at line 94 delimited by end-of-file (wanted `eof’)
    -bash: /etc/bashrc: line 104: syntax error: unexpected end of file
    Thankyou a lot

    Mark

    • Francois Scheurer

      Hi Mark,

      i updated the script and rewrite also some explanations to make them more clear.
      basically when you see something like:
      cat >>/etc/ssh/sshd_config <<"EOF"
      ForceCommand "/etc/forcecommand.sh"
      EOF

      it is a ‘here-document’, so just copy-paste the whole block in your terminal to modify the file (here ‘sshd_config ‘). you can also edit the file manually as you did but then just add the stuff between <<"EOF" and EOF.

  7. Hi

    i set this profile:

    export TTY_NAME=`tty|sed -e ‘s|/dev/||’`
    export REAL_LOGIN=`logname`
    PROMPT_COMMAND=’history -a >(tee -a ~/.bash_history | logger -p user.info -t “$USER[$$][$REAL_LOGIN][$TTY_NAME] $SSH_CONNECTION $PWD”)’
    export HISTTIMEFORMAT=”%F %T ”
    export PS1=’\[\e[1;31m\][\u@\h \W]\$\[\e[0m\] ‘
    readonly REAL_LOGIN
    readonly PROMPT_COMMAND

    But now, when i do “sudo -s” then i m in root, and typing command, if i do “ctrl-c” before hiting ENTER, it logged out from sudo… very annoying because i always do “ctrl-c” before hiting ENTER when i do some mistake in the shell. (for instance i type the begining of a command then i changed my mind, so instead of deleting the command, i just do a ctrl-c).

    How to avoid this behavior with ctrl-c ?

    THanks
    Regards

    • Francois Scheurer

      Hi Thoht

      I could not reproduce your issue, but to become root your can use ‘sudo -i’ or ‘su -’.
      ‘sudo -s’ will not set the user environnement variables…

      As I wrote in the blog article, the solution using ‘PROMPT_COMMAND’ with ‘history -a >(tee -a ~/.bash_history | logger -p user.info’ is working but the syslog messages are sent after the command execution.
      This is not ideal and causes ‘su’ or ‘sudo’ commands to appear only after logouts, and ‘cd’ commands to display the wrong working directory.

      That’s why I would recommend you to use the whole script instead. You can download it here bash_franzi and then source it (‘. /etc/bash_franzi’) manually or automatically by changing your ‘~/.bashrc’.

      If you do not want to use/configure your ‘rsyslogd’ you can comment out or delete the line ‘declare -rx AUDIT_SYSLOG=”1″‘ .

      cheers

  8. Christof

    Thank you very much for this very useful script. It works almost perfect on CentOS 6.4.

    One issue:

    1. login with user “christof”
    2. su -
    3. tail -f /var/log/userlog.info
    4. new login with user christof
    5. now i just press enter -> the command at line 2 is logged again
    6. su -
    7. again just enter -> the command at line 3 is logged again

    Any hints?

    • Francois.Scheurer

      Hi Christof

      Normally the problem you describe was already fixed in the script, at the line:
      if [ "${AUDIT_HISTLINE}" != "${AUDIT_LASTHISTLINE}" ] #avoid logging unexecuted commands after ‘ctrl-c’, ‘empty+enter’, or after ‘ctrl-d’

      Maybe your shell does not support extended pattern matching operators?
      Which shell version are you using?

      Can you manually test this:
      bash -version
      shopt -s extglob
      C=”$(history 1)”
      echo “$C => ${C%%+([^ 0-9])*}”

      by me I get something like:
      root@ZRHSV-TST01:~# bash -version
      GNU bash, version 4.2.37(1)-release (x86_64-pc-linux-gnu) [...]
      root@ZRHSV-TST01:~# shopt -s extglob
      root@ZRHSV-TST01:~# C=”$(history 1)”
      root@ZRHSV-TST01:~# echo “$C => ${C%%+([^ 0-9])*}”
      43483 C=”$(history 1)” => 43483

      If you do not get twice the same number then it explains the problem (your shell does not support parameter expansion).
      In this case you should try to replace the line:
      AUDIT_HISTLINE=”${AUDIT_CMD%%+([^ 0-9])*}”
      with something using extern sed-program in place of the bash builtins:
      AUDIT_HISTLINE=”$(sed ‘s/^([ 0-9]+).*/1/’ <<<$AUDIT_CMD)”

      i hope it helps…
      cheers!

      • Christof

        Thank you!
        I used bash version 4.1.2 on CentOS 6.4 and i did not get the same number twice when i executed the commands… so parameter extension was not supported.
        So i tried using sed… same result.
        So i compiled and installed bash version 4.2.0. Now i do get the same number twice. But i still have duplicate entries in userlog.info after a new login.
        Maybe you have an other hint for me.
        Anyway, thanks for spending the time…

        • Francois Scheurer

          Hi Christof,

          I just noticed the same problem by me (double entries) for the first command after a new login, if the history is empty. this may happen for a new user where .bash_history does not exist.

          I updated the bash_franzi script to fix that rare issue.
          maybe it also solve yours.

          cheers

  9. Daniel La Laina

    Hi, thank you very much !!!

    daniel.lalaina@dwarf10:~$ sudo /etc/init.d/syslog stop
    Shutting down kernel logger: [ OK ]
    Shutting down system logger: [ OK ]

    daniel.lalaina@dwarf10:~$ sudo rm -rf /var/log/message*

    daniel.lalaina@dwarf10:~$ sudo rm -rf /var/log/secure*

    daniel.lalaina@dwarf10:~$ sudo /etc/init.d/rsyslog start
    Starting system logger: [ OK ]

    daniel.lalaina@dwarf10:~$ cat aa

    daniel.lalaina@dwarf10:~$ sudo cat /var/log/messages
    Feb 28 00:19:06 dwarf10 kernel: imklog 3.22.1, log source = /proc/kmsg started.
    Feb 28 00:19:06 dwarf10 rsyslogd: [origin software="rsyslogd" swVersion="3.22.1" x-pid="17470" x-info="http://www.rsyslog.com"] (re)start
    Feb 28 00:19:06 dwarf10 rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c3 as the first rsyslogd option.
    Feb 28 00:19:06 dwarf10 rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock
    Feb 28 00:19:14 dwarf10 [audit daniel.lalaina/30539 as daniel.lalaina/30544 on pts/1/200.143.XXX.XXX:51467->200.219.XXX.XXX:22] /home/daniel.lalaina: 28/02/2013 00:19:14 cat aa
    Feb 28 00:19:29 dwarf10 [audit daniel.lalaina/30539 as daniel.lalaina/30544 on pts/1/200.143.XXX.XXX:51467->200.219.XXX.XXX:22] /home/daniel.lalaina: 28/02/2013 00:19:29 sudo cat /var/log/messages

    daniel.lalaina@dwarf10:~$ sudo cat /var/log/secure
    Feb 28 00:19:29 dwarf10 sudo: daniel.lalaina : TTY=pts/1 ; PWD=/home/daniel.lalaina ; USER=root ; COMMAND=/bin/cat /var/log/messages
    Feb 28 00:19:41 dwarf10 sudo: daniel.lalaina : TTY=pts/1 ; PWD=/home/daniel.lalaina ; USER=root ; COMMAND=/bin/cat /var/log/secure

    daniel.lalaina@dwarf10:~$ sudo cat /var/log/userlog.info
    2013-02-28T00:19:14.241482+00:00 dwarf10 [audit daniel.lalaina/30539 as daniel.lalaina/30544 on pts/1/200.143.57.148:51467->200.219.xxx:22] /home/daniel.lalaina: 28/02/2013 00:19:14 cat aa
    2013-02-28T00:19:29.630405+00:00 dwarf10 [audit daniel.lalaina/30539 as daniel.lalaina/30544 on pts/1/200.143.57.148:51467->200.219.xxx:22] /home/daniel.lalaina: 28/02/2013 00:19:29 sudo cat /var/log/messages
    2013-02-28T00:19:41.038860+00:00 dwarf10 [audit daniel.lalaina/30539 as daniel.lalaina/30544 on pts/1/200.143.57.148:51467->200.219.xxx:22] /home/daniel.lalaina: 28/02/2013 00:19:41 sudo cat /var/log/secure
    2013-02-28T00:25:17.954531+00:00 dwarf10 [audit daniel.lalaina/30539 as daniel.lalaina/30544 on pts/1/200.143.57.148:51467->200.219.xxx:22] /home/daniel.lalaina: 28/02/2013 00:25:17 sudo cat /var/log/userlog.info

    Could you help me with this:

    1 – As you can see above, it´s triplicating sudo commands in the logs, and duplicating normal commands. I don´t need any type of command to be writen on ‘messages’, and the same for sudo commands on ‘secure’(keeping sshd, etc).

    2 – I installed rsyslog by yum, and stopped syslog, am I going to loose anything with it? I’m asking it because I saw that syslog has both ‘Shutting down kernel logger’ and ‘Shutting down system logger’, and the rsyslog has just the ‘Starting system logger’.

    Again, really really thank you!

    Regards.

    • Francois.Scheurer

      hi Daniel!

      thx for your comment.

      1- If you see the bash audit lines in many log files, you can disable user.info for theses files, just edit your syslog / rsyslog configuration, for example:
      user.*;user.!info -/var/log/messages

      see http://linux.about.com/od/commands/l/blcmdl5_syslogc.htm for more info.

      2- You can also keep using syslog instead of rsyslog .
      you will only need to check that syslog configuration
      (/etc/syslog.conf) includes a line like:
      user.info /var/log/userlog.info

      And if you use rsyslog you can check that it has a line with kern.* -/var/log/kern.log .
      I would recommend you to keep the default logger of your distribution, also in your case syslog.

      cheers

      • Daniel La Laina

        Thanks!

        I removed rsyslog, and put on syslog.conf:
        *.info;user.none,mail.none;authpriv.none;cron.none /var/log/messages
        user.* /var/log/userlog.info

        Just one more help, every time I login it says:
        -bash: PROMPT_COMMAND: readonly variable

        • Daniel La Laina

          -bash: PROMPT_COMMAND: readonly variable
          This was happening because /etc/bash_franzi sourced by /etc/profile declared PROMPT_COMMAND as read only, and then /etc/bashrc tries to modify.
          I removed from the source from /etc/profile, and put on the end of /etc/bashrc, now it´s working fine, even when the user type the command ‘bash’ (which was a vulnerability when sourced by /etc/profile).
          Now I just source /etc/bash_franzi from /etc/bashrc, and not on /home/.bashrc, /etc/profile, etc, etc. It´s working fine, and I don´t have to add the source /etc/bash_franzi on /home/.bashrc everytime I add a new user.
          Oh, and a good thing to include on your script, is to remove all other shells from /etc/shells, because if the user type ‘csh’ on command line for example, next commands will not be logged.
          I have changed all /home/.bashrc and /home/.bash_profile to read only, is there a way to do that automatically for every new users?

          • Francois.Scheurer

            So you found a solution to avoid the re-assignement. ^^
            Regarding security, i think it will be hard if not impossible to avoid 100% a user to bypass the audit.
            For example a user could write the commands in a file then execute that file to hide himself.
            I see this audit more as a helping tool for sysadmins: to coordinate themeselves, to remember past changes, to troubleshoot problems by finding time correlations between timestamps of files, log content and bash commands, etc. .
            But still, it is good to make it as most secure as possible, many thanks for your idea about /etc/shells.
            To make the user files read-only you can apply the changes on /etc/skel, it will be then used for future user accounts.

            Important: even the read-only setting on a file can be bypassed by a user: he can delete or rename the file and then replace it by a new one…
            To avoid this you can make the user folder read-only (but then the user may only add files into subfolders… quite tough).
            Another work-around is to make the user folder sticky (chmod o+t), set root as owner of the user folder and of the file, and set the file read-only.
            At last solution you can use ‘chattr +i’ on that file to make it immutable; this is a bit an overkill and unrecommended solution (too far from standard/maintstream, will break compatibility with maintenance scripts, etc.) and it will also need to be done on each user (not possible to use (etc/skel here).

        • Francois.Scheurer

          Yes this readonly variable error is actually a “security feature” ;-)
          The bash_franzi script protect PROMPT_COMMAND against modification, and another another initialization script tried to change it.
          (maybe /etc/profile or ~/.bashrc ?)
          You may change my script by replacing “declare -rx PROMPT_COMMAND” with “declare -x PROMPT_COMMAND” or change the other script by commenting out the line that try to change PROMPT_COMMAND.

          Or like me you may just ignore that error. ;-)

        • Francois.Scheurer

          BTW, be careful that cron, authpriv, mail and user facilities are still written in some log files after doing that change in syslog.conf .

  10. Francois.Scheurer

    For info, I just posted an update of the script today with a new pattern matching in AUDIT_DEBUG() that is improved for speed.
    The older pattern was sometimes VERY slow, especially with long multi-lines commands like:
    cat < ...
    EOF

    You can download it here: http://blog.pointsoftware.ch/wp-content/uploads/2012/09/bash_franzi.txt

  11. Awesome script! Using it successfully on Debian, CentOS, RHEL and SLES. Great job, Franzi!

    • Francois.Scheurer

      Thank you Claudio!

  12. Peter

    Greetings,

    This seems to work very well. I’ve tweaked the message formatting a bit to be more machine-readable, but other than that it’s very nice.

    I was wondering, though, if you were experiencing the following little bug:

    sudo su -
    ^D

    (That is, ctrl-D after successfully becoming root with “sudo su -”.) In my syslogs, I get the last line of my /root/.bash_history file, even though I didn’t execute that command. Unfortunately, I’m not quite smart enough to figure out where the problem lies, but can you confirm that you see this problem?

    Like I said, it’s pretty minor and won’t keep me from using this approach. But I thought I’d mention it. :-)

    Peter

    • Peter

      Ack, please ignore my last comment. It would appear that by messing with the logging format, I broke the protection against “logging unexecuted commands”. I can fix it on my end. :-)

      Thanks again,
      Peter

      • Francois.Scheurer

        Hi Peter

        thx for posting.

        did you fix the problem?

        else check this line from the script:
        if [ "${AUDIT_HISTLINE}" != "${AUDIT_LASTHISTLINE}" ] #avoid logging unexecuted commands after ‘ctrl-c’, ‘empty+enter’, or after ‘ctrl-d’

        this line especially avoid ctrl-D to log the last command twice.

        best regards
        francois

  13. Fagood

    Working perfectly here on multi users production Linux servers with Debian squeeze ! Very useful when you have several sys admins using the root user, or developers working on a services and modifying files. Also interesting to monitor and audit changes and even to have some sort of commands backup when installing something.Thanks for sharing ! Keep up the good work :) Cheers

    • Francois.Scheurer

      Thank you for the post! And also for having shown me the great parallel dancing shell ‘pdsh’.

      I will post a howto on this topic here later.

  14. Halocaridina

    Hi Francois,

    Thank you for the reply. Yes, I work at a University in the US. I’ll be contacting you at the email that you provided to supply the link to the Scientific Linux ISO.

    Cheers,

    Halocaridina

  15. Halocaridina

    Hi Francois,

    Thank you again for the suggestion. I commented out the $PS1 variable as well as the echo command and the wrap-around issue disappeared as predicted. Auditing continues to work as expected, so it is definitely something to do with $PS1. My users are happy with the standard prompt since the wrap-around issue was more difficult live with.

    The information that you requested is:

    Fully updated Scientific Linux (a RHEL clone) 6.1

    Linux 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 15:16:22 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux

    GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)

    The system is headless and all access is by SSH. Every terminal application used was affected, including iTerm, OS X Terminal, Terminator, Gnome Terminal, PuTTy, etc.

    I’ll be interested to see what you find.

    Cheers,

    Halocaridina

    • Francois.Scheurer

      Hi Halocaradina

      Thanks for the feedback, I am glad that you have it working now for your users.
      May I ask if you are working for a university or a scientific group?
      BTW my Email is francois d0t scheurer @t pointsoftware d0t ch .

      Regarding the prompt variable, I should give a try with your Scientific Linux distrib.
      Do you have a download link? ideal would be a Live-CD ISO file.

      Cheers,
      Francois

  16. Halocaridina

    Hi Francois,

    Thank you for the information. Unfortunately, it doesn’t appear to be an issue with the $COLUMNS shell variable.

    Issuing:

    set | egrep ‘(COLUMNS|LINES)’

    in any user account (which source the audit system and experiences the wrap-around issue) produces:

    COLUMNS=222
    LINES=67

    which is identical in output when running the same command as root (which does not source the audit system and wraps correctly).

    Any further suggestions?

    Cheers and thank you again,

    Halocaridina

    • Francois.Scheurer

      Hi Halocaridina

      Ok so as you wrote in the 1st message your issue is then probably related to the colorized prompt and its $PS1 variable that contains escape codes.

      You can check if the issue disappears when you keep your prompt unmodified by removing the lines:

      PS1=”[${_backblue}${_frontgreen_b}]u@h:[${_backblack}${_frontblue_b}]w\$[${_backnone}${_frontgreen_b}] ”

      (at the beginning of the script)

      echo -ne “${_backnone}${_frontgrey}” #disable prompt colors for the command’s output

      (in BASH_AUDIT function)

      Can you say the version of your ‘bash’, your operating system and the name of your terminal? (putty, gnome-terminal, etc.).
      I would be interested to reproduce it here for testing.

      Anyway if your issue is solved with the standart prompt then the auditing will work as it should, the only difference will be cosmetical while it will not have the same colors in your prompt.

      cheers
      Francois

  17. Adi

    Interesting Article;

    How to implement this method for solaris ?

    Thnxs

    • Francois.Scheurer

      Hi Adi

      Yes it works with Solaris with minor changes, but I did not test it completely under this OS.

      In Solaris please use ‘grep’ without the ‘-q’ option, like this:
      if groups | grep root &>/dev/null

      Please also remove the following line (chattr is unsupported in Solaris):
      chattr +a “$HISTFILE”

      Then modify your /etc/syslog.conf to include this line:
      user.info /var/adm/userlog.info

      To assign ‘bash’ as the login shell in Solaris: passwd -e /bin/bash .

      Make sure that the audit-script is sourced (=included) correctly during the bash invocation (Step 2 above).

      best regards

      PS: if your bash version is too old, $HISTCONTROL will not allow you to log duplicated commands correctly.

      • Hi,

        I followed your instructions to run the script in Solaris 11 but did not succeed. One message that I get is:

        cat: cannot open /proc/1576/cmdline: No such file or directory

        Nor will dump commands to log file.

        • Francois.Scheurer

          Hi Juan!

          thanks for your comment.

          ok have now disabled this check on /proc/$$/cmdline and the other section with ‘chattr’ from the script, if $OSTYPE is starting with the string ‘solaris’. it should then work for you.

          just tell me if the script is still having problem with solaris.

          cheers

  18. Halocaridina

    Dear Francois,

    Thank you for posting this. It works very well in my testing. The only issue I have run into is that when one types a long command at the prompt, the command will wrap over itself (cover up the beginning of the command being typed) in a random fashion. The command will successfully complete once you press “Enter”, so it looks to be something with the prompt (PS1).

    Any suggestions on what might be the problem? This is with:

    GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)

    following the above directions for installing and configuring the audit system.

    Thank you in advance.

    • Francois.Scheurer

      Hi Halocaridina

      Sorry for the late reply.. It seems that you have an issue with the $COLUMNS shell variable.

      You may try to type the command ‘resize’ if you have installed it to retrieve the correct terminal’s width.

      Or you can try in bash to type ‘shopt -s checkwinsize’ to recheck the window’s width after every command. Or send a signal with ‘kill -WINCH $$’

      Finally you can also set it manually, e.g.:
      stty rows 24 columns 80
      and verify with:
      set | egrep ‘(COLUMNS|LINES)’
      or:
      stty size

      Hope that it helps..

      cheers

Hinterlasse eine Antwort

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind markiert *