Bash has handy brace expansion powers that I’ve belatedly discovered.
$ echo I love hippo{griffs,potamuses,dromes}
I love hippogriffs hippopotamuses hippodromes
For example, you can quickly diff a file with and without a suffix:
$ echo diff .env{,.example}
diff .env .env.example
Or tail multiple log files:
$ echo tail -f /var/log/{messages,secure}
tail -f /var/log/messages /var/log/secure
Bash brace expansion can do other things too, such as specify a range with a .. operator.
You may need to update your OpenLDAP SSL certificate, as well as the CA certificate and signing key on a regular basis. I ran into an issue that was ultimately resolved by doing that.
Connections to an OpenLDAP server I administer stopped working with this error:
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
The server itself was up and the relevant ports were accessible. In fact, unencrypted LDAP continued to work while LDAPS saw the error above.
I restarted slapd with -d 255 flag (-d 8 is sufficient for this error) and started seeing this error:
TLS: error: could not initialize moznss security context - error -5925:The one-time function was previously called and failed. Its error code is no longer available TLS: can't create ssl handle.
At the start of the log, I saw several related errors including this one:
... is not valid - error -8181:Peer's Certificate has expired..
Ultimately this meant that I had to replace not just my certificate but also the CA certificate and the signing key in OpenLDAP’s moznss database. I believe my CA’s certificate had to be replaced because of the SHA1 retirement last year.
The steps I had to follow were surprisingly involved and undocumented:
certutil -L -d .
certutil -D -d . -n "OpenLDAP Server" certutil -D -d . -n "My CA Certificate"
certutil -A -n "OpenLDAP Server" -t CTu,u,u -d . -a -i ../ssl/my_certificate.bundle.crt certutil -A -n "My CA Certificate" -t CT,C,c -d . -a -i ../ssl/my_CA_certificate.intermediate.crt
certutil -L -d .
openssl pkcs12 -export -out ../ssl/my_certificate.key.pkcs12 -inkey ../ssl/my_certificate.key -in ../ssl/my_certificate.bundle.crt -certfile ../ssl/my_CA_certificate.intermediate.crt
pk12util -i ../ssl/my_certificate.key.pkcs12 -d . # Database password is in /etc/openssl/certs/password # Key password is what you set above
service slapd restart
I hope that’s enough to help anyone facing the same problem on CentOS, RHEL, Fedora, and possibly other distros.
I am pretty excited about the release of git 2.9. It brings several new features that make reviewing changes easier and more sensible. It has better change grouping, and it can highlight individual changed words. Everyone should set these configuration options to enable better git diffs.
Before you can enable the new settings, you have to upgrade your git installation.
If you already have git installed through homebrew, you can upgrade it as follows:
brew update && brew upgrade
If you do not have git installed through homebrew, you’ll want to override your ancient Mac git by installing it as follows:
brew install git
Once you have upgraded your git, you can put the new configuration in place.
The first major change is an improvement to how git groups changes in a diff. When you add a new block of code, it’s now likelier to see the whole block as a change rather than misinterpreting it as an insertion splitting an existing block into two.
The second change is the addition of more places for you to hook in the diff-highlight utility.
diff-highlight post-processes your diffs to add more highlighting to the specific changes between two lines when you just change a few words in a line.
You can enable all of these by running the following commands in your terminal:
git config --global diff.compactionHeuristic 1 git config --global pager.log "`brew --prefix`/share/git-core/contrib/diff-highlight/diff-highlight | less" git config --global pager.show "`brew --prefix`/share/git-core/contrib/diff-highlight/diff-highlight | less" git config --global pager.diff "`brew --prefix`/share/git-core/contrib/diff-highlight/diff-highlight | less" git config --global interactive.diffFilter "`brew --prefix`/share/git-core/contrib/diff-highlight/diff-highlight"
The configuration will persist in a ~/.gitconfig file.
You should now have easier to read and compare git diffs. I certainly appreciate the changes.
I use the bash command line on my Mac a lot. I typically have multiple tabs with multiple terminal panes open in iTerm2, often with multiple ssh sessions running. By default, the last terminal session to close trashes the bash history of all the other sessions. Is it possible to configure the terminal to preserve bash history?
It’s actually fairly straightforward to preserve the history.
Open up your ~/.bash_profile configuration file in an editor of your choice such as nano. Once you have it open, add these lines at the end:
# Maximum number of history lines in memory
export HISTSIZE=50000
# Maximum number of history lines on disk
export HISTFILESIZE=50000
# Ignore duplicate lines
export HISTCONTROL=ignoredups:erasedups
# When the shell exits, append to the history file
# instead of overwriting it
shopt -s histappend
# After each command, append to the history file
# and reread it
export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"
Save the file an exit. In order for your configuration change to take effect, you will need to reload the configuration in all your open terminal sessions:
source ~/.bash_profile
This configuration change has to be done per user per machine.
I use mackup together with Dropbox to keep my bash and other command line configuration files backed up. This makes it easy to transfer your command line configuration to a new primary machine.
iTerm2 is my terminal of choice on the Mac. It has great tab and pane management accessible via both keyboard and mouse, and some subtle quality of life features.
For example, if you ssh somewhere, it sets the tab title to the hostname of the remote machine, or the name of the local directory.
One drastic alternative would be to migrate from Bash to an alternate shell like Fish or, in the future, the Next Generation Shell (NGS).
The project I’m working on right now involve not just Dockerized Rails microservices, Meteor JS, and a data set measured in tens of terabytes, but also a big Bash code base.
Bash is a language that makes it easy to shoot yourself in the foot. I have some thoughts on how to write robust, modular, loosely coupled, unit tested Bash that go beyond Bash strict mode and shellcheck. However, let’s save those for later.
Here’s a useful shell command for today: watch
watch will run a command for you every few seconds and output the results on a clean screen. If you combine it with a split-screen or split-pane tool, you can quickly create a mini-dashboard.
For example, watch -n15 df -h will print your free disk space every 15 seconds:
Every 15.0s: df -h Tue Apr 14 13:43:25 2015 Filesystem Size Used Avail Use% Mounted on /dev/sda3 1.8T 691G 1.1T 40% / tmpfs 253G 12K 253G 1% /dev/shm /dev/sda1 248M 58M 178M 25% /boot /dev/sdb1 30T 12T 18T 41% /my_disk
By way of another example, watch -n60 db2 list utilities show detail will check on the status of your DB2 load and other operations every 60 seconds.
Every 60.0s: db2 list utilities show detail Mon Apr 13 12:06:24 2015 ID = 4 Type = LOAD Database Name = MY_DB Member Number = 0 Description = [LOADID: 1177.2015-04-13-15.58.12.282010.0 (20;15)] [*LOCAL.MY_USER.150413204511] OFFLINE LOAD DEL AUTOMATIC INDEXING INSERT NON-RECOVERABLE MY_SCHEMA .MY_TABLE Start Time = 04/13/2015 15:58:12.298788 State = Executing Invocation Type = User Progress Monitoring: Phase Number = 1 Description = SETUP Total Work = 0 bytes Completed Work = 0 bytes Start Time = 04/13/2015 15:58:12.298799 Phase Number [Current] = 2 Description = LOAD Total Work = 4105362974 rows Completed Work = 1069904365 rows Start Time = 04/13/2015 15:58:15.821695 Phase Number = 3 Description = BUILD Total Work = 2 indexes Completed Work = 0 indexes Start Time = Not Started
Mac OS X does not include watch by default. Assuming you have Homebrew, the following will install watch for you:
brew install watch
I got this error starting a ruby application:
/usr/local/rvm/gems/ruby-1.9.3-p547/gems/bundler-1.7.3/lib/bundler/runtime.rb:76:in `require': libdb2.so.1: cannot open shared object file: No such file or directory - /usr/local/rvm/gems/ruby-1.9.3-p547/extensions/x86_64-linux/1.9.1/ibm_db-2.5.11/ibm_db.so (LoadError)
An .so is a Linux library, equivalent to a .dll on Windows or a .dylib on Mac. Note that there are two different libraries mentioned. ibm_db.so is present, while libdb2.so.1 is missing.
You can verify the dependencies using the ldd command:
$ ldd /usr/local/rvm/gems/ruby-1.9.3-p547/extensions/x86_64-linux/1.9.1/ibm_db-2.5.11/ibm_db.so
linux-vdso.so.1 => (0x00007fff93545000)
libruby.so.1.9 => /usr/local/rvm/rubies/ruby-1.9.3-p547/lib/libruby.so.1.9 (0x00007fb8ba5a5000)
libdb2.so.1 => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb8ba37f000)
librt.so.1 => /lib64/librt.so.1 (0x00007fb8ba177000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fb8b9f72000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fb8b9d3b000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb8b9ab7000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb8b9722000)
/lib64/ld-linux-x86-64.so.2 (0x0000003730400000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007fb8b94ab000)
For me, the issue was caused by a missing IBM Data Server Driver directory. IBM_DB_LIB was pointing to a non-existent directory:
$ set | grep IBM IBM_DB_DIR=/tmp/dsdriver/. IBM_DB_HOME=/tmp/dsdriver/. IBM_DB_INCLUDE=/tmp/dsdriver/./include IBM_DB_LIB=/tmp/dsdriver/./lib
Reinstalling the Data Server Driver restored the libdb2.so.1 and eliminated the error.
(If you run into this issue with different libraries, you will likely need to examine your LD_LIBRARY_PATH environment variable and use the ldconfig command to reload any changes.)
Do you ever need to kick off a long-running command while SSHed to a server, but be able to disconnect and reconnect at will? You can do this with screen.
Before doing anything, start a screen session:
screen
When you’re ready to put your work on hold, detach the screen:
screen -d
If you have a long-running command running, you can detach that screen from a different shell session by specifying the process id:
# look up the process id ps aux | grep SCREEN # detach that screen screen -d <my_process_id>
You can now disconnect from the server safely.
When you reconnect, you can also reconnect to your screen session:
screen -r
banner can be a useful command for setting login and welcome messages (e.g via /etc/profile).
$ banner PRODUCTION ###### ###### ####### ###### # # ##### ####### ### ####### # # # # # # # # # # # # # # # # # # ## # # # # # # # # # # # # # # # # # # # ###### ###### # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ## # # # ####### ###### ##### ##### # ### ####### # #
The Mac implementation is less useful than the GNU/Linux one.
$ banner -w 40 PRO
# #
# #
###########################
###########################
# # #
# # #
# #
# #
## ##
## ###
##########
########
####
# #
# #
###########################
###########################
# ## #
# ##### #
######## #
######### # #
# ######### ## ##
######### ## ###
##### ##########
### ########
# ####
#
#######
###############
###################
###### #######
#### ###
## ##
# #
# #
# ##
### ###
###### #####
#####################
#################
###########
I ran into this error when running ec2-upload-bundle:The specified bucket is not S3 v2 safe (see S3 documentation for details)This was due to uppercase letters or underscores. Later I also ran into an issue with periods in bucket names which showed up as this error message:ERROR: Error talking to S3: Server.AccessDenied(403): Access DeniedHere is an easy command to sanitize the bucket names:
sanitized_name=$( echo $name | tr [:upper:] [:lower:] | tr [:punct:] - )
It will lowercase all letters and convert all punctuation to dashes.
By default, Bash treats uninitialized variables the same way as Perl — they are blank strings. If you want them treated more like Python, you can issue the following command in your bash script:
set -u
You will then start seeing warning messages like the following:
./my_script.sh: line 419: FOO_BAR: unbound variable
Note that this mean you can’t check for the non-existence of environment variables with a simple [[ -z “$ENVIRONMENT_VARIABLE” ]]. Instead, you could do something like the following:
[[ $( set | grep "ENVIRONMENT_VARIABLE=" | wc -l ) -lt 1 ]]