Category: Unix

You may need to update your OpenLDAP SSL certificate, as well as the CA certificate and signing key on a regular basis. I ran into an issue that was ultimately resolved by doing that.

Connections to an OpenLDAP server I administer stopped working with this error:

ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

The server itself was up and the relevant ports were accessible. In fact, unencrypted LDAP continued to work while LDAPS saw the error above.

I restarted slapd with -d 255 flag (-d 8 is sufficient for this error) and started seeing this error:

TLS: error: could not initialize moznss security context - error -5925:The one-time function was previously called and failed. Its error code is no longer available TLS: can't create ssl handle.

At the start of the log, I saw several related errors including this one:

... is not valid - error -8181:Peer's Certificate has expired..

Ultimately this meant that I had to replace not just my certificate but also the CA certificate and the signing key in OpenLDAP’s moznss database. I believe my CA’s certificate had to be replaced because of the SHA1 retirement last year.

The steps I had to follow were surprisingly involved and undocumented:

• Upload the new certificates to /etc/openldap/ssl
• cd /etc/openldap/certs
• List the existing certificates in the database:

certutil -L -d .

• Remove the existing certs:

certutil -D -d . -n "OpenLDAP Server"

certutil -D -d . -n "My CA Certificate"

• Load the new OpenLDAP SSL certificate and CA certificate:

certutil -A -n "OpenLDAP Server" -t CTu,u,u -d . -a -i ../ssl/my_certificate.bundle.crt

certutil -A -n "My CA Certificate" -t CT,C,c -d . -a -i ../ssl/my_CA_certificate.intermediate.crt

• Verify:

certutil -L -d .

• Convert the key to pkcs12 format:

openssl pkcs12 -export -out ../ssl/my_certificate.key.pkcs12 -inkey ../ssl/my_certificate.key -in ../ssl/my_certificate.bundle.crt -certfile ../ssl/my_CA_certificate.intermediate.crt

• Import the signing key:

pk12util -i ../ssl/my_certificate.key.pkcs12 -d .

# Database password is in /etc/openssl/certs/password

# Key password is what you set above

• Restart slapd:

service slapd restart

I hope that’s enough to help anyone facing the same problem on CentOS, RHEL, Fedora, and possibly other distros.

See Also

• Managing certificate databases on RHEL
• Add an existing key to the certutil key database

I am pretty excited about the release of git 2.9. It brings several new features that make reviewing changes easier and more sensible.  It has better change grouping, and it can highlight individual changed words. Everyone should set these configuration options to enable better git diffs.

Upgrade your git

Before you can enable the new settings, you have to upgrade your git installation.

If you already have git installed through homebrew, you can upgrade it as follows:

brew update && brew upgrade

If you do not have git installed through homebrew, you’ll want to override your ancient Mac git by installing it as follows:

brew install git

Enable better git diffs

Once you have upgraded your git, you can put the new configuration in place.

The first major change is an improvement to how git groups changes in a diff. When you add a new block of code, it’s now likelier to see the whole block as a change rather than misinterpreting it as an insertion splitting an existing block into two.

The second change is the addition of more places for you to hook in the diff-highlight utility.

diff-highlight post-processes your diffs to add more highlighting to the specific changes between two lines when you just change a few words in a line.

You can enable all of these by running the following commands in your terminal:

git config --global diff.compactionHeuristic 1

git config --global pager.log "`brew --prefix`/share/git-core/contrib/diff-highlight/diff-highlight | less"
git config --global pager.show "`brew --prefix`/share/git-core/contrib/diff-highlight/diff-highlight | less"
git config --global pager.diff "`brew --prefix`/share/git-core/contrib/diff-highlight/diff-highlight | less"

git config --global interactive.diffFilter "`brew --prefix`/share/git-core/contrib/diff-highlight/diff-highlight"

The configuration will persist in a ~/.gitconfig file.

You should now have easier to read and compare git diffs. I certainly appreciate the changes.

Save

The project I’m working on right now involve not just Dockerized Rails microservices, Meteor JS, and a data set measured in tens of terabytes, but also a big Bash code base.

Bash is a language that makes it easy to shoot yourself in the foot. I have some thoughts on how to write robust, modular, loosely coupled, unit tested Bash that go beyond Bash strict mode and shellcheck. However, let’s save those for later.

Here’s a useful shell command for today: watch

watch will run a command for you every few seconds and output the results on a clean screen. If you combine it with a split-screen or split-pane tool, you can quickly create a mini-dashboard.

For example, watch -n15 df -h will print your free disk space every 15 seconds:

Every 15.0s: df -h        Tue Apr 14 13:43:25 2015

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             1.8T  691G  1.1T  40% /
tmpfs                 253G   12K  253G   1% /dev/shm
/dev/sda1             248M   58M  178M  25% /boot
/dev/sdb1              30T   12T   18T  41% /my_disk

By way of another example, watch -n60 db2 list utilities show detail will check on the status of your DB2 load and other operations every 60 seconds.

Every 60.0s: db2 list utilities show detail            Mon Apr 13 12:06:24 2015

ID                               = 4
Type                             = LOAD
Database Name                    = MY_DB
Member Number                    = 0
Description                      = [LOADID: 1177.2015-04-13-15.58.12.282010.0 (20;15)] 
[*LOCAL.MY_USER.150413204511] OFFLINE LOAD DEL
AUTOMATIC INDEXING INSERT NON-RECOVERABLE MY_SCHEMA    .MY_TABLE
Start Time                       = 04/13/2015 15:58:12.298788
State                            = Executing
Invocation Type                  = User
Progress Monitoring:
   Phase Number                  = 1
      Description                = SETUP
      Total Work                 = 0 bytes
      Completed Work             = 0 bytes
      Start Time                 = 04/13/2015 15:58:12.298799

   Phase Number [Current]        = 2
      Description                = LOAD
      Total Work                 = 4105362974 rows
      Completed Work             = 1069904365 rows
      Start Time                 = 04/13/2015 15:58:15.821695

   Phase Number                  = 3
      Description                = BUILD
      Total Work                 = 2 indexes
      Completed Work             = 0 indexes
      Start Time                 = Not Started

Mac OS X does not include watch by default. Assuming you have Homebrew, the following will install watch for you:

brew install watch

I got this error starting a ruby application:

/usr/local/rvm/gems/ruby-1.9.3-p547/gems/bundler-1.7.3/lib/bundler/runtime.rb:76:in `require': 
libdb2.so.1: cannot open shared object file: 
No such file or directory - 
/usr/local/rvm/gems/ruby-1.9.3-p547/extensions/x86_64-linux/1.9.1/ibm_db-2.5.11/ibm_db.so (LoadError)

An .so is a Linux library, equivalent to a .dll on Windows or a .dylib on Mac. Note that there are two different libraries mentioned. ibm_db.so is present, while libdb2.so.1 is missing.

You can verify the dependencies using the ldd command:

$ ldd /usr/local/rvm/gems/ruby-1.9.3-p547/extensions/x86_64-linux/1.9.1/ibm_db-2.5.11/ibm_db.so
    linux-vdso.so.1 =>  (0x00007fff93545000)
    libruby.so.1.9 => /usr/local/rvm/rubies/ruby-1.9.3-p547/lib/libruby.so.1.9 (0x00007fb8ba5a5000)
    libdb2.so.1 => not found
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb8ba37f000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fb8ba177000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fb8b9f72000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fb8b9d3b000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fb8b9ab7000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fb8b9722000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003730400000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007fb8b94ab000)

For me, the issue was caused by a missing IBM Data Server Driver directory. IBM_DB_LIB was pointing to a non-existent directory:

$ set | grep IBM
IBM_DB_DIR=/tmp/dsdriver/.
IBM_DB_HOME=/tmp/dsdriver/.
IBM_DB_INCLUDE=/tmp/dsdriver/./include
IBM_DB_LIB=/tmp/dsdriver/./lib

Reinstalling the Data Server Driver restored the libdb2.so.1 and eliminated the error.

(If you run into this issue with different libraries, you will likely need to examine your LD_LIBRARY_PATH environment variable and use the ldconfig command to reload any changes.)

Do you ever need to kick off a long-running command while SSHed to a server, but be able to disconnect and reconnect at will? You can do this with screen.

Before doing anything, start a screen session:

screen

When you’re ready to put your work on hold, detach the screen:

screen -d

If you have a long-running command running, you can detach that screen from a different shell session by specifying the process id:

# look up the process id
ps aux | grep SCREEN
# detach that screen
screen -d <my_process_id>

You can now disconnect from the server safely.

When you reconnect, you can also reconnect to your screen session:

screen -r

 

banner can be a useful command for setting login and welcome messages (e.g via /etc/profile).

$ banner PRODUCTION
######  ######  ####### ######  #     #  #####  #######   ###   ####### #     #
#     # #     # #     # #     # #     # #     #    #       #    #     # ##    #
#     # #     # #     # #     # #     # #          #       #    #     # # #   #
######  ######  #     # #     # #     # #          #       #    #     # #  #  #
#       #   #   #     # #     # #     # #          #       #    #     # #   # #
#       #    #  #     # #     # #     # #     #    #       #    #     # #    ##
#       #     # ####### ######   #####   #####     #      ###   ####### #     #

The Mac implementation is less useful than the GNU/Linux one.

$ banner -w 40 PRO
          #                         #
          #                         #
          ###########################
          ###########################
          #              #          #
          #              #          #
                         #          #
                         #          #
                         ##        ##
                         ##       ###
                          ##########
                           ########
                             ####
          #                         #
          #                         #
          ###########################
          ###########################
          #             ##          #
          #          #####          #
                  ########          #
               ######### #          #
          # #########    ##        ##
          #########      ##       ###
          #####           ##########
          ###              ########
          #                  ####
          #
                    #######
                ###############
              ###################
            ######          #######
          ####                   ###
          ##                       ##
          #                         #
          #                         #
          #                        ##
          ###                     ###
           ######             #####
             #####################
               #################
                  ###########

 

I ran into this error when running ec2-upload-bundle:The specified bucket is not S3 v2 safe (see S3 documentation for details)This was due to uppercase letters or underscores. Later I also ran into an issue with periods in bucket names which showed up as this error message:ERROR: Error talking to S3: Server.AccessDenied(403): Access DeniedHere is an easy command to sanitize the bucket names:

sanitized_name=$( echo $name | tr [:upper:] [:lower:] | tr [:punct:] - )

It will lowercase all letters and convert all punctuation to dashes.

By default, Bash treats uninitialized variables the same way as Perl — they are blank strings. If you want them treated more like Python, you can issue the following command in your bash script:

set -u

You will then start seeing warning messages like the following:

./my_script.sh: line 419: FOO_BAR: unbound variable

Note that this mean you can’t check for the non-existence of environment variables with a simple [[ -z “$ENVIRONMENT_VARIABLE” ]]. Instead, you could do something like the following:

[[ $( set | grep "ENVIRONMENT_VARIABLE=" | wc -l ) -lt 1 ]]

 

mkdir -p is a command second only to touch in succinct utility.

touch creates a file if it does not exist, or updates its timestamp if it does. It’s handy if you want to write to a file without checking for its existence, as otherwise you’d need to determine whether or not append is the correct mode. It’s also handy for setting flags for yourself on the filesystem.

mkdir -p creates a path if it does not exist, or does nothing if the path already exists. mkdir -p /foo/bar/baz will create /foo, /foo/bar, and /foo/bar/baz for you. Conversely, mkdir -p /usr/local/bin will not complain because those directories already exist.

Why would you need this? A couple reasons that came up for me tonight:

• You cannot redirect output to a file if the file is in a directory that does not yet exist
• You cannot create a symbolic link in a directory that does not yet exist