Investigating UMB problems

How to investigate problems with interfacing to the Red Hat Universal Message Bus

Problem

Some piece of software has problems connecting to the Universal Message Bus (UMB) or with authentication.

Steps

  1. Try to manually connect to the servers. The server URLs can be found in secrets.yml in cee/deployment-all, or via oc describe dc/amqp-bridge-umb. For example, look for errors in the output of

    openssl s_client \
        -connect messaging-devops-broker01.web.prod.ext.phx2.redhat.com:5671 | \
        grep error
    openssl s_client \
        -connect messaging-devops-broker02.web.prod.ext.phx2.redhat.com:5671 | \
        grep error
    

    For “Connection refused”, check that the URLs are correct. The expected response is a “bad certificate error SSL alert 42”, which basically means that the client certificate is missing.

    140197084915520:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../ssl/record/rec_layer_s3.c:1543:SSL alert number 42
    
  2. Provide the appropriate client certificate. In a checkout of cee/deployment-all, run

    echo "$UMB_MESSENGER_CKI_BOT_PROD_PEM" | \
        base64 --decode | \
        openssl s_client \
        -connect messaging-devops-broker02.web.prod.ext.phx2.redhat.com:5671 \
        -cert - | \
        grep error
    

    The server might return another error message if it does not accept the certificate:

    140405938968384:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../ssl/record/rec_layer_s3.c:1543:SSL alert number 42
    
  3. Check the certificate, especially the validity interval:

    echo "$UMB_MESSENGER_CKI_BOT_PROD_PEM" | \
        base64 --decode | \
        openssl x509 -text
    
Last modified June 3, 2021: Attempt to make the 'branch' config docs more clear (5a0d271)