My Notes

DevOps, Email, Kafka, Kubernetes, Productivity, Programming, Python, Mongo DB, macOS, Powershell, Spark, REST, RDBMS, SCM, Security, Unix Tools,

Notes on Security

Hashes

In the WSL

md5sum <filename>

You can check the validaity of the certificates using the https://www.geocerts.com/ssl-checker.

Winodows certificates on WSL

To display the Windows certificate, use the certmgr.msc. To enable Windows certificates in WSL, you need to export them from Windows and import them into the WSL certificate store. Here are several methods:

Install the following packages to count the current certificates:

sudo apt install p11-kit p11-kit-modules

Count the existing certificates:

trust list | grep "pkcs11*" | wc -l

Create the following directories:

sudo mkdir -p /usr/local/share/ca-certificates/windows-certs
mkdir -p /mnt/c/temp/wsl-certs

Clear any existing temporary certificates

rm -f /mnt/c/temp/wsl-certs/*.cer

Export Windows certificates using PowerShell with correct path handling

powershell.exe -Command "Get-ChildItem -Path Cert:\CurrentUser\Root | ForEach-Object { Export-Certificate -Cert \$_ -FilePath \"C:\\temp\\wsl-certs\\\$(\$_.Thumbprint).cer\" -Type CERT }"

Convert and import each certificate to WSL

#!/bin/bash
for cert in /mnt/c/temp/wsl-certs/*.cer; do
    if [ -f "$cert" ]; then
        dest="/usr/local/share/ca-certificates/windows-certs/$(basename ${cert%.*}).crt"
        sudo openssl x509 -inform DER -in "$cert" -out "$dest"
    fi
done

Update the CA certificate store

sudo update-ca-certificates

Now, when you run the command trust list | grep "pkcs11*" | wc -l, it should show you the difference value.

to lList only trusted CA certificates

trust list --filter=ca-anchors

List blacklisted certificates

trust list --filter=blacklist

Show details of specific certificate:

# First extract certificate to a file
trust extract --format=pem-bundle --filter=ca-anchors --purpose=server-auth /tmp/anchors.pem

# Then examine a specific certificate
openssl x509 -in /tmp/anchors.pem -text -noout

Extract certificates for specific purposes:

# Extract certificates trusted for server authentication
trust extract --format=pem-bundle --filter=ca-anchors --purpose=server-auth /tmp/server-auth.pem

# Extract certificates trusted for email
trust extract --format=pem-bundle --filter=ca-anchors --purpose=email /tmp/email.pem

Show Windows Certificates

List certificates in Trusted Root store

Get-ChildItem -Path Cert:\CurrentUser\Root

For more details on certificates

Get-ChildItem -Path Cert:\CurrentUser\Root | Format-List Subject, Issuer, Thumbprint, NotBefore, NotAfter

Filter by issuer example

Get-ChildItem -Path Cert:\CurrentUser\Root | Where-Object {$_.Issuer -like "*DigiCert*"}

List certificates in ca-certificates.crt bundle

awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

View a specific certificate

openssl x509 -in /etc/ssl/certs/certificate-name.pem -text -noout

List all the certificates

dpkg -L ca-certificates

Remove Certificates

Remove Winodows Certificates from WSL

# Remove the entire windows-certs directory
sudo rm -rf /usr/local/share/ca-certificates/windows-certs/

# Update the certificate store to apply changes
sudo update-ca-certificates --fresh

If you only want to remove specific certificates:

# List all imported Windows certificates
ls -la /usr/local/share/ca-certificates/windows-certs/

# Remove a specific certificate
sudo rm /usr/local/share/ca-certificates/windows-certs/CERTIFICATE_FILENAME.crt

# Update the certificate store
sudo update-ca-certificates

To remove using p11-Kit:

# List certificates to identify what to remove
trust list

# Remove a specific certificate by its path
sudo trust anchor --remove /usr/local/share/ca-certificates/windows-certs/CERTIFICATE_FILENAME.crt

CURL_CA_BUNDLE Environment Variable

The CURL_CA_BUNDLE environment variable is used by curl and libcurl to specify the path to a certificate bundle file containing trusted Certificate Authority (CA) certificates.

Purpose:

System trust stores are:

# Debian/Ubuntu
sudo cp /path/to/self-signed-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# RHEL/CentOS/Fedora
sudo cp /path/to/self-signed-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

Debugging Certificate Issues

# Show verbose output to diagnose certificate problems
curl -v https://example.com

# Even more detailed SSL information
curl --trace-ascii curl_trace.txt https://example.com

Information for the Jupyter in WSL.

Python Requests

A CA-bundle (Certificate Authority bundle) is a collection of trusted root certificates combined into a single file that applications use to verify the authenticity of SSL/TLS connections.

What It Is:

Key Purposes:

common locations:

# Debian/Ubuntu
/etc/ssl/certs/ca-certificates.crt

# RHEL/CentOS/Fedora
/etc/pki/tls/certs/ca-bundle.crt

# macOS
/etc/ssl/cert.pem

# Windows
C:\Windows\System32\curl-ca-bundle.crt (for curl)

Using SSL Certificates

Basic Certificate Verification

import requests

# Basic HTTPS request (uses system CA bundle)
response = requests.get('https://example.com')

# Specify a custom CA bundle
response = requests.get('https://example.com', verify='/path/to/ca-bundle.crt')

# Disable certificate verification (not recommended for production)
response = requests.get('https://example.com', verify=False)
# This will show a warning unless you disable it:
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

Using Client Certificates (mTLS)

# Client certificate and key in separate files
response = requests.get('https://example.com', 
                      cert=('/path/to/client.crt', '/path/to/client.key'))

# Client certificate with key in single file
response = requests.get('https://example.com', 
                      cert='/path/to/client.pem')

# With password-protected private key
response = requests.get('https://example.com',
                      cert=('/path/to/client.crt', ('/path/to/client.key', 'password')))

Using Session Objects (for multiple requests)

session = requests.Session()
session.verify = '/path/to/ca-bundle.crt'
session.cert = ('/path/to/client.crt', '/path/to/client.key')

response1 = session.get('https://example.com/endpoint1')
response2 = session.get('https://example.com/endpoint2')

Using environment variables

# Set these before running your Python script
import os
os.environ['REQUESTS_CA_BUNDLE'] = '/path/to/ca-bundle.crt'

# Now all requests will use this CA bundle by default
requests.get('https://example.com')

Advanced Certificate Handling

import ssl
import requests

# Create a custom SSL context
context = ssl.create_default_context(cafile='/path/to/ca-bundle.crt')
context.load_cert_chain('/path/to/client.crt', '/path/to/client.key')
context.verify_mode = ssl.CERT_REQUIRED

# Use with requests via an adapter (more advanced)
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager

class SSLAdapter(HTTPAdapter):
    def __init__(self, ssl_context=None, **kwargs):
        self.ssl_context = ssl_context
        super().__init__(**kwargs)
        
    def init_poolmanager(self, *args, **kwargs):
        kwargs['ssl_context'] = self.ssl_context
        return super().init_poolmanager(*args, **kwargs)

session = requests.Session()
adapter = SSLAdapter(ssl_context=context)
session.mount('https://', adapter)
response = session.get('https://example.com')

Error Handling

try:
    response = requests.get('https://example.com', verify='/path/to/ca-bundle.crt')
    response.raise_for_status()
except requests.exceptions.SSLError as e:
    print(f"SSL Error: {e}")
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")

Full certificate chain

Connect to a server and show the full certificate chain. The -showcerts option with s_client will display the entire chain as presented by the server. Look for multiple BEGIN CERTIFICATE blocks.

openssl s_client -connect example.com:443 -showcerts

Step 1: Extract and Save the Certificate Chain

# Save the full certificate chain to a file
openssl s_client -connect example.com:443 -showcerts </dev/null | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > certificate_chain.pem

Step 2: Using with curl

# Use the entire chain
curl --cacert certificate_chain.pem https://example.com

# Or set as environment variable
export CURL_CA_BUNDLE=certificate_chain.pem
curl https://example.com

To extract individual certificates if needed:

# Split the PEM file into individual certificates
csplit -f cert- certificate_chain.pem '/-----BEGIN CERTIFICATE-----/' '{*}'

Step 3: Using with Python Requests

import requests

# Basic usage with certificate chain
response = requests.get('https://example.com', verify='certificate_chain.pem')

# For a session (multiple requests)
session = requests.Session()
session.verify = 'certificate_chain.pem'
response = session.get('https://example.com')

# Using environment variable approach
import os
os.environ['REQUESTS_CA_BUNDLE'] = 'certificate_chain.pem'
response = requests.get('https://example.com')

To extract only the root or a specific certificate:

# Get the first certificate (usually the server certificate)
openssl s_client -connect example.com:443 </dev/null | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | head -n 25 > server_cert.pem

# Extract all individual certificates from the chain
openssl s_client -connect example.com:443 -showcerts </dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if(/BEGIN CERTIFICATE/){a++}; out="cert"a".pem"; print >out}'

For client certificate authentication:

# With curl
curl --cacert certificate_chain.pem --cert client.crt --key client.key https://example.com

# With Python
response = requests.get('https://example.com', 
                      verify='certificate_chain.pem',
                      cert=('client.crt', 'client.key'))

OpenSSL

Openssl supports X.509 certificates, SSL, TLS, and DTLS protocols, as well as other less popular cryptography-related technologies.

To get information of a handshake:

openssl s_client -connect example.org:443

Verification of the server certificate and its hostname by adding the -verify_return_error:

openssl s_client -connect example.org:443 -verify_return_error -verify_hostname example.org

In the Ubuntu Linux system, the system-installed OpenSSL looks for the trusted certificates in the /etc/ssl/certs directory

You can provide trusted CA certificates to OpenSSL using any of the following methods:

You can download Mozilla’s trusted CA certificate Mozilla’s trusted CA certificate:

curl --remote-name https://curl.se/ca/cacert.pem

Provide the downloaded certificate bundle to Openssl using any of the following methods:

OCSP on the command line

List the certificates:

echo | openssl s_client -connect revoked.badssl.com:443 -showcerts

Save the first and second certificates to cert1.pem and cert2.pem

Run and get the OCSP uri

openssl x509 -in cert1.pem -noout -ocsp_uri

output is http://e5.o.lencr.org.

Check the certificate validity:

openssl ocsp -issuer cert2.pem -cert cert1.pem -url http://e5.o.lencr.org

img000295@2x

Issuer Certificate

Find issuer certificate:

openssl x509 -in cert1.pem -text

and find the signing URL

img000296@2x

Now download the signing certificate:

curl http://e5.i.lencr.org/ >signer.der

convert the signing certificate to pem

openssl x509 -inform der -in signer.der -out signer.pem

Display the issuer certificate:

openssl x509 -in signer.pem -text