My Notes

Data Engineering DevOps Email Kafka Kubernetes macOS Mongo DB Productivity Programming Python Powershell REST RDBMS SCM Security Spark Unix Tools

Notes on Windows PowerShell

Syntax
Parameters
Calculated parameters
Grep
Tail
WSL 2
AWS Linux 2 Docker
- Develop Glue on Amazon Linux 2 on Windows
- Export Site Certificate

Syntax

Here the syntax for the commandlet.

Verb-noun [-parameter list]

Above commandlet is not sensitivities. For example similar to UNIX pwd is:

Get-Location

The alias for the above command is pwd. All the aliases can be found as follows:

Get-Alias

If you want to get all the aliases available for the commandlet, for example Get-Process:

Get-Alias | Where-Object { $_.definition -eq "get-process"}

You can create new alias with Set-Alias or New-Alias.

Set-Alias xxx Get-xxxx
New-Alias yyy Get-yyyy

NOTE: Set-Alias overwrite the existing alias. New-Alias will fail if the alias is already exists. You can use the -description` to create the note.

You can use `Set-Alias to create alias for the application as well.

 Set-Alias np C:\LocalDev\npp.7.8.bin\notepad++.exe

Now the alias np you can use to open any file in the CLI.

Parameters

Parameters can be positional or with the name such as Filter:

 Get-ChildItem 'C:\Local\xxxx\forms' -Filter *.pdf

This will shows only the PDF files.

Calculated parameters

you can calculate the process id 1100 as follows

 Get-Process -id (1000 + 100)

The short form for the above commandlet is ps.

Another example to exclude services start withv to w

Get-Service -exclude "[v-w]*"

Grep

I need to find all the lines of all the files where the particular string is exists using Select-String:

Select-String -Path .\<file name>.log.* -Pattern "-511055-141"

Here the way to extract all the strings which are matched with the regex TEST0002H-.*-[0-9].? :

Select-String -Path .\<file-name>.log.* -Pattern "TEST0002H-.*-[0-9].?" -AllMatches |  % { $_.Matches } | % {$_.Value } | select -Unique

if you want only the file name:

Select-String -Path .\*.log -Pattern '<regex>'   |  foreach {$_.Filename} | select -Unique

when executed, you will get all the strings matching wit the regex in the <file-name>.log.* files.

Tail

Here the Unix tail equivalent:

 Get-Content .\<log file name>.log -Tail 4 -Wait

## Kill Process

  netstat -ano | findstr :9007
  taskkill /PID 4448 /F

WSL 2

You can install WSL 2, but first step you have to elevate Powershell as follows after open with “Run as DefendPoint Admin”:

start-process powershell –verb runAs

As explained in the https://serverfault.com/questions/11879/gaining-administrator-privileges-in-powershell.

AWS installation : Developing on Amazon Linux 2 using Windows

Uninstall

first find the Linux version you are using:

wslconfig /l

Then uninstall:

wslconfig /u Amazon2

Install Amazon Linux 2

Set the default to WSL 2 First update to the WSL2 Linux kernel update package for x64 machines. then set to the version 2:

 wsl --set-default-version 2

Download Amazon Linux 2 and extract to the c:\wsl folder.

C:\WSL\Amazon2\Amazon2.exe

Now run the following command:

wsl -s Amazon2

now list the available WSL distro

wsl -l -v

you should get

  NAME       STATE           VERSION
* Amazon2    Stopped         2

to start distro

wsl

Install Ubuntu

Follow the [Install WSL on Windows 10 Microsoft Docs](https://docs.microsoft.com/en-us/windows/wsl/install-win10)
[Create user account for Linux distribution Microsoft Docs](https://docs.microsoft.com/en-us/windows/wsl/user-support)

AWS Linux 2 Docker

If you want install docker container Install AWS Linux docker container

docker run -itd -v $env:userprofile\.aws:/root/.aws:rw --name al amazonlinux bash

Getting to the AWS Linux docker container:

docker exec -it al bash

Install AWS CLI 2

# download
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

# install unzip
yum install unzip

# unzip to out folder
unzip awscliv2.zip -d out

# install
cd out
aws/install

NOTE: This is the best container to execute Dynamodb queries

Develop Glue on Amazon Linux 2 on Windows

Read # Developing and Testing ETL Scripts Locally Using the AWS Glue ETL Library: here is my command summary.

NOTE: you must unchecked the Docker -> Settings->Resources -> WSL INTEGRATION -> Enable integation with my default WSL distro. This caused to read only mapping of windows .aws folder in the docker instance.

First install the docker as explain here.
go to the bash prompt of the docker container and execute the aws command to cache the MFA code:

docker exec -it glue_jupyter bash

In the docker container bash prompt:

aws s3 ls

this command will prompt to enter the MFA code. As a result you should be able to see the list of available s3 buckets.

run the Jupyter notebooks (pyspark):

docker run -itd -p 8888:8888 -p 4040:4040 -v $env:userprofile\.aws:/root/.aws:rw --name glue_jupyter amazon/aws-glue-libs:glue_libs_1.0.0_image_01 /home/jupyter/jupyter_start.sh

To run Zeppelin:

 docker run -itd -p 8080:8080 -p 4040:4040 -v $env:userprofile\.aws:/root/.aws:rw --name glue_zeppelin amazon/aws-glue-libs:glue_libs_1.0.0_image_01 /home/zeppelin/bin/zeppelin.sh

and open the notebook: http://localhost:8888

This is sample testing code to run in the Jupyter notes¹

import os

import boto3
import botocore.session
from botocore import credentials

# By default the cache path is ~/.aws/boto/cache
cli_cache = os.path.join(os.path.expanduser('~'), '.aws/cli/cache')

# Construct botocore session with cache
session = botocore.session.get_session()
session.get_component('credential_provider').get_provider('assume-role').cache = credentials.JSONFileCache(cli_cache)

s3 = boto3.Session(botocore_session=session).client('s3')
response = s3.list_buckets()
print(response)

In the above, I am using default AWS profile and the MFA authentication:

[default]
region = ap-southeast-2
output = json

role_arn = arn:aws:iam::<account>:role/<role-name>
source_profile = default

mfa_serial = arn:aws:iam::<account>:mfa/ojitha.kumanayaka@....com
role_session_name = ojitha.kumanayaka@.....com
s3 = 
    signature_version = s3v4

Export Site Certificate

function Export-SiteCertificate {
    param(
        [Parameter(Mandatory=$true)][string]$ComputerName,
        [string]$OutputPath = ".\certificate.cer"
    )
    
    try {
        $webRequest = [Net.HttpWebRequest]::Create("https://$ComputerName")
        $webRequest.GetResponse().Dispose()
        
        $cert = New-Object Security.Cryptography.X509Certificates.X509Certificate2($webRequest.ServicePoint.Certificate)
        $bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
        [IO.File]::WriteAllBytes($OutputPath, $bytes)
        
        Write-Host "Certificate exported to $OutputPath" -ForegroundColor Green
    }
    catch {
        Write-Host "Failed to export certificate: $_" -ForegroundColor Red
    }
}

# Usage
Export-SiteCertificate -ComputerName "example.com" -OutputPath ".\example_cert.cer"

copy windows to WSL

# Export a specific certificate by thumbprint
Export-Certificate -Cert (Get-Item Cert:\LocalMachine\Root\[THUMBPRINT]) -FilePath C:\temp\certificate.cer

# Or export all root certificates
Get-ChildItem -Path Cert:\LocalMachine\Root | ForEach-Object { Export-Certificate -Cert $_ -FilePath "C:\temp\$($_.Thumbprint).cer" }

import to WSL

# In WSL, install the ca-certificates package if not already installed
sudo apt update && sudo apt install -y ca-certificates

# Copy the exported certificate
sudo cp /mnt/c/temp/certificate.cer /usr/local/share/ca-certificates/windows-cert.crt

# Update the certificate store
sudo update-ca-certificates

boto3 not caching STS MFA sessions ↩