Data Engineering DevOps Email Kafka Kubernetes macOS Mongo DB Productivity Programming Python Powershell REST RDBMS SCM Security Spark Unix Tools

Shell

change the shell to zsh

chsh -s /bin/zsh

Edit a file in a TextEdit

open -a TextEdit ~/.zshrc

Find files

Find the file names only with the text recusively

grep -il -r --include "*.py" 'text' .

Find all the files with wild card recursively:

find . -name "*.pyc" -exec ls {} \;

Another recursive example with include and exclude directory:

grep -rI --include='*.py'  --exclude-dir='lib' "S3Service" *

To show all the CHAPTERS:

grep -i chapter novel.txt

Cut only the chapter headings

grep -i chapter novel.txt | cut -d' ' -f3-

Read the line where ‘CHAPTER’ appears in the novel.txt, but start with ‘the’.

grep -E '^CHAPTER (.*)\. The' novel.txt

Every word with 5 or more characters

grep -oE '\w{5,}' novel.txt

Search, that combine find and grep:

find . -name "Dockerfile" -exec grep -l "glue" {} \;

In RedHat, to start the firewall GUI, type the following command:

sudo system-config-firewall

Here is the way to validate an XML file for a schema and generate the report

	xmllint  --noout --valid --schema UserCentralClients.xsd sample2.xml > error.txt 2>&1

noout will stop xml written to the error.txt.

Tools

Here some important tools to use in the shell.

awk

Commandline

awk -F, '{$2="***"; print $1 $2}'  test.txt

Above logic in the file called f:

awk -F, -f t test.txt

Both the above commands produce the same.

here the contents of the f file:

BEGIN {
  print "1 st \t 2nd"
  print "==== \t ==="
}

{
  $2="***";
  print $1 "\t" $2
}

END{
  print "the end"
}

Logging

For the log stream

log stream

To filter

log stream --predicate 'eventMessage contains "<searh word>"'

Filter by the specific process:

log stream  --process 153
# or
log stream --predicate '(process == <PID>)'

Convert between file formats

To convert files:

textutil -convert doc textfile.rtf -output msword.doc

Scriptable Image Processing System (sips)

To make the jpeg to 200 pixels wide

sips --resampleWidth 200 image_file.jpeg

IP

To get current public IPv4 address:

curl -4 https://icanhazip.com/; echo

Flush DNS Cache

To reset the DNS cache

sudo killall -HUP mDNSResponder

You have to provide root password.

Empty Trash forcefully

Sometime it is not possible to empty the trash. Run the following command:

sudo rm -ri ~/.Trash/*

Disk utitlity

diskutil list

nslookup

In addition to showing you a host’s domain name or IP address, nslookup gives you the IP address of the DNS server i if you’re trying to diagnose a DNS problem:

nslookup ojitha.blogspot.com
nslookup ojitha.github.io

Domain information

Find out what person or organization owns a domain name

whois ojitha.blogspot.com

lsof

Which apps have open TCP network connections:

lsof -i

###

Terminal Commands

Find the hidden files

defaults write com.apple.finder AppleShowAllFiles true; killall Finder

Change the screenshot format (bmp,gif,pdf,png,tiff or jpeg)

defaults write com.apple.screencapture type -string "jpeg"; killall SystemUIServer

Software updates CLI:

sudo softwareupdate -i -a

Here, -i and -a flags to go ahead and install every available update.

To list the last reboots:

last reboot

Find the uptime:

uptime

History of user Loggins:

last

To add the new user

sysadminctl -addUser <username> -fullName "<Full Name>" -password <password>

To change the password of the user:

sysadminctl -resetPasswordFor <username> -newPassword <new password> -passwordHint "<password hint>"

Find the type of the command

type -a pwd

create tar file

tar -czvf mylogs.tar.gz logs-*.log

find the directory in the Unix

find / -type d -name 'pyve*' 2>&1 | grep -v "Permission denied"
#or
find . -type d -name 'pyve*'  2>/dev/null

filter the file lines:

sed -n '1 , 3 p' my.txt > test.txt

Top occurrence words in a page

curl -s http://www.gutenberg.org/files/4300/4300-0.txt | tr '[:upper:]' '[:lower:]' | grep -oE '\w+' | sort | uniq -c | sort -nr | head -n 10

Use of Python as command line tool

#!/usr/bin/env python
from sys import stdin, stdout
while True:
    line = stdin.readline()
    if not line:
        break
    stdout.write("%d\n" % int(line))
    stdout.flush()

Above will display the lines generated from the sequence.

#permission
chmod u+x stream.py

#create the pipeline
seq 1000 | ./stream.py

Substitute seq

seq -f "Line %g" 10

Here the equlents, lines is a file

< lines head -n 3
< lines sed -n 1,3p
< lines awk 'NR<=3'

Shortcuts

If you want to select all and only the output from the most recent command press ⌘-Shift-A.

image-20220806200426196

Disk Management

To display size of the folder

du -hsx *

this will display the usage of the directory. The following code show the top 20 files which used most of the file space

du -a | sort -n -r | head -n 20

This will show directory wise usage of file space

du -hsx * | sort -n -r | head -n 20

Here

  • du command -h option : display sizes in human readable format.
  • du command -s option : show only a total for each argument (summary).
  • du command -x option : skip directories on different file systems.

Enable Clipboard reddit

1.Run Terminal

2.Type the following to check to see if ClipboardSharingEnabled is equal to zero. (if so, proceed to #3)

defaults read ~/Library/Preferences com.apple.coreservices.useractivityd.plist

3.Type the following to delete the ClipboardSharingEnabled = 0 setting.

defaults delete ~/Library/Preferences/com.apple.coreservices.useractivityd.plist ClipboardSharingEnabled

4.Reboot your mac. Copy paste across devices should now work.

Alternatively, type the following to enable ClipboardSharingEnabled as the default:

defaults write ~/Library/Preferences/com.apple.coreservices.useractivityd.plist ClipboardSharingEnabled 1

File sharing to Ubuntu

To share only one specific folder from your MacBook instead of the entire user directory, you need to set up a specific folder share on macOS first, then mount just that folder on Ubuntu.

Step 1: Share the specific folder on your MacBook

  1. Open System Preferences/Settings on your MacBook
  2. Go to Sharing
  3. Select File Sharing from the list
  4. Click the + button under “Shared Folders”
  5. Navigate to and select your specific folder (e.g., ~/.../myfolder)
  6. Set permissions as needed (Read & Write for your user)
  7. Note the share name that appears (it might be something like myfolder)

Step 2: Mount only that specific folder on Ubuntu

  1. Create a mount point for the specific folder: 
      sudo mkdir /mnt/myfolder
  2. Mount the specific shared folder: 
      sudo mount -t cifs //...-macbook-pro.local/myfolder /mnt/myfolder -o username=your_macbook_username,uid=1000,gid=1000,iocharset=utf8
  3. Update your docker-compose.yml (optional):
  volumes:
    - /mnt/myfolder:/app:rw

To make it persistent:

You need the optional step I mentioned at the end - adding an entry to /etc/fstab. Here’s how:

Make the mount persistent across reboots:

  1. First, unmount if currently mounted: 
      sudo umount /mnt/myfolder
  2. Edit the fstab file: 
      sudo nano /etc/fstab
  3. Add this line to the end of the file: 
      //...-macbook-pro.local/myfolder /mnt/myfolder cifs username=your_username,password=your_password,uid=1000,gid=1000,iocharset=utf8,noauto,user 0 0
  4. For security, create a credentials file instead of putting password in fstab: 
      sudo nano /etc/cifs-credentials

    Add:

      username=your_macbook_username
  password=your_macbook_password
  5. Secure the credentials file: 
      sudo chmod 600 /etc/cifs-credentials
  6. Update fstab to use credentials file: 
      //...-macbook-pro.local/myfolder /mnt/myfolder cifs credentials=/etc/cifs-credentials,uid=1000,gid=1000,iocharset=utf8,noauto,user 0 0
  7. Test the mount: 
      mount /mnt/myfolder

Now the mount will be available after reboots, but won’t auto-mount (due to noauto option). You can manually mount it when needed, or remove noauto if you want it to mount automatically at boot.