Kofler Michael / Кофлер Майкл - Scripting: Automation with Bash, PowerShell, and Python / Написание сценариев: автоматизация с помощью Bash, PowerShell и Python [2024, PDF/EPUB, ENG]

Part I Scripting Languages
1 Scripting: Doing One Thing
1.1 What Does Scripting Mean?
1.1.1 Scripting versus Programming
1.1.2 Glue Languages and Glue Code
1.1.3 Do One Thing and Do It Well
1.2 Scripting Languages
1.2.1 Bash and Zsh
1.2.2 PowerShell
1.2.3 Python
1.2.4 Many Similarities, Even More Differences
1.3 The Agony of Choice
2 Ten Times Ten Lines
2.1 Markdown Spell Checker (Bash)
2.2 Sorting Images by Date (PowerShell)
2.3 Converting a JSON File to XML Format (Python)
2.4 Daily Server Backups (Bash)
2.5 Web Scraping (Python)
2.6 Logging Weather Data (Python)
2.7 Microsoft Hyper-V Cleanup (PowerShell)
2.8 Statistical Analysis of a Logging File (Bash)
2.9 Uploading Files to the Cloud (PowerShell)
2.10 Cloning Virtual Machines (Bash)
3 Bash and Zsh
3.1 Terminal, Shell, and Bash
3.1.1 Determining the Current Shell
3.2 Installation
3.2.1 Using Bash in a Virtual Machine
3.2.2 Windows Subsystem for Linux
3.2.3 Git Bash
3.2.4 Docker
3.2.5 Bash Configuration (/etc/profile and .bashrc)
3.3 Running Commands Interactively
3.3.1 Operating the Terminal
3.3.2 Online Help
3.3.3 Command Reference
3.4 Zsh as an Alternative to Bash
3.4.1 Installation
3.4.2 Zsh Configuration (/etc/zshrc and .zshrc)
3.4.3 Oh My Zsh
3.5 The First Bash Script
3.5.1 Hash Bang (Shebang)
3.5.2 Shebang Variants
3.5.3 Making the Script Executable (chmod +x)
3.5.4 Hello, World!
3.5.5 Elementary Syntax Rules
3.5.6 A “Real” Example: Backup Script
3.5.7 Running Scripts without ./
3.6 Running Commands
3.6.1 Serial Execution
3.6.2 Conditional Command Execution
3.6.3 Running Commands in the Background
3.6.4 Running Commands in a Subshell
3.7 Standard Input and Standard Output
3.7.1 Redirecting Input and Output
3.7.2 The Pipe Operator
3.7.3 Redirecting and Displaying Outputs at the Same Time
3.8 Globbing, Brace Extension, and Handling File and Directory Names
3.8.1 Recursive Globbing
3.8.2 Access to Important Directories
3.8.3 Brace Extension
3.8.4 Filenames and Directory Names with Spaces
3.9 Variables
3.9.1 Initializing and Deleting Variables
3.9.2 Declaring Variables
3.9.3 Performing Calculations in Bash
3.9.4 Arrays
3.9.5 Predefined Variables
3.9.6 Environment Variables
3.10 Strings
3.10.1 Single versus Double Quotation Marks
3.10.2 Outputting Strings (echo)
3.10.3 Colors
3.10.4 Entering or Reading Character Strings (read)
3.10.5 Substitution and Expansion Mechanisms
3.10.6 Parameter Substitution
3.10.7 Heredocs and Herestrings
3.10.8 The Backslash
3.11 Branches
3.11.1 if Short Notation with && or ||
3.11.2 Conditions
3.11.3 Branches with case
3.11.4 Parameter Analysis via case
3.12 Loops
3.12.1 Processing Filenames with Spaces
3.12.2 while and until
3.12.3 Loops over Text Files
3.13 Functions
3.13.1 Local Variables
3.14 Error Protection
3.14.1 Detecting Errors
3.14.2 Canceling in Case of Errors
3.14.3 exit
3.14.4 Responding to Signals (trap)
3.14.5 Timeout
4 PowerShell
4.1 Installation
4.1.1 Installation on Linux
4.1.2 Installation on macOS
4.1.3 Limitations of PowerShell on Windows versus Linux/macOS
4.1.4 Configuration
4.2 Windows Terminal
4.2.1 Configuration
4.2.2 Operation
4.3 Calling cmdlets and Functions
4.3.1 cmdlets
4.3.2 The Verb-Noun Terminology
4.3.3 Aliases
4.3.4 Parameters and Options
4.3.5 Splatting
4.3.6 Functions
4.3.7 Running Conventional Commands
4.3.8 Online Help
4.4 Combining Commands
4.4.1 Chains of Commands
4.4.2 Multiline Statements
4.4.3 Dealing with Special Characters
4.5 The First Script
4.5.1 Hello, World!
4.5.2 Trouble with the Execution Policy
4.5.3 Setting Up a Custom Script Directory
4.5.4 Running Scripts on Linux and macOS
4.5.5 Example: Cleaning Up the Downloads Directory
4.5.6 Differentiating Between PowerShell Versions
4.6 Variables, Strings, and Objects
4.6.1 Data Types
4.6.2 Calculating and Comparing
4.6.3 Strings
4.6.4 Command Substitution
4.6.5 Objects
4.6.6 Assembling Custom Objects
4.6.7 Predefined Variables
4.6.8 Environment Variables
4.7 Arrays and Hash Tables
4.7.1 Creating and Processing Custom Arrays
4.7.2 Hash Tables
4.8 Output Redirection
4.8.1 Streams
4.8.2 Output Redirection
4.8.3 Duplicating Output (Tee-Object)
4.8.4 (Not) Formatting the Output
4.8.5 No Input Redirection
4.9 Loops
4.9.1 ForEach-Object
4.9.2 for, while, do-while, and do-until
4.10 Branches
4.10.1 Case Distinctions Using switch
4.11 Functions and Parameters
4.11.1 Function Results
4.11.2 return
4.11.3 Parameters
4.11.4 Calling Functions
4.11.5 Processing Standard Input
4.11.6 Syntax Variants
4.12 Modules
4.12.1 Module Directories
4.12.2 Modules for Experts
4.13 Error Protection
4.13.1 Set-StrictMode
4.13.2 Aborting Scripts on Errors
4.13.3 Separating Error Messages from Other Output
4.13.4 Error Protection Using try/catch
4.13.5 Terminating the Script Execution Prematurely (Exit)
5 Python
5.1 Installing Python
5.1.1 Windows
5.1.2 macOS
5.2 Getting to Know Python in a Terminal Window
5.3 Programming Custom Scripts
5.3.1 Shebang
5.4 Elementary Syntax Rules
5.4.1 Block Elements
5.4.2 Comments
5.5 Numbers
5.5.1 Floats
5.5.2 Random Numbers
5.6 Strings
5.6.1 Raw Strings
5.6.2 Processing Strings
5.6.3 Slicing
5.6.4 print and input
5.6.5 Formatting Outputs
5.7 Lists
5.7.1 List Comprehension
5.7.2 map and filter
5.8 Tuples, Sets, and Dictionaries
5.8.1 Sets
5.8.2 Dictionaries
5.9 Variables
5.9.1 Assignments
5.9.2 Data Types
5.9.3 Mutable or Immutable?
5.10 Operators
5.10.1 Combining Assignment and Evaluation
5.11 Branches (if)
5.11.1 Conditions
5.11.2 Short Notation with if
5.12 Loops (for and while)
5.12.1 break, continue, and else
5.12.2 Loops over Number Ranges (range)
5.12.3 Loops over the Characters of a String
5.12.4 Loops over Lists, Tuples, and Sets
5.12.5 Loops over Dictionaries
5.12.6 Loops over the Parameters of a Script
5.12.7 Globbing on Windows
5.12.8 Loops over the Lines of a Text File
5.12.9 Loops over All Files in a Directory
5.13 Functions
5.13.1 Global and Local Variables
5.13.2 Parameters
5.13.3 Optional Parameters
5.13.4 Variable Number of Parameters
5.13.5 Lambda Functions
5.14 Processing Text Files
5.14.1 Example: Analyzing a CSV File
5.14.2 Example: Creating SQL Commands
5.15 Error Protection
5.15.1 try/except
5.16 System Functions
5.16.1 Access to the Program Parameters
5.16.2 Access to Standard Input and Standard Output
5.16.3 Exiting a Program
5.16.4 Calling Linux Commands
5.16.5 Processing Results
5.16.6 Running Commands through the Shell
5.16.7 Error during a Command Call
5.16.8 Waiting (sleep)
5.17 Modules
5.17.1 import
5.17.2 Distributing Custom Scripts across Multiple Files
5.18 Installing Additional Modules Using pip
5.18.1 Installing pip
5.18.2 Applying pip
5.18.3 pip Problems with Windows
5.18.4 pip Problems with Linux
5.18.5 requirements.txt
5.18.6 pipenv
Part II Work Techniques and Tools
6 Linux Toolbox
6.1 Directories and Files
6.1.1 Listing, Copying, Moving, and Deleting Files
6.2 Finding Files
6.2.1 Text Search Using grep
6.3 Compressing and Archiving Files
6.3.1 Archiving Files Using tar
6.3.2 ZIP Files
6.4 Using Root Privileges
6.4.1 sudo Privileges for Selected Accounts Only
6.4.2 Issues with Output Redirection
6.4.3 Access Rights
6.4.4 Managing Processes
6.4.5 Terminating Processes
6.4.6 Background Processes and System Services
6.4.7 Logging Files
6.4.8 Determining Free Memory
6.4.9 Determining Other System Information
6.5 Software Installation
6.5.1 Updating Software
6.5.2 Installing Additional Packages
6.6 Other Commands
7 cmdlets for PowerShell
7.1 Directories and Files
7.1.1 Listing, Copying, Moving, and Deleting Files
7.1.2 Reading and Writing Text
7.1.3 Inputting and Outputting Text
7.2 Finding Files
7.2.1 Searching Text Files
7.2.2 Text versus Objects
7.3 Compressing and Archiving Files
7.3.1 Compressing Search Results
7.4 Process Management
7.4.1 Launching Other PowerShell Scripts
7.4.2 Running Commands in the Background
7.4.3 Managing Services
7.4.4 Working on Other Computers
7.5 Registration Database and System Information
7.5.1 Determining System Information
7.5.2 Evaluating the Logging System
7.6 Processing cmdlet Results
7.6.1 Select-Object and Sort-Object
7.6.2 Where-Object
7.6.3 Group-Object
7.6.4 ForEach-Item
7.6.5 Measure-Object
7.6.6 Formatting and Exporting cmdlet Results
7.7 Other cmdlets
7.8 Installing Additional Modules
7.8.1 Listing Modules, Commands, and Package Sources
7.8.2 NuGet and winget
7.9 Standard Aliases
8 Analyzing Texts with Filters and Pipes
8.1 grep, sort, cut, and uniq
8.1.1 grep
8.1.2 wc
8.1.3 cut
8.1.4 sort
8.1.5 head and tail
8.1.6 uniq
8.1.7 tr
8.1.8 awk and sed
8.2 Example: Statistical Data Analysis
8.2.1 Data Extraction via Script
8.3 Example: ping Analysis
8.3.1 ping Call via Script
8.4 Example: Apache Log Analysis
8.4.1 Extracting IP Addresses
8.4.2 Identifying Popular Pages
8.5 CSV Files
8.5.1 Processing CSV Files Using Python
8.5.2 Processing CSV Files in PowerShell
9 Regular Expressions
9.1 Syntax Rules for Regular Expressions
9.1.1 Characters
9.2 Groups and Alternatives
9.2.1 Quantifiers
9.2.2 On Greed (Greedy versus Lazy)
9.2.3 Alpha and Omega
9.2.4 Exercise Pages and Cheat Sheets
9.2.5 Example: Recognizing a Date
9.2.6 Example: Recognizing IPv4 Addresses
9.2.7 Example: Recognizing Email Addresses
9.3 Regular Expressions in Bash (grep, sed)
9.3.1 Bash Comparison Operator for Regular Expressions
9.3.2 Filtering Text Using grep
9.3.3 The "sed" Stream Editor
9.3.4 Searching and Replacing Using sed
9.3.5 sed Example: Manipulating Paths of Image Files
9.4 Regular Expressions in PowerShell
9.4.1 The match and replace Operators
9.4.2 Select-String
9.4.3 Example: Input Validation
9.4.4 Example: List of Figures
9.5 Regular Expressions in Python (re Module)
9.5.1 Example: Verifying a MAC Address
9.5.2 Example: Anonymizing a Logging File
10 JSON, XML, and INI
10.1 JSON in PowerShell
10.1.1 Example: Saving Event Log Entries in JSON Format
10.1.2 Example: Analyzing Domain Queries
10.1.3 Example: Converting between CSV and JSON
10.2 JSON and Python
10.2.1 Example: Collecting Birthdays
10.2.2 Example: Determining Holidays
10.3 JSON in Bash
10.3.1 Viewing JSON Files Interactively Using fx
10.4 XML in PowerShell
10.4.1 XML Data Type
10.4.2 Select-Xml
10.4.3 ConvertTo-Xml, Export-Clixml, and Import-Clixml
10.5 XML and Python
10.5.1 Example: Creating a Dictionary for Country Codes
10.5.2 Example: Analyzing an RSS Feed
10.5.3 Example: Extracting MAC Addresses from Virtual Machine Files
10.6 XML in Bash
10.6.1 xmllint
10.6.2 XMLStarlet
10.7 INI Files
10.7.1 Python
10.7.2 PowerShell
10.7.3 Bash
11 Running Scripts Automatically
11.1 cron
11.1.1 /etc/crontab
11.1.2 Personal crontab Files
11.1.3 Hourly, Daily, Weekly, and Monthly Directories
11.1.4 Alternatives to cron
11.1.5 Starting Jobs Automatically on macOS
11.2 Example: Web Server Monitoring
11.2.1 Testing and Troubleshooting
11.2.2 Real Monitoring
11.3 Microsoft Windows Task Scheduler
11.3.1 Troubleshooting
11.3.2 Setting Up Tasks via cmdlets
11.4 Example: Saving Exchange Rates
11.5 Tracking File System Changes
11.5.1 inotify
11.5.2 Alternatives to inotify
12 SSH
12.1 Installing the SSH Client and Server
12.1.1 Linux
12.1.2 macOS
12.1.3 Windows
12.1.4 Using an Editor in an SSH Session
12.1.5 Securing an SSH Server
12.2 Working with SSH
12.2.1 Host Verification
12.2.2 Potential Issues and Their Cause
12.2.3 Running Linux and macOS Commands
12.2.4 Running Windows Commands
12.2.5 SSH Remoting in PowerShell
12.3 scp and rsync
12.3.1 Copy-Item with SSH Remoting
12.3.2 rsync
12.4 SSH Authentication with Keys
12.4.1 Generating a Key Pair
12.4.2 Storing the Public Component of the Key on the Server (macOS and Linux)
12.4.3 Storing the Public Component of the Key on the Server (Windows)
12.5 Example: Uploading Images to a Linux Web Server
12.5.1 Preparatory Measures
12.5.2 Bash Script
12.5.3 PowerShell Script
12.6 Example: Analyzing Virtual Machines
13 Visual Studio Code
13.1 Introduction
13.1.1 Comparing VS Code, VSCodium, and Visual Studio
13.1.2 Think in Terms of Directories, Not Files!
13.2 Language-Specific VS Code Extensions
13.2.1 PowerShell Extension
13.2.2 Python Extension
13.2.3 Bash/Shell Extensions
13.3 Remote – SSH Extension
13.3.1 Applying the Remote – SSH Extension
13.3.2 Limitations
14 Git
14.1 Git Crash Course
14.1.1 Preparation Tasks
14.1.2 The First Repository
14.1.3 The First Commit
14.1.4 Additional Commits
14.1.5 Setting Up the Repository on a Second Computer
14.1.6 Git Status
14.1.7 Excluding Files from Git (.gitignore)
14.1.8 Transferring Existing Code to a New Repository
14.2 Handling Settings and Passwords Correctly
14.2.1 What’s a Better Method?
14.2.2 Example
14.3 Git Automation
14.3.1 Controlling GitHub Remotely Using gh
14.4 Git Hooks
14.4.1 Example: Detecting Unversioned Files Before the Commit
Part III Applications and Examples
15 Backups
15.1 Synchronizing Directories to External Storage
15.1.1 PowerShell Script with robocopy
15.1.2 Ideas for Improvement
15.1.3 Bash Script with rsync
15.2 WordPress Backup
15.3 SQL Server Backup
15.3.1 Backing Up and Compressing all Databases via Script
16 Image Processing
16.1 Manipulating Image Files
16.1.1 Installing ImageMagick
16.1.2 Trying Out ImageMagick
16.1.3 Example: convert2eps (Bash Variant)
16.1.4 Example: convert2eps (PowerShell Variant)
16.2 Sorting Photos by Date Taken
16.2.1 Installing and Trying Out ExifTool
16.2.2 Bash Example
16.2.3 PowerShell Example
16.3 Converting Exif Metadata to SQL Commands
16.3.1 PyExifTool
16.3.2 EXIF2SQL
16.3.3 Options for Enhancement
17 Web Scraping
17.1 Limitations
17.2 Web Scraping, Web Crawling, and Data Mining
17.3 Downloading Websites Using wget
17.3.1 Example 1: Downloading Directly Linked Images
17.3.2 Example 2: Downloading all PDF Files Linked on the Website
17.3.3 Example 3: Creating a Local Copy of the Website
17.4 Web Scraping with Regular Expressions
17.5 Web Scraping with Beautiful Soup
17.5.1 Beautiful Soup and Requests
17.5.2 Hello, Beautiful Soup!
17.5.3 Example: Determining a List of Top Titles from Rheinwerk Publishing
17.6 Web Scraping with Requests-HTML
17.6.1 Example: Analyzing the GitHub Status
17.7 Web Scraping with PowerShell
17.7.1 PowerHTML Module
18 Using REST APIs
18.1 Tools
18.2 Sample APIs to Try Out
18.3 Implementing Custom REST APIs
18.4 curl and wget
18.4.1 curl
18.4.2 wget
18.4.3 Using REST APIs in Bash Scripts
18.5 Using REST APIs in PowerShell
18.5.1 Options
18.6 Example: Determining the Current Weather
18.7 Using REST APIs in Python
18.8 Example: Determining Electricity Prices and Displaying Them Graphically
18.8.1 aWATTar API
18.8.2 Analysis of the Data
18.8.3 Matplotlib
18.8.4 Controlling the Energy Consumption
19 Databases
19.1 Updating and Maintaining Databases
19.1.1 PowerShell and sqlcmd
19.2 Creating a New Customer Account
19.2.1 Account Data
19.2.2 Structure of the Script
19.3 Storing Exif Metadata in a Database
19.3.1 PyMySQL
19.3.2 Saving Exif Metadata
19.4 Importing JSON Data into a Table
20 Scripting in the Cloud
20.1 AWS CLI
20.1.1 Installation on Linux and macOS
20.1.2 Configuration
20.1.3 Getting Started
20.1.4 Encrypting Files
20.2 Example: Uploading Encrypted Backup Files to the Cloud
20.3 AWS PowerShell Module
20.3.1 Getting Started
20.3.2 Copying Files
20.4 Example: Offloading Large Files from a Website to the Cloud
20.4.1 Preparations
20.4.2 Script
20.4.3 Limitations
21 Virtual Machines
21.1 Setting Up and Running Virtual Machines (KVMs)
21.1.1 Cloning Virtual Machines
21.1.2 Starting and Shutting Down Virtual Machines
21.1.3 Running Scripts on Multiple Virtual Machines
21.2 Automating the Network Configuration (KVMs)
21.2.1 Starting Point
21.3 Controlling Hyper-V
21.3.1 Cloning Virtual Machines
The Author
Index
Service Pages
Legal Notes