2025-01-08
This is a collection of personal notes. Optimistically, I will keep updating it to include new content.
The method for command substitution is backticks `command` or $(command), placing the execution result (output) of the command in place. For example:
get_username() {
echo "john"
}
username=$(get_username)
DATE=$(date)
Forms like expr 11 / 2 or $((1 + 3)). An example combining arithmetic evaluation and command substitution:
calculate_sum() {
echo $(( $1 + $2 )) # arithmetic evaluation, not command substitution.
}
result=$(calculate_sum 5 7) # command substitution
echo "The sum is: $result" # 12
Note that arithmetic evaluation uses integer operations by default. So expr 11 / 2 will print 5.
$2 or ${2} is used to refer to the second parameter, as shown in the example above. For shell scripts, $0 is the filename of the shell script. $@ represents all parameters.
echo $0
for arg in "$@"; do
echo "argument" ${arg}
if [[ ${arg} == "--task_dir" ]]; then
echo "\${2}" "${2}"
fi
done
If you run the above script ./loop_all.sh C --task_dir A B, the expected output is
./loop_all.sh
argument C
argument --task_dir
${2} --task_dir
argument A
argument B
The example above uses variable replacement: $name or ${name} will replace the variable name with its content, contrasting with command substitution $(command). Note that command substitution occurs even within double quotes, but not within single quotes:
$ echo "I am $USER"
I am cloud-user
$ echo 'I am $USER'
I am $USER
If replacing within double quotes and there is no space immediately after the variable, you must use the ${name} form:
for subset in "science" "tech"; do
if [ -f "/absolute/path/here/encode-${subset}.pkl"]; then
echo "encode-${subset}.pkl exists"
else
echo "encode-${subset}.pkl does not exist"
fi
done
Globbing does not occur within double quotes, which is evident in the following example:
$ ls *.txt
a.txt b.txt
$ ls "*.txt"
ls: cannot access '*.txt': No such file or directory
Thus, you can avoid globbing by using double quotes, which is less obvious in the following example:
$ python example.py --resource_files encoding.*.pkl
example.py will receive every path that matches the pattern encoding.*.pkl
$ python example.py --resource_files "encoding.*.pkl"
in example.py, args.resource_files will be a string that is literally "encoding.*.pkl"
Integer comparison: -eq -ne -gt -lt -ge -le. Note: only supported as conditions within square brackets, as in the example below:
if [$a -ne $b] then
echo "not equal"
fi
Logical operators: -o (or), -a (and), !, &&, ||. Example: if [[ $a -lt 100 && $b -gt 100 ]]
[ -z $a ] checks if the string length is zero, [ -n "$a" ] checks if the string length is not zero. The difference between single and double brackets is here.
There are also file test operators: -b (is block device?), -c (is character device?), -d (is directory?), -f (is regular file nor dir or device?), -g, -k, -p, -u, -{r,w,x}, -s, -e (file exists?)
An example: using -eq to check if the previous command exited without error:
if [ $? -eq 0 ]; then
echo "previous command successful"
else
echo "previous command error, returned $?"
fi
To ignore all standard error output streams (stderr) of a program, you can redirect it like program -arg argument 2> /dev/null.
Chaining Python commands in a pipeline might be feasible: program -arg argument | python -c "import sys; result=sys.stdin.readlines()[-1]; print(result.split()[-1], end=',')" | tee -a out.csv
To remove the newline following echo output, you can use the -n command line parameter.
cd is a shell built-in command and does not correspond to a non-executable file (which cd returns empty).
The principle of sudo is forking a process and execve to execute the executable file. The permission bits of /usr/bin/sudo are -rwsr-xr-x, owned by root:root, ensuring any user can execute sudo, and the process generated by executing sudo has a UID of root, with sufficient permissions.
Since cd does not correspond to an executable file, sudo cd will result in the error: sudo: cd: command not found.
**find** [-H] [-L] [-P] [-D debugopts] [-Olevel] [starting-point...] [expression]
Overview: Starting from the starting-point as the root, traverse the file tree, match files or directories according to the criteria listed in the expression, and perform corresponding actions on successfully matched files or directories.
The starting-point is a list. For example, you can execute find dir1 dir2.
The expression consists of tests, actions, operators, etc.
Tests are in the form of -testname and usually require parameters. For numerical parameters n, passing +n means greater than n, -n means less than n, and n means exactly equal to n.
-amin n: Number of minutes since last access
-atime n: Number of days since last access
-cmin n: Number of minutes since last metadata change
-ctime n: Number of days since last metadata change
-empty: File or directory is empty
-executable: Is an executable file or traversable directory
-iname pattern: Equivalent to case-insensitive -name
-mmin n: Number of minutes since last file content modification
-mtime n: Number of days since last file content modification
-name pattern: Match pattern with the file name without path
-path pattern: Match pattern with the file name with path
-perm mode: Permission bits equal to mode
-size n[cwbkMG]: File size
-type c: File type
Note the distinction among amin, cmin, mmin in the options above. This distinction also appears when using stat filename, corresponding to the "access", "change", "modify" timestamps:
Editing file content with vim updates all three
Renaming a file updates change
Listing file content with cat updates access
Appending content to a file with echo > file updates change and modify
In summary, when the read system call is used to read a file, access is updated according to the definition; when the write system call is used to write file content, modify is updated according to the definition, and since the file size metadata changes, change is also updated; when moving or renaming a file, change is updated.
-delete: Delete
-exec command;: Execute command
-printf format: Print in format
find root_path -type d -iname '*lib*' List all directories with names containing lib (case-insensitive)
find root_path -name '*.py' -not -path '*/site-packages/*' List all .py files, excluding those with site-packages in the path
find root_path -maxdepth 1 -size +500k -size -10M Descend at most one level, find all files between 500k and 10M in size
find . -not -type d -exec wc -l {} \; Count lines for all files