Let me tell you why you're here.
You're here because you know something, but you can't explain it.
You've felt it your entire career.
There's something wrong in your front end developer skillset -- you don't know what it is, but it's there.
Like a splinter in your mind. Driving you mad.
It is this feeling that has brought you to me. Do you know what I'm talking about?
Of course you do. It's Unix shell scripting ?
You never learned this world of scripting -- bash commands and .sh files just looked like something other devs did. You, much like me, just copied and pasted commands and hoped for the best.
It's good to take a little time to learn shell scripting, because it's helpful and it's actually not that hard to learn. The basic commands will take you far.
Knowing how to use shell commands will also make you a more valuable software developer. Plus, you will be just that much closer to seeing the matrix like Neo ?
In this first post of my Unix shell scripting series, I will try to explain shell scripting in a simple way, and I'll show you the foundational commands that will make you productive right away.
With that said, pick the red pill, stay in wonderland, and let's see how far the rabbit hole goes...
Unix on MacOs
You've been living in a dream world, Neo
The MacOS user interface is hiding from you a lot of what happens behind the scenes. The reality is, everything you do on a Mac is being controlled by Unix commands -- moving files around, copying/saving, searching for files, downloading -- the list goes on and on. The visual layer that we interact with, is just a pretty looking design that lets us control the lower things.
OK, So What is Unix exactly?
Unix is the core of the Mac operating system. It is what runs "under the hood" of the pretty UI we are used to and interact with. You don't need to know much about Unix to get started using it, but here are some key points that are helpful:
- It is a mainframe OS. This just means it controls low level things like the device's memory, processor, and file system, to name a few things.
- Not all Unix distributions are the same.
- Unix on MacOS is based on The BSD Unix (Berkley Software Distribution)
- The MacOS version of Unix is called Darwin, Apple's proprietary improvements built on top of BSD Unix
- Unix has no pretty UI, just a blank terminal screen (a command line interface) that takes commands
- You can interact directly with Unix through the MacOS terminal
Let's get you ready for our ride. Sit still. This won't hurt...
That helps, but what is bash scripting, and what is the Unix shell?
OK, you may have heard these before, but never really took time to know what they were.
Let's get these terms out of the way. You will see they are very easy to understand.
You control Unix through a command line interface, as I said earlier. Unix commands need to be interpreted into something the innermost/lower-level components of Unix can understand. Unix requires a "shell" to do this.
The shell is the program that sits between us and the low level Unix things, that manages a lot of technical details, and interprets our commands to Unix in a way Unix can understand and execute our commands. Any time we write a command on the command line, we are doing it through a shell, therefore we never write commands outside of a shell program.
Unix is not married to one specific shell. This means you can actually pick a shell you like to work with, and use that. One popular shell is called Bash.
By default, MacOS uses the Bash shell. After some years of using Bash I thought I'd try a different shell that seemed to have some additional features. So, right now I personally use a shell called "Zsh". If you're interested in the differences between these two shells you might find this article intertesting.
That's pretty much all the background info you probably need about the Unix shell. Let's dive deeper.
What Are You Waiting for Neo? You are faster than this
You'll be suprised what you can do on the command-line to speed up your workflow. You can automate moving and processing files, downloading data, replacing text in files, creating files and directories, searching files, and much much more.
The main idea with scripting is to make your work life easier. You can create commands that will automate repetitive tasks, and in the end make you work faster.
In front end development, this usually means we will automate some part of our project's build process, but there are many other reasons to use shell commands.
Can I use programs I have installed on my machine to help automate things?
Yes you can. This is a big reason for automation. Most programs are built to allow you to use them in the command line, through a CLI (command line interface).
An example of a CLI can be seen when running Node.js. Node.js doesn't ship with a visual interface. You can only interact with it through commands.
Other examples are Java, or the Android SDK, or Xcode. They all have a CLI that lets you interact with these programs. These are just a small list of programs. Like I said, most programs out there that you will use in front end development will probably have a CLI.
Note: It's important for programs to be this flexible, because automations are rarely done on a user interface. Without the command line access, automating program tasks would be really difficult to do.
So, lets start with basic Unix commands you should know, but first, I want to make sure you know exactly where to write the commands.
How to open the terminal
Before we start, I'll show you how to actually open the terminal on MacOS, so you can write commands into it.
To open it, simply open your finder, and open Applications/utilities/Terminal
You might want to add the terminal program to your dock, so it's easy for you to open later.
By default, MacOS will use the Bash shell. So when you open the terminal, you will be logged into a Bash shell instance.
Basic Shell Commands
As I said earlier, you interact with Unix by executing commands. To execute a command, simply type a command in the terminal, and hit enter.
There are commands that allow you to do probably more than you will ever need. Since we are talking about front end developers here, we will stick to what is most useful to front end developers, and I will touch on some other commands that are good to know as well.
Let's get into it. First, I'll show you a useful command, that helps you get info on any command you want to learn about. There is a handy command in Unix, called man, that does this.
man
man = The command manual that will tell you anything you want to know about any Unix command.
Bash Script Example
This example would output the manual for the ls command (the ls command just lists files on a directory) and would tell you the options associated with it. This is really helpful when you are not sure what options a command has.
man ls
Output
The manual on how to use the ls command, separated in pages.
While viewing a manual:
- Hit the spacebar to go to the next page
- Hit f to scroll the page (basically the same as spacebar)
- Hit b to go to previous page, which basically scrolls the page back
- Hit q to close the manual
The next command you typically learn is echo
echo
echo = Lets you print values on to the console. Similar to console.log in Javascript. You can print messages or print variable names to see their value.
Bash Script Example: Printing a message
echo Hello World
Output
Hello World
Bash Script Example: Printing a variable
This will output a series of paths, tied to the $PATH variable:
echo $PATH
Output
/Users/embp/.nvm/versions/node/v10.9.0/bin:/Users/embp/.fastlane/bin:/Users/embp/android-sdk-macosx/platform-tools...
Note: More on the $PATH variable later, but it is basically a very commonly used global variable in Unix that stores paths
Exiting
Exiting is odd to me, it is not consistent. It is annoying enough that I had to make sure I got in all the possible ways to exit here.
Typically, you can hit q when you are in a paginated output (such as the output the man command gives you)
You can also try x, ctrl + q, ctrl + x, or ESC
The more common ways to exit would be force quit with ctrl + c, or type exit.
We haven't really done much with commands that would require and exit yet, but I have written this and will be here for you when you need it.
Going to a Previous Command
When you have a lot of commands you've written, you don't want to write them over again later. You can press the ⬆️ or ⬇️ arrows in your keyboard, to go back/forward to the commands you have already used.
Going to the beginning or end of a command, or to a specific character in a command
Sometimes you want to edit a command you wrote already, this will help you go to the exact spot you need to edit.
ctrl + a = Move cursor to beginning of the line
ctrl + e = Move cursor to the end of the line
option + click = Moves cursor to where you clicked
Autocomplete commands or file names
You can save some time and have the shell help you autocomplete commands of file names for you
tab = Auto complete command or file name
tab + tab = Whenever auto complete doesn't work, it will show a list of possible matches
Clearing your screen of output
Sometimes you have too much output in your Terminal. Run cmd + k to clear your output.
** Note: This will not exit/force quit your running script. It will just clear the output. **
Multiple Commands
You don't need to run a command one by one. You can separate commands with semicolons.
Bash Script Example
ls; echo $PATH; echo complete!
That concludes the absolute basics. Are you ready for more?
Cool! Let's continue by navigating the file system!
Navigating the File System
Unix lets you manipulate files in any way possible. You can view files in the command line, you can create new files, you can list files/directories, or launch an app that shows the file you want to view.
To do this, it's first important to know that you need to target the directory the file is located in first, before you can actually do anything with a file. To do this, you typically will need to change directory from where you are to the directory the file is located in.
When you open the Terminal, the default directory you are targeting at that time is typically your user directory (the directory with your name). To switch to another directory, you will use the cd command.
cd
cd = Change directories
Bash Script Example
cd to/some/other/directory
Note: If you know the name of the directory, you can type some of the name, then hit tab to autocomplete the rest of the name. If you are not sure what the name of the directory is, you can hit tab twice to get a list of options.
Other cd tricks
You can type cd and press the up/down arrow key to get the last directories you changed to.
To go back to the parent folder, just use cd ..
To go back multiple parent folders, just use cd ../../../, adding as many ../ as you need to go to the right parent folder
To go to the top most folder in your Mac, type cd /
To go to your user folder (the folder with your name on it), type cd ~
To toggle between current and previous directory, type cd -
ls
Now that you can change directories, you might want to know what files are in a directory. To do this just use the ls command.
This command will list all files and directories in the current directory you are in.
There are a few options you can add to the command, which are explained below:
ls = List files/folders located in the current directory
ls -l = List files vertically, and display details about each file/directory
ls -la = List all files, including hidden files/directories
ls -lah = List all files as -la, but also showing file sizes
pwd
If you are ever confused on what directory you are targeting commands into, simply execute the pwd command. This returns the present working directory.
Manipulating the File System
Now that you know how to change to directories and list files, let me show you how you can read, modify, create and delete files.
How to Read Files
There's a few useful commands for reading files:
cat = Outputs the contents of the file.
Bash Script Example:
cat filename.txt
less = Outputs contents of files, with pagination.
Bash Script Example:
less filename.txt
cat -n = Will output the contents of multiple files.
Bash SCript Example:
cat -n file.text file2.txt
Note: Doesn't have to be .txt, can be any file type that can be read.
How to Modify Files
mv = Move file -- also commonly used for renaming files
Bash Script Example: Move a File
mv myFile.txt my-directory-name
Bash Script Example: Rename a File
mv myFile.txt my-renamed-file.txt
cp = Copies a file. You can't copy a directory with this command.
cp -R = Use this to copy directories. The -R option tells the cp command to copy recursively. This means it will copy a directory and its children.
Bash Script Example
In this example, "some-file.txt" is copied as "some-file-copy.txt":
cp some-file.txt some-file-copy.txt
In this example, "some-directory" is copied as "some-directory-copy". All files and directories inside of "some-directory" will also be copied into "some-directory-copy":
cp -R some-directory some-directory-copy
How to Create New Files/Directories
mkdir = Creates a directory
Bash Script Example
mkdir myNewDirectory
touch = Updates access time of a file, or if the file doesn't exist, will create it with default permissions.
Bash Script Example
touch someFileName.txt
How to Delete Files
rm = Removes a file (does not send to trash!)
rm -rf = Deletes a file or directory recursively (does not send to trash!)
Bash Script Example
I tend to always use rm -rf even on single files. This command will remove "some-directory" and any files/folders inside of it.
rm -rf some-directory
Note: Worth noting again that using these commands will not send the files to the trash. You will delete them immediately. So make sure you really want to delete the file before you run this command!
I have shown you a good amount of commands so far. Play with those commands and make sure you are comfortable with them.
Once you are ready lets continue our trip through the file system and see what else we can do.
User Types And Their Privileges
In software development, you will run into issues where you cannot run a command because the folder you want to execute a command on doesn't allow you to do it.
This happens a lot when running build scripts on your machine and things like that.
Here are some tools that will let you make a Matrix move around all those pesky directory and file ownership issues.
User Types
When you use a shell, you are typically defaulted to the MacOS logged in user. There are folders that have different owners, and some commands may not be executable if the user you are logged in as doesn't have permission. This is the reason for the sometimes frustrating situation where you try to modify a file, or run an installation script, but it won't let you because you don't have permission.
It means the user type you are logged into in your Mac, does not allow you to perform the operation.
There are three types of users you should be aware of.
The first is the root user, second is an admin user, and the other is a non-admin type of user.
The Root User
The root user refers to the top user who can modify and execute anything in the OS, outside of admin users. You will not ever need to log in as the root user, and in fact it is a security issue to do so.
The root user is disabled by default on MacOS.
Admin User
This is a user that has elevated privileges -- more so than regular users, but less privileges than the root user.
It will be common to have an admin user in your Mac to have freedom of making changes to the system and files.
Non-admin User
This is a more restricted user type, which is necessary to disallow unathorized users from modifying your computer in ways they should not be allowed to do so.
Take on Privileges, With the sudo Command
As I said, login as root user is disabled, but it does have some execution privileges you may need. If that's the case, this is where the sudo command comes into play.
sudo Stands for "substitute user and do". It executes the provided command as root, which is able to do anything in the computer.
Note: Notice I said it executes AS root, but it does not log you in as root. It is the recommended way to execute root commands, in a secure manner.
Bash Script Example
sudo mv my-file.txt my-renamed-file.txt
When you run the sudo command, you may be prompted for a password. This should be the password you established as your admin user.
With sudo you can also take on the privileges of other users, not just root:
sudo -u someotheruser commandtoexecute
Not everyone can do sudo, only admins. This is setup when creating a user account.
Determine Who You Are Logged in as
If you are unsure of what user you are logged in as. Try these commands below:
whoami = Tells you the user you are logged in as.
groups = The groups that your user belongs to.
File/directory permissions
Unix lets you control permissions on a file pretty easily. Before I show you how to update permissions on a file, let me show you how to determine what permissions are applied to a file before you actually modify a permission.
Determine existing permissions first
Let's go back to the ls command, using the -l option:
ls -l
This is going to give you a list of files for your current directory. To the left of that list, in the first column, you will see some odd-looking characters next to each item, that look something like this:
drwxr-xr-x
-rw-r--r--
drwxrwxr-x
These characters are telling you whether the item is a file or directory, and what permissions are applied to that item.
Directories will always start with a 'd' as the first character:
drwxr-xr-x
Items that are not directories will start with a '-':
-rw-r--r--
After this initial character, you will have a series of characters, 'r', 'w', 'x', and '-'. These characters define the permissions the item has:
'r' = This file/directory is allowed to be read
'w' = This file/directory is allowed to be written to
'x' = This file/directory is allowed to be executed
'-' = This file/directory does not have this permission. The permission affected depends on where the '-' is set.
You'll notice these characters can be repeated many times. This is because each file or directory has 3 "user" categories permissions can be set to.
The categories, from left to right, are "user", "group", and "other".
Unix gives you full control for setting permissions for each category. Since there are 3 permissions you can set, each category will have 3 slots you can activate a permission onto it.
Bash Script Example: User permissions
For example, if a file had read, write, and executing permissions for a user, but not for groups or other, you would see a permission that looks like this:
-rwx------
If the file had only read permissions for a user it would look like this:
-r--------
Notice how the '-' character works. For permissions that are denied, a '-' is used.
Note: Remember the first '-' in the line of my example just tells you this item is a file. It is not telling you about a denied permission.
Bash Script Example: Group and other permissions
If a file only had group permissions you would see this:
----rwx---
If the permissions were set for the "other" category of users. You would see this:
-------rwx
Bash Script Example: Permissions on all categories
If we allowed all permissions for all 3 categories, you would see this:
-rwxrwxrwx
Now you should be able to read the permissions like a kung fu masta :)
Setting permissions with chmod
Now that you know how to read permissions. Setting them will be just as easy.
The command you will use is chmod.
chmod = Stands for "change mode"
You can write the command in two different ways.
1. The less-common way
You will write chmod, then as options you will target each category I talked about earlier ("user", "group", "other") and set the permissions to it.
Bash Script Example
The below example sets permissions to each category, replacing whatever permissions where previously set:
chmod u=rwx,g=rwx,o=rwx someFile.ext
Bash Script Example: Modify permission
Slightly quicker way to write. This will update user and group ("ug") to add (+) the write permission. Existing permissions are not affected:
chmod ug+w someFile.ext
You can also remove a permission by changing the + into a -:
chmod ug-w someFile.ext
Bash Script Example: Modify permissions recursively
So far we have only modified permissions for one file or folder.
If you modify a folder, its subfolders are actually not modified with the permission.
To make sure all sub folders and files are modified as well, use the -R option:
chmod -R a+x somedirectory
Octal notation permissions
There is an even quicker way to set permissions. And this one is the most common way to do it. You write numbers that equal to the permissions you want to set.
With octal notation, read = 4, write = 2, execute = 1, when we add up the 3 numbers when setting for each category, it represents the state of the permissions for a file.
Bash Script Example
The highest number (777), adds permissions for users, groups, other, and gives read, write, execute permissions:
chmod 777 filename.ext
This would set read, write, execute. Everyone else just read:
chmod 755 filename.ext
This would give no permissions to anyone:
chmod 000 filename.ext
Changing File/Directory Ownership
In Unix, you can take ownership of a file away from the one who owns it. In my experience as a software developer, I have used this typically to gain read, write, and execute permissions on a file. This is what you will most likely use it for as well.
chown = Stands for change ownership
Bash Script Example
chown kevin:somegroup somefile.txt
chown kevin somfile.txt = Changes owner to the kevin user
chown :somegroup somefile.text = Changes owner to the group provided
chown -R kevin:group somedirectory = Changes owner and everything in it. -R stands for recursive. You will need sudo to make this work.
I know what you're trying to do...
Congrats Neo, you are well on your way to confronting agent Smith.
You now know a whole lot about shell scripting and can begin to read the Matrix, sorta.
This trip goes deeper. I can only show you the door, you're the one that has to walk through it.
The journey continues on the next part, coming soon...
