Chapter 1 - Vagrantfile, Virtual Box, and Docker for Mac Installation
- Nets Docker For Mac Download
- Nets Docker For Mac Catalina
- Nets Docker For Mac Os
- Nets Docker For Mac Installer
CONTAINER=slave-play # Name of the docker container HOSTDEV=ethHOST # Name of the ethernet device on the host GUESTDEV=test10gb # Target name for the same device in the container ADDRESSANDNET=10.101.0.5/24 # Next three lines hooks up the docker container's network namespace # such that the ip netns commands below will work mkdir -p /var. A few tricky implementation details. ip netns commands do not work by default with docker net NS. Workaround: nsenter or symlink /var/run/docker/netns to /var/run/netns. vxlan must be created in host Network NS and moved in the overlay NS. Keeps a link with the host eth0 interface. Otherwise vxlan will not be able to go outside. #!usr/bin/env bash: function setupoverlay : ip netns add overlay: ip netns exec overlay ip li ad dev br0 type bridge: ip li add dev vxlan42 type vxlan id 42 l2miss l3miss proxy learning dstport 4789. Not a member of Pastebin yet? DEBU0265 Setting bridge mac address to 02:42:8f:50:4b:fc DEBU0265.
- Download Vagrant Up for Machttps://www.vagrantup.com/downloads.html
- Download Virtual Box for Machttps://www.virtualbox.org/wiki/Downloads
- Download Docker Toolbox for Machttps://www.docker.com/products/docker-toolbox
- Download Docker for Machttps://www.docker.com/products/docker#/mac
- Forum for Docker https://forums.docker.com/c/docker-for-mac
- Guide for Docker for Machttps://docs.docker.com/docker-for-mac/
- Examples for Docker for Machttps://docs.docker.com/docker-for-mac/examples/
- Setup Docker Compose and MySQLhttps://docs.docker.com/compose/rails/
Check Docker Client
Create CoreOS Docker host using Vagrant VM by:
- Note: Use of Vagrant seed for docker development is an alternative approach to using
docker-machine
. - Cloning coreos-vagrant Git Repo into host machine directory
docker-host
- Making the following changes to the Vagrantfile
- Expose Docker TCP connection port in
config.rb
of coreos-vagrant directory- Note that Vagrantfile creates a CoreOS VM using VirtualBox software hypervisor
- Note that Vagrantfile parses
config.rb
containing a set of options configuring CoreOS cluster
- Generate
user-data
file containing#cloud-config
to simplify provisioning of CoreOS VM using built-incoreos-cloudinit
- Startup and SSH using VirtualBox Provider (default Vagrant provider)
- Note
vagrant up
triggers vagrant to download CoreOS image (if necessary) and relaunch instance - Note
vagrant ssh
connects to VM from directory with Vagrantfile
- Note
Use Vagrant VM:
- Setup shell environment so local Docker client may communicate with Docker daemon on VM
- Show Vagrant environments on host machine
- Check Docker environment variables
- Important: Restart Bash terminal to avoid error connecting to Docker Daemon
- Connect to Docker Daemon (running on port 2375). Note use unencrypted only in Development.
- Important Note: Docker Daemon of Docker for Mac or Docker Toolbox already runs in a Linux VM, so you do not need to (and cannot) run it manually with
dockerd
. It is already running in the whale icon is in the top bar. https://github.com/docker/docker/issues/27102 - Connect to Shell on Vagrant VM
Share Folders between Vagrant VM (CoreOS) and Host (OSX). IDE may now be used to edit files on Vagrant VM.
- Enable synced folders by editing Vagrantfile as follows (uncommenting the
config.vm.synced_folder
line of code)
- Restart Vagrant VM
- Note: If an exports invalid error occurs then open and validate the contents of
/etc/exports
on the Host (OSX) - Check that files synchronise between these folders
Interactive Docker Containers within Vagrant VM (Docker host):
- From the shell of Vagrant VM, test download a Docker Image from Docker Hub for the base distribution specified using Docker Daemon(i.e. Ubuntu, Fedora, and CentOS). A Docker Container inside Vagrant VM filesystem is created based on the downloaded Docker Image.The new Docker Container has a network, IP addresss, and bridge to communicate with localhost.The command and flags run by Docker on the new container launch an interactive Bash shell instance of the host (i.e. Ubuntu) in it that we are logged into as root user.
- List all available Linux commands, aliases, built-ins, keywords, and functions
- Show Docker Container Host Entries
- Show Docker Container IP address
- Show Docker Container Running Processes
- Install software package on Docker Container
- Exit Docker Container to return to Vagrant VM
- List Docker Containers (both stopped and running) from within Vagrant VM. Note: Those that were run with
--rm
flag are not shown.
- List Docker Containers (only last 2 that were stopped or running)
- Create Container (but not Run), Start, Restart, Re-Attach (to process using original Options, i.e. command, flags), and Delete Docker Container
Important Note: Press Enter after Re-Attach to Container (otherwise it hangs)
Daemonized (non-Interactive) Docker Containers within Vagrant VM (Docker host) for running apps and services:
- Create and run a Daemonized Docker Container within Vagrant. Flags detach the container to the background and performs a command until process stopped.
- View under the hood of the Docker Container by debugging the last few log entries
- Monitor (follow) the Docker Container log entries using the
-f
flag (similar to thetail -f
binary) with timestamps and without reading previous entries to the log file
- Note: Transport Layer Security (TLS) is a cryptographic protocol that provides communications security over a network (predecessor SSL)
- Inspect processes (and PIDs) of Daemonized Docker Container
- Statistics (CPU, memory, network, storage I/O, metrics) of Daemonized Docker Container
- Background Processes (using detach
-d
flag) started on already running containers are executed (management, monitoring, or maintenance) with Process Owner in Daemonized Docker Container
- Note:
docker exec
works on already running containers, otherwise error returned:
- Inspect Docker Container (config info, names, commands)
- Selectively Inspect Docker Container (using GoLang template with
-f
flag https://golang.org/pkg/text/template/)
- View Docker Container storage location on Vagrant VM (i.e. CoreOS). Start shell as superuser first
Log Collection
Nets Docker For Mac Download
Chapter 2 - Dockerfile, Docker Images
- Docker Container based on a Docker Image (based on a public image or custom) stored in Repository of a Registry
- Docker Image has filesystem layers with mapping to each build step
- Docker Image Management storage backend communicates with underlying Linux filesystem to build layers into usable image
- Docker Image Management storage backends include fast Copy-on-Write (CoW)
- Docker Image Management storage backends supported include AUFS, BTRS, Device-mapper, and overlayfs
- Docker Hub is the default Registry
- Top-Level Repositories are managed by Docker and vendors (i.e.
ubuntu
) - Top-Level Docker Images offer bare runtime to run the distribution
- User Repositories are by individual developers (i.e. ltfschoen/ruby)
- Tag comprises a series of image layers in single repository (representative of version control)
- Docker Image is built upon the Union Mount of its parent image layers above the base image (each layer is a filesystem)
- Show Docker Images downloaded from repositories that contain layers and metadata
- Go to local storage location of Docker Images
- Pull specific Tag (i.e. 3.4) for Top-Level Docker Image
- Search Docker Hub for publicly available Docker Images that may already be pre-installed with applications
- Login/Logout of Docker Hub
Docker Build Command (using Dockerfile)
Docker File
- Dockerfile and
docker build
is repeatable, transparent, and idempotent, and so is preferred overdocker commit
- Dockerfile instructs how Docker Image is to be assembled using application code.
- Dockerfile uses DSL with instructions paired with arguments on each line of code
- Dockerfile lines of code are processed from top to bottom
- Each line of code creates a new Docker Image Layer.
- Only layers deviating from previous Docker Image Layer must be built
- Dockerfile
docker build
process is:1) Docker runs an initial new Docker Container from the base Docker Image (FROM
)2) Docker executes instruction (first line of code) inside initial Docker Container3) Docker runs adocker commit
to commit initial Docker Image Layer4) Docker runs a second new Docker Container based on the initial Docker Image Layer5) Docker executes instruction (next line of code) inside second Docker Container6) Docker runs adocker commit
to commit second Docker Image Layer7) Docker runs a third new Docker Container based on the second Docker Image Layer8) Docker repeats steps 5) to 7) for subsequent lines of code to build up a Docker Container - Docker Container that is built houses the application (i.e. Rails-based app) with all dependencies
- Debug errors when building from Dockerfile by running Docker Container from the last successfully created Docker Image
- Dockerfile
FROM
- Default instance is base Linux image
- Custom Docker Registry offers official framework Docker Images and Tags (i.e. Node.js). Docker Image base inherits from Docker Container defined by
FROM
(i.e.node:0.10.33
specifically locks Docker Image base to specific Ubuntu Linux image running Node 0.10.33)
- Dockerfile
MAINTAINER
- provides author field metadata in Docker Image - Dockerfile
LABEL
- provides KV pairs of metadata in Docker Image for search purposes. Find withdocker inspect
- Dockerfile
USER
- Important: Docker Containers run processes on the host machine kernel. Do not be run as root user in Production for security
- Dockerfile
ENV
- set shell environment variables used in build process on the image - Dockerfile
RUN
- execute instruction/command to append new Docker Image Layer on the current Docker Image
- start and create file structure, and install dependencies
- commit the new Docker Image Layer if command successful
- Important: Reduce build time by abstracting common
RUN
processes (i.e. updates) into code lines (Docker Image layers) of Base Image - Important: Reduce rebuild time by ordering commands with code imports at the end (as all layers after introduced change are rebuilt)
- Important: Combine logically grouped commands into single Dockerfile code line since each instruction creates a new Docker Image layer (i.e. use
ADD
to import scripts and required supporting files from local filesystem into image) - Dockerfile
WORKDIR
- changes working directory in image for remaining build instructions - Dockerfile
CMD
- final instruction launches process to run in Docker Container - Dockerfile
EXPOSE
- ports Docker Container listens to.docker run
flag (--expose
) opens these ports at runtime - Important: Single function in
CMD
to easy horizontal scaling of individual functions in architecture
Build Custom Container
- Build new Custom Container based on Top-Level Container that we build from a Top-Level Image
Nets Docker For Mac Catalina
Example - Static Web Server
- Create build environment (build context) for Docker to upload to Docker Daemon when build is run.
- Note: Previously we configred the Vagantfile to synchronise the following folders between Vagrant VM (CoreOS) and Host (OSX):
~/code/docker-host-coreos-share
(Host)/home/core/share
(Vagrant VM)
- Create Dockerfile that builds a Docker Image where the Docker Container is a static web server
- Note: Perform these on Host machine since folders are synchronised
- Build Dockerfile into Docker Image.Tagging with repository/image_name:version.Use
-f
flag to specify directory within build context containing Dockerfile. - Note:
docker build
must be run on the Vagrant VM (CoreOS), otherwise the following error message appearsCannot connect to the Docker daemon. Is the docker daemon running on this host?
- Check Docker Images updated
- Check Docker Image history (Docker Image Layers are shown)
Docker Image Templates using Build Cache
- Docker uses cache and re-builds from earliest image layer change.Bypass using
--no-cache
flag. - Dockerfile should have template at top of file with package repositories and updates so cache is hit.
Launch Container using Docker Image
-d
flag detaches in background for long processes (i.e. Nginx daemon)-p
flag manages network ports published by Docker at container runtime to the host machine, with options including:- Mapping of randomly assigned Docker Host port(i.e. 32768 to 61000) to Port 80 on Docker Container
- Mapping of specific Docker Host port (i.e. 8080) direct binding to Port 80 on Docker Container(i.e.
sudo docker run -d -p 8080:80 --name static_web ltfschoen/static_web_8080 nginx -g 'daemon off;'
) - Mapping of specific Interface (i.e.
127.0.0.1
) and specific Docker Host port (i.e. 8080) direct binding to Port 80 on Docker Container(i.e.sudo docker run -d -p 127.0.0.1:8080:80 --name static_web ltfschoen/static_web_8080_i127_0_0_1:v1 nginx -g 'daemon off;'
) - Mapping of specific Interface (i.e.
127.0.0.1
) and random Docker Host port to Port 80 on Docker Container(i.e.sudo docker run -d -p 127.0.0.1::80 --name static_web ltfschoen/static_web_random_i127_0_0_1 nginx -g 'daemon off;'
) - Mapping of random Docker Host port to Port 80 on Docker Container and publish additional ports specified with
EXPOSE
instructions(i.e.sudo docker run -d -P --name static_web_all ltfschoen/static_web_all nginx -g 'daemon off;'
) - Note: Random port is assigned when not specified
- Note: Only one Docker Container may be bound to a specific port on host machine
- Note: Bind UDP ports by appending to port binding
/udp
- Command to run (i.e.
nginx -g 'daemon off;'
) launches Nginx in foreground to run web server
- Check Port mapping on host machine assigned to a given port 80 of a Docker Container ID/name
- Selectively Inspect Docker Container
- Connect to the Docker Container housing the Nginx web server using cURL
- Attempt with the following:
- Destroy Docker Containers and re-build Docker Image with different Dockerfilewith change to
EXPOSE 80
(not necessary to delete the Docker Container withdocker rmi ltfschoen/static_web:v1
though due to caching)
- Check port 80 is correctly binded by Nginx
- Check IP Table rules that are in place
- Attempt Bridged Adapter option
- Error below when running
vagrant up
after changing from NAT to Bridge Adapter in VirtualBox GUI > Settings > Network > Adapter 1 because Adapter 3 is already set to Bridge Adapter, and making following changes to Vagrantfile:
- Solved with
vagrant halt
, started VM from Virtual Box GUI, then goto bar menu and chose quit VM. https://github.com/mitchellh/vagrant/issues/1809Note: This solution causes another cascading problem: - Error encountered when run
vagrant up
again
- Solved by having both Public and Private in Vagrantfilehttps://github.com/mitchellh/vagrant/issues/2748http://stackoverflow.com/questions/24984020/unable-to-connect-to-vagrant-apache-on-one-computer
- Note: Still unable to get response from VM when curl from Host to Docker Container IP/port (i.e. curl 172.18.0.2:8080)
Misc Links:http://stackoverflow.com/questions/33814696/how-to-connect-to-a-docker-container-from-outside-the-host-same-network-windohttps://www.virtualbox.org/manual/ch06.htmlhttp://stackoverflow.com/questions/25433488/vagrant-can-not-ping-guest-machine-from-the-hosthttp://serverfault.com/questions/495914/vagrant-slow-internet-connection-in-guesthttps://friendsofvagrant.github.io/v1/docs/bridged_networking.html
TODO Default Router approach:http://superuser.com/questions/752954/need-to-do-bridged-adapter-only-in-vagrant-no-nathttps://www.vagrantup.com/docs/networking/public_network.html
TODO Key Link:Alternative using Docker Compose instead of Vagrant https://hharnisc.github.io/
In progress up to CMD section on page 99
Other Links
Other
- Note: Docker Toolbox is for older Macs.
- Uninstall Docker Toolbox. Download shell script and run with
sudo bash uninstall.sh
https://github.com/docker/toolbox/blob/master/osx/uninstall.sh
TODO
- Relevant folders:
- /Users/Ls/code/docker-host-coreos-share/ltfschoen/static_web
- /Users/Ls/code/docker-host/coreos-vagrant
- Learn Chef with Docker
- Control Docker Daemon with:
dockerd
- Apache Kafka (broker) and Apache Zookeeper (run using Docker)
- https://github.com/HackerHappyHour/docker-toolbox-vagrant
- https://docs.docker.com/engine/tutorials/dockerimages/#building-an-image-from-a-dockerfile
- https://docs.docker.com/engine/reference/builder/
- https://www.percona.com/blog/2016/05/11/quick-start-mysql-testing-using-docker-mac/
The information in this section explains configuring container networks within the Docker default bridge. This is a
bridge
network named bridge
createdautomatically when you install Docker.Note: The Docker networks feature allows you to create user-defined networks in addition to the default bridge network.
While Docker is under active development and continues to tweak and improve its network configuration logic, the shell commands in this section are rough equivalents to the steps that Docker takes when configuring networking for each new container.
Review some basics
To communicate using the Internet Protocol (IP), a machine needs access to at least one network interface at which packets can be sent and received, and a routing table that defines the range of IP addresses reachable through that interface. Network interfaces do not have to be physical devices. In fact, the
lo
loopback interface available on every Linux machine (and inside each Docker container) is entirely virtual – the Linux kernel simply copies loopback packets directly from the sender’s memory into the receiver’s memory.Docker uses special virtual interfaces to let containers communicate with the host machine – pairs of virtual interfaces called “peers” that are linked inside of the host machine’s kernel so that packets can travel between them. They are simple to create, as we will see in a moment.
The steps with which Docker configures a container are:
Nets Docker For Mac Os
- Create a pair of peer virtual interfaces.
- Give one of them a unique name like
veth65f9
, keep it inside of the main Docker host, and bind it todocker0
or whatever bridge Docker is supposed to be using. - Toss the other interface over the wall into the new container (which will already have been provided with an
lo
interface) and rename it to the much prettier nameeth0
since, inside of the container’s separate and unique network interface namespace, there are no physical interfaces with which this name could collide. - Set the interface’s MAC address according to the
--mac-address
parameter or generate a random one. - Give the container’s
eth0
a new IP address from within the bridge’s range of network addresses. The default route is set to the IP address passed to the Docker daemon using the--default-gateway
option if specified, otherwise to the IP address that the Docker host owns on the bridge. The MAC address is generated from the IP address unless otherwise specified. This prevents ARP cache invalidation problems, when a new container comes up with an IP used in the past by another container with another MAC.
With these steps complete, the container now possesses an
eth0
(virtual) network card and will find itself able to communicate with other containers and the rest of the Internet.You can opt out of the above process for a particular container by giving the
--net=
option to docker run
, which takes these possible values.--net=bridge
– The default action, that connects the container to the Docker bridge as described above.--net=host
– Tells Docker to skip placing the container inside of a separate network stack. In essence, this choice tells Docker to not containerize the container’s networking! While container processes will still be confined to their own filesystem and process list and resource limits, a quickip addr
command will show you that, network-wise, they live “outside” in the main Docker host and have full access to its network interfaces. Note that this does not let the container reconfigure the host network stack – that would require--privileged=true
– but it does let container processes open low-numbered ports like any other root process. It also allows the container to access local network services like D-bus. This can lead to processes in the container being able to do unexpected things like restart your computer. You should use this option with caution.--net=container:NAME_or_ID
– Tells Docker to put this container’s processes inside of the network stack that has already been created inside of another container. The new container’s processes will be confined to their own filesystem and process list and resource limits, but will share the same IP address and port numbers as the first container, and processes on the two containers will be able to connect to each other over the loopback interface.--net=none
– Tells Docker to put the container inside of its own network stack but not to take any steps to configure its network, leaving you free to build any of the custom configurations explored in the last few sections of this document.--net=<network-name>|<network-id>
– Tells Docker to connect the container to a user-defined network.
Manually network
To get an idea of the steps that are necessary if you use
--net=none
as described in that last bullet point, here are the commands that you would run to reach roughly the same configuration as if you had let Docker do all of the configuration:Nets Docker For Mac Installer
At this point your container should be able to perform networking operations as usual.
When you finally exit the shell and Docker cleans up the container, the network namespace is destroyed along with our virtual
eth0
– whose destruction in turn destroys interface A
out in the Docker host and automatically un-registers it from the docker0
bridge. So everything gets cleaned up without our having to run any extra commands! Well, almost everything:Also note that while the script above used modern
ip
command instead of old deprecated wrappers like ipconfig
and route
, these older commands would also have worked inside of our container. The ip addr
command can be typed as ip a
if you are in a hurry.Finally, note the importance of the
ip netns exec
command, which let us reach inside and configure a network namespace as root. The same commands would not have worked if run inside of the container, because part of safe containerization is that Docker strips container processes of the right to configure their own networks. Using ip netns exec
is what let us finish up the configuration without having to take the dangerous step of running the container itself with --privileged=true
.