Recently I switched ISP providers. The one I was using started the price increase game. We picked up the ISP while it was $70 per month for 1 GB. They are the only physical line provider in my town. The town has a law that “beautifies” the power lines. This means only one ISP with cables is allowed to use them. They also have a digging bill. It was designed by a corrupt city council to keep this one ISP as the primary provider. Well, this ISP has started their price walking. They start off with $10 fines here and there. Finally, it reaches $200 after a year for 1 GB. For years, the only other option has been HughesNet. Thankfully, T-Mobile has stepped in. However, they are CG-Nated and a home lab is hard to work with.
CG-NAT
Most people don’t know what CG-NAT is and how ugly it can get. Carrier-grade NAT is where the carrier places you inside a natted bubble. This saves IP addresses. You are essentially part of a nested network. It’s all nated inside each other. It tends to be standard to have your own network, then the carriers, then the public. So while you can reach therandomadmin.com with a public IP address, you can’t reach my home network with one. CGNs do not allow for port forwarding. These networks are ideal for simple plug-and-play networks. However, they are sad for us IT home lab people.
There is a ray of hope with these types of networks. VPN over SSL. This can be done via WireGuard and other VPN software. The thing to remember is you need an external, public-facing device. That’s where Oracle comes into play. Remember in my blog about free WordPress, we built an Oracle Cloud server. We are going to use this beast to get around our CG-NAT issue and open some of our services to the world. This makes a CG-NATed home lab really hard to work with, but we have magic on our side.
The Magical Connections
Ok, so we are going to use our Oracle Cloud server and combine it with our Tailscale network. Our Oracle Cloud server already has an Apache server on it. All we have to do is enable the reverse proxy settings on it.
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod headers
sudo systemctl reload apache2
Now we need to take our Tailscale network and make some reverse proxies. We are going to be doing the Docker world this time around. You can read about setting up your Docker in WordPress here. Once you have your Docker setup on your home lab server, take note of the IP address and the port number of that Docker. Please ensure you can access it locally before proceeding with any other tasks. With this first step, our CG-NATed home lab is going public.
DNS
The next part is important. I use Namecheap DNS. What you need to do is set the DNS to the Oracle host. For my example, I am pointing to therandomadmin.com host IP address for it.dpb.one. DNS can take some time to populate as well. A good rule of thumb I follow is, if you can’t ping it and get the IP address you want, wait.
Reverse Proxy
Now we will make our reverse proxy on the Apache side. First you will need to make the conf file inside your site’s available location.
sudo nano /etc/apache2/sites-available/it.dpb.one.conf
The above command will create the file and drop you into the nano prompt. This is a configuration file for one of my home lab devices. I want you to read over this file.
<VirtualHost *:80>
ServerName it.dpb.one
ProxyPreserveHost On
# Point directly to lab app's port (9000 in this example)
ProxyPass / http://100.104.5.106:9000/
ProxyPassReverse / http://100.104.5.106:9000/
RequestHeader set X-Forwarded-Proto "http"
RequestHeader set X-Forwarded-For "%{REMOTE_ADDR}s"
ErrorLog ${APACHE_LOG_DIR}/it.dpb.one-error.log
CustomLog ${APACHE_LOG_DIR}/it.dpb.one-access.log combined
</VirtualHost>
breaking it down
Just like any other virtual host, we are starting off with our tags. Then we have the name of the server. So, when DNS points the it.dpb.one to the IP address. Apache will be like, I know this. It knows it’s a reverse proxy because it sees that Proxy Preserve Host on. Then it grabs the proxypass and the proxy return pass. This is where we want to put the IP addresses.
ProxyPass / http://100.104.5.106:9000/
ProxyPassReverse / http://100.104.5.106:9000/
Notice this line: ProxyPass / This / is important. If you want to make a subdirectory, this is where you would place it. So, let’s say I wanted it.dpb.one/nextcloud to point to another server that hosts my Nextcloud; that is where I would place it. The next part will be the inside of your network. Notice the IP address is the Tailscale IP address.
RequestHeader set X-Forwarded-Proto "http"
RequestHeader set X-Forwarded-For "%{REMOTE_ADDR}s"
The next two lines are for custom items. The first tells us we are using HTTP. So the requests internally are HTTP. You can set this to HTTPS if you like. However, you will get cert errors. The second line is to pass the remote user’s IP address and not the host’s to the machine in question. If you don’t want the user’s IP address to hit your machine, then leave this line out. While the final two lines are for logging.
Now run the next two commands; the first is to enable the conf file, and the final is to load it into the Apache system.
sudo a2ensite it.dpb.one.conf
sudo systemctl reload apache2
Certbot
Finally, we like having SSL on things. I feel like if you are not using an SSL on a site, something is wrong with you. SSLs are free thanks to Certbot and the Electronic Frontier Foundation. Please donate to these people as they are doing great work. So on the Oracle Cloud server, you can run the certbot command for the domain. If you don’t have Certbot installed, here are some nice instructions.
sudo certbot --apache -d it.dpb.one
Once you have the cert setup, you will notice another file is created inside your sites-available location. Everything else is done for you at this point. Now you can access the website from the remote world. You can’t, however, access SSH remotely. This is a good protection. But, but, Random, I want to access my home lab via SSH.
Setting up SSH Jump Box
So, you want to be able to type ssh and your site’s name and go straight to your home lab. Here is what I have to say: don’t. Years of experience tell me this is a bad idea.
However, what you can do is SSH into the Oracle Cloud, then SSH into the Tailscale box. Your Oracle Cloud is your jump box now. It’s best to keep it that way for security. What’s amazing about Tailscale is if you have it on your PC, you can access your machines via the Tailscale IP address with SSH. If you don’t, just use the Oracle Cloud’s SSH.
Warnings
A CG-NATed home lab is much slower than a regular home lab. The reason why is you are going in and out of that Tailscale network over and over again. It slows things down. It gets much slower when you add the SSL. The first time you connect to it, it takes a good 10 seconds to load because it’s grabbing all of that information. So don’t use it for the following:
- Gaming
- Plex/Jellyfin
- audiobooks
All of those take up large amounts of data. It’s not worth the effort. Your Plex will work locally inside your home network. You may need a DHCP box of some sort; other than that, you are good to go.
What can we learn as a person
While cutting my grass, I noticed that the grass was growing through the asphalt. A little dandelion. I like these flowers, as they make great tea. It was growing through that hard rock. While the others were growing in the grass. This little weed grew and became a beautiful yellow flower. In the plant world, if it didn’t make it through the rock, it would have died. Often times we see these “weeds” growing through these rocks, like concrete. I love seeing a dandelion growing through a concrete slab. It takes advantage of the cracks in the stone and shines.
Often times people face hardships and stop growing. These people don’t make it a habit to find a way around those hardships. Those systems that are in place to keep us locked down. For years women in the science community have been like this. They are beautiful dandelions who have brought so much to this world, but because of their physical gender, many of their discoveries were taken credit for by men. However, when you see one who gets the credit, they shine. The asphalt is the systems we are born into. The hardships we have to face. No one sees those hardships unless they look for them. It’s not part of human nature yet to look for them.
So, if you ever see someone and think, They should be farther along than they are, what hardships did they face, and how much of a dandelion are they? The same is true for those who are shining. There is a story behind everyone, and to explore that story is to find connection and community. With community we can find real joy. We are social creatures, and we all have our own unique stories. Maybe it’s time for us to share more of it with each other.