For those that have encountered me enough times “LaneCloud” will usually come up with at least a passive mention. Some assume it’s just some kind of inside joke. Some think it’s some sort of homelab. My favorite description comes from a former colleague and friend who described it along the lines of “The strangest experiment I’ve ever seen.”

There’s some truth in all those conclusions. To be clear, the official tagline is:

The premiere not-for-hire cloud provider serving niche tech communities and driving innovation

Organic Origins

My participation in the Armbian project led me down the path of placing a 40 thread server in my rack at home. This REALLY sped up kernel compilation–which is something you do constantly if your hobby is oriented around the Armbian Build Framework to make Linux images for weird SBCs.

I built the server to be pretty flexible. It could run docker natively as well as run VMs with libvirt. Although that sounds trivial, it’s actually quite a pain, as docker is really aggressive in inserting itself at the end of the firewall chain. Custom rules have to be added for network traffic to flow properly on other bridge interfaces.

Serving Others

Is it really a server if there’s just one user? Obviously I’m not compiling kernels all day long, so it was very reasonable to want to give more people access to the speed I had. I provided a few very large build VMs to some people in the ARM SBC community so they could enjoy the same time-saving performance I did when building kernels. Additionally, a few GitHub Actions runners for Armbian were also added to share the pipeline load.

Late one afternoon a co-worker pinged me. They were working on an internal project, and their GitLab CI pipeline was giving weird errors. IT WAS DNS and our European counterparts that maintain those runners would unavailable until the next business day. There’s nothing worse than being in the flow and then being obstructed by somebody else’s IT problem.

Never to leave a comrade behind when they’re down, I took action. I configured a Gitlab Runner agent with DIND on my big iron, registered it to our GitLab server, added some labels, and assigned it to his project.

“Try Now.”

Productivity resumed…and all those threads really made the pipeline much faster. We kept the runner bound to the project indefinitely.

Increasing Criticality

True to Cloud TCO rhetoric, you will occasionally encounter hardware problems when running your own hardware. My experience is no exception. My OS drive was a first gen SSD which had been fine, but as this server’s responsibilities had increased, something was occasionally causing the SSD firmware to glitch and cause the server to misbehave.

Glitches like that are quite insightful. They remind you that your are providing services that other people rely on. The DM’s come fast. “I can’t reach my VM.” “Is the server down?” “The CI Runner is hung.” “There’s something wrong with your network.”

After one such issue resolution I jokingly replied: “The issue has been resolved. Sorry for the inconvenience. Thank you for your patience and thanks for choosing LaneCloud.”

Black Friday Deals

It doesn’t take a QuinnyPig hot-take to figure out that some customers don’t need the level of service and associated costs that come with using a top-tier hyperscaler. You’re likely aware of the next tier down–Linode, Digital Ocean, Vultr, Hetzner, etc. …but you can keep going down and start to find specialty hosting companies, and indie operations.

Maybe you’ve heard of LowEndBox, a fairly organized index of low-cost hosting options. Now we’re getting close with LowEndBox to what we’re seeking. We’ve left the business district and gone to the commercial district, but still entering through the store-front.

It’s time to go to the flea market. The LowEndTalk forums are the definitive flea market for cloud services, colocation, and hosting. Like a true flea market, there’s strange etiquette, questionable vendors, reputable vendors, amazing deals, haggling, and plenty of buyer-be-ware scenarios. There’s a lot of overlap between LowEndBox and LowEndTalk vendors, but the LowEndTalk community gives you a lot of perspective–including realizing there’s plenty of organizations still happily running LAMP stacks and simple services.

Between Black Friday and March Madness deals where you pre-pay for 1 or 2 years of hosting for pennies on the dollar, I found myself with an inventory at least 10 VPS instances distributed across the globe. The compute capacity of LaneCloud was expanding.

Branding Spiral

By this point, the “LaneCloud” references were common Slack jokes, and we’re in peak covid lockdown so I decided it was time seek out a professional and obtain an official Logo. Fiverr.com to the rescue.

I found an artist that specialized in minimalist 2d logos and enlisted their services. They gave me 4 choices… I selected 2 of them. One which I thought was more practical and I consider to be the official logo and another one that was a little more retro and technical which I opted to keep as an unofficial logo for backend resources or testing.

“Would you also like the social media kit?” …uh sure.

The results were complete and definitely worth the modest rate the artist charged. I had PNGS, SVGS, EPS as well as Adobe Illustrator original files. The social media kit educated me on the existence of custom headers and footers you can apply to LinkedIn, Xitter, and Facebook. I started feathering some of these new graphics into my profiles.

I had such a good experience, I wanted to see what else I could get on Fiverr.
Well, If you write your own script–with a little bit of luck you can hire a professional actor for $60 and get it back and 4 hours.

Since I had invested in branding and marketing, it felt prudent to commit to the name and purchase some domains. I secured lanecloud.cloud as my official public-facing domain. The sharks at afternic, GoDaddy’s domain squatting division, rejected my generous $200 counter-offer for lanecloud.com. They periodically send insulting emails to see if that will motivate me to buy.

Hi Lane,

Have you given up on this project? Is it safe to assume that you are no longer interested?

Let me know if you would like to make an offer.

I always make the same offer and they don’t respond.

Being lanecloud.com-less and having $200 remaining in my “marketing budget,” I clearly needed to solve this problem with swag. StickerMule is absolutely amazing. Their UI is easy, and their design team responds back quickly with proofs. They’ll gladly make kerning adjustments and other reasonable modifications without additional fees. Armed with die cut LaneCloud stickers, static clings, and even branded packing tape, it was time define a product.

Product Vision Attempt

I found myself with a problem in search of a problem. I had all these cheap VPSs doing nothing–“Idlers” in LowEndTalk parlance. Servers with nobody to serve is a sad problem.

I concluded that meshing my pool of random VPS nodes into a unified Nomad Cluster would be a great way to leverage the resources. That solved my first problem. The answer to my second problem was revealed through the process of writing my fiverr commercial.

Pilot your new applications in LaneCloud

My general idea evolved into this basic thinking:

As an SRE-oriented person, I felt like it was still tragically too common that indie developers are still spinning up VPS instances and manually operating their apps for POCs and pilots. IMHO this means they’re missing out on a good metrics and alerting solution, good log aggregation and dashboard solution, and a sane solutions for secrets management. I felt like I could package up an opinionated solution that made sure developers had these things challenges solved…and strongly encouraged them to use it.

Clearly, this was somewhat inspired by Heroku, but I wanted this to be more container oriented–perhaps even exposing some aspects of Nomad to the users as well. The boundaries of LaneCloud would be crisp. Ephemeral service deployment, with monitoring, logging, and secrets management provided. For any other persistent storage, the customer would need to use a cloud solution such as object storage or a managed database.

The purpose of LaneCloud was to provide a better starting point for developers that want to build an app and take it through a pilot stage. Besides the observability and secrets features, LaneCloud would be able to provide moderate scaling across it’s pool of inexpensive compute resources. This allows a project to grow organically for a longer period of time, before needing to shift to a higher SLA cloud provider, or needing to rush and refactor their stack away from a single ec2 or VPS instance.

Following the same formula, I realized in addition to developers, LaneCloud could help hobbyist run popular “self-hosted” genre of applications such as the ones packaged on linuxserver.io. Having two viable use cases with the same base solution gave me confidence and a clear-enough vision that I could start planning.

Architecture Brainstorm

I started sketching down all the core infrastructure needs and tasks I’d ultimately need to tackle. It consumed several pages in my field notes and decided it was worth hanging onto in a more legible medium. I typed it up in a document as a basic “How to build a cloud” framework.

In reality, it’s just things you need to consider for any “production” greenfield environment. Even with managed services, these things still matter.

Here’s the raw headings from the outline:

My “how to build a cloud” rough outline

## Deploy an Inventory IPAM tool

## Define a basic taxonomy

## Make an out-of-band control Plane

## Build Monitoring Stack

## Internal Network

## Documentation and Version Control System

## Choose a secrets manager

## Solve for DNS

## Plan for User Management and Access Control

## Pick a task scheduler

## Persistent Storage Plan

## Worker Instances

## Service Discovery

## Ingress Control

## Edge Security

## Artifact Management

Home Base Required

Although the core model behind LaneCloud was distributed low-cost VPS instances, I intended to treat them like spot instances and assume a lack of reliability. I’d need a reliable hub to operate my nomad control plane, coordinate mesh networking, etc.

As this central location idea was brewing, I had summer with several power outages. On several occasions I found myself sprinting to plug in the generator before my UPS ran out. I was seldom fast enough.

Familiar messages would appear: “I can’t reach my VM.” “Is the server down?” “The CI Runner is hung.”

Signs were pointing me into a single direction. It was becoming evident that my server needed to go into colocation.

Texas Hospitality

I got a few quotes locally, but they weren’t quite in the price range I was looking for.
My search expanded to posting on LowEndTalk for more hosting options in my area. Not to many leads, but Josh from Data Ideas replied to my post and invited me to “come down to Texas.”

I hadn’t considered Texas. I’m spoiled with my sub 10ms ping times from Richmond to Ashburn–how could I ever consider something 40ms away? I began considering that I would live and popped onto the Data Ideas Discord server to learn more about this small provider.

Data Ideas is a small business that specializes in providing Colocation, VPS, and even RPI hosting to tech-enthusiasts like me. This means they’re much more willing to take a customer like my seriously, help get my equipment online, deal with any irregular non-enterprise hardware I might have, and just generally be very accommodating. The pricing was also fantastic.

I began working out an arrangement to send 2 servers, a switch, a router and an Odroid HC4. Sending an ARM powered driver-toaster is a great example of that irregular hardware case.

Before finalizing my arrangements, an existing customer on the Data Ideas Discord pinged me. He was wanting to expand his footprint and had a proposal for me to split 10U, a lot of network bandwidth and a generous amount of IPv4 addresses. I probed a little more to make sure I wasn’t setting myself up for trouble. The guy wasn’t doing anything nefarious with his half the resources, was very talented at BGP routing, and generally was a good match.

That’s how I met my rackmate, and how I also invented the word rackmate.

I started boxing up my iron, and thanks to Pirate Ship, LaneCloud went to Texas.

large box in truck bed sealed with lanecloud branded tape combined server resources in rack

Control Plane V1

Although the long-term vision was certainly containers, I felt it was important to be able to provision VMs gracefully as a first step. My need was legitimate, but I was partially inspired by the fact that many VPS providers in the LowEndTalk space were using primitive solutions. I wanted to demonstrate a “simplified” approach using modern tooling.

The provisioning tooling for LaneCloud Alpha could easily be it’s own blog post. I’ll just enumerate some highlights:

  • automation is driven by Ansible
  • IP is assigned from Netbox
  • VM scheduled by Nomad’s qemu driver
  • custom cloud-init config generated by Ansible and Nomad template
  • additional user customization available as a cloud-init include stored in consul
  • Public DNS entry updated via Ansible module for dns-lexicon
  • VM fully provisioned and available on internet with public IP in under 2 minutes.

Check out my LaneCloud snippets repo for more details.

Re-thinking Objectives

As time has progressed, my intentions around LaneCloud have changed a few times.

The lack of native ARM runners with GitHub Actions (now resolved) inspired me to pursue ARM bare-metal hosting using Raxda Rock 5B boards. Sadly I underestimated how long until the RK3588 would actually become viable to reliable run VM’s and containers. I should certainly know better by now.

Fast forward 2 years and the RK3588 is generally viable for such a use, but as time progressed I realized this wasn’t a path I wanted to go down. Low-cost hosting is a race to the bottom. Observing all the dialog on LowEndTalk made me realize that public customers in this space are brutal, prone to fraud, and expect a lot for a little. This was not the direction I wanted to go.

I’ve also began reconsidering my entire nomad-based architecture. It’s clever, but really isn’t made for persistent VMs. It’s difficult to manage console connections and maintain state. Between the recent HashiCorp License Changes and Kubernetes tooling at critical mass adoption, I’m ready to switch to Kubernetes and also use KubeVirt for any VM needs. Maybe one-day I’ll even get to MicroVMs.

What is probably next

As far as getting LaneCloud into the marketplace, I’m not sure I’m really going to pursue anything directly right now. It’s probably best at doing what it already does, which is serve the needs of myself, friends, and some of the tech community. I like the idea a “core services” platform-engineering style offering that I could make available to some folks. Perhaps operating as a co-op to let a small group folks take advantage of pre-configured tooling and resources is the most practical path.

Regarding the architecture, it’s definitely time to replace my Hashistack components with Kubernetes. As previously mentioned, it’s reached a state where I’m sold on it. The operator model is great, and there’s plenty of ways for me to easily integrate things. I really hope I can make it to MicroVMs.

So what is LaneCloud Really

Mainly LaneCloud is a way to keep me grounded in technology and remind me what it means to run things in the wild. With my $dayjob in consulting, I operate in a lot of spaces, but don’t always get to observe them long term. It helps me maintain several perspectives, including why most organizations should just use the cloud instead of running their own stuff.

I do enjoy having all the resource capability at a fixed cost. That’s not a financial argument tho, LaneCloud is definitely still a loss-leader.

Many may still just view it as a glorified homelab that escaped the lab, but I much prefer to view it as the strangest experiment my friend has ever seen.