This blog post is going to be super sloppy. Human sloppy. I just felt like I should at least log something.

Okay fine, I have to try it

All my nerd feeds are talking about it… It’s super intriguing. The youtubers.. Naturally anybody that has a tech job in the US also needs to level up on AI as much as possible. So okay fine. I have to try it.

I’m cheap

I’ve been limping at home these days just fine using the Google Gemini plus that comes with my basic Google One account. It’s pretty good, but I have just been using the chat box. I also can run some local LLMs on my M3 macbook pro. That was fun last year… but a lot has changed.

Guardrails

I’ll deploy a VM on one of my more isolated VLANS and kind of go from there.

preflight

Naturally my VM tooling I’ve made in the past might need some attention. I’m going to install the gemini-cli client to help give me a boost with any preflight setup to assure I can live my values a bit.

installing gemini

Naturally i’m annoyed because it’s NPM based. Whatever.. brew install gemini-cli. Don’t forget to start a new terminal session afterwards. I haven’t needed the cli so far… but let me assure you that the normal chat session is helping me do my homework at a rapid pace. It took it a bit to realize openclaw’s config format had changed, but corrected itself quickly.

deploying VM

My existing scripts from my armlab naturally needed some very light refreshing. Okay I have a debian-13 VM provisioned now. It’s running on a VLAN I use for github runners, and has isolation similar to a guest network.

VM setup

Since I’m just testing, I installed a few basic things

PACKAGES="git curl jq podman-docker"
apt update && apt upgrade -y
apt install -y ${PACKAGES}

My hope was since podman-docker can run rootless, we could just give openclaw docker for it to extend its capabilities, and let it run in a standard user account.

Womp womp.. the lazy curl mysite | bash installer needs sudo.. I’ll try harder later. It can have sudo for now.

Openclaw setup

As mentioned I lazy-bombed with the easy installer and just mashed buttons.

I skipped the model setup since it was unclear how to manage using local models.

As far as interacting with openclaw, i just used openclaw tui to start with

lmstudio

I really wanted to just try melting my macbook before melting my wallet. Unfortunately it only has 18GB of RAM.. but I tried anyway with some smaller models.

I updated my network firewall to permit traffic from Openclaw’s VLAN to the lmstudio server port on my macbook. In lmstudio, there wasn’t much work, just a flag, to tell the server to listen on the LAN instead of 127.0.0.1.

To get openclaw to talk to LMstudio. I ended up with a config like this

{
  "models": {
    "providers": {
      "lmstudio": {
        "baseUrl": "http://MACBOOK_IP:1234/v1",
        "apiKey": "lm-studio",
        "api": "openai-responses",
        "models": [
          {
            "id": "qwen/qwen3-4b-thinking-2507",
            "name": "qwen3",
            "reasoning": false,
            "input": ["text"],
            "cost": {
              "input": 0,
              "output": 0,
              "cacheRead": 0,
              "cacheWrite": 0
            },
            "contextWindow": 16000,
            "maxTokens": 4096
          }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": {
        "primary": "lmstudio/qwen3"
      },
      "workspace": "/home/lane/.openclaw/workspace",
      "maxConcurrent": 4,
      "subagents": {
        "maxConcurrent": 8
      }
    }
  }
}

I tried different context window tweaks.. Openclaw requires min 16000. I enabled some experimental quantization setting on the server side to help compress, but ultimately after not much running… lmstudio’s server would just fall-over and unload the model. I’d get maybe 5 iterations for it would die. Maybe we’ll try this again later.

throwing money at it

Based on what I see on reddit, I’m probably not throwing enough money at it, but I’m going to try the synthetic.new $20/mo plan. I’m too afraid of getting burned on TOS, or other rate limiting to try the oauth with Gemini. I get too much value there now.

synthetic.new setup

I created an account.

I went to billing.

J/K THEY’RE OUT OF CAPACITY JOIN THE WAITING LIST.

THE END

:facepalm:

Trying again

All-in with Openrouter.ai

Gemini led me to Openrouter.ai as another practical model provider for Openclaw.

I’ll say the auto model router feature is pretty cool. I burned about $14 worth of Tokens arguing with openclaw claw to get things setting properly including memory embedding, and the perplexity search capabilities in Openrouter. It also helped me get a slackbot integration configured properly.

All the AI kept trying to patch the config file with an imaginary --json argument. That took a while to unlearn.

There was also cruft in .openclaw/agents/main/agent/auth-profiles.json from openclaw configure that I had to remove.

Anyway I’ll share a working config file for using Openrouter for LLM, embedding, and for search.

Note. I had to set API vars keys in ./openclaw/.env that was the sanest way to do it. Amusingly even tho I was using Openrouter for everything… I had to set an anthropic and openapi key as well just for some of the things to work correctly through open router. That’s not a great explanation…. but it’s a breadcrumb for ya!

.openclaw/.env file

OPENAI_API_KEY=YOUR_OPEN_ROUTER_KEY
OPENAI_API_BASE=https://openrouter.ai/api/v1
OPENROUTER_API_KEY=YOUR_OPEN_ROUTER_KEY
ANTHROPIC_API_KEY=YOUR_OPEN_ROUTER_KEY

.openclaw/openclaw.json config. There’s cruft, but hopefully helpful. BTW I got burned pulling json snippets from Openclaw’s documentation website, because their documentation syntax formatting for json removes the quotes making it invalid json you can’t paste…. yet again.. Thanks gemini

{
  "meta": {
    "lastTouchedVersion": "2026.2.6-3",
    "lastTouchedAt": "2026-02-08T22:08:22.838Z"
  },
  "wizard": {
    "lastRunAt": "2026-02-08T18:05:00.747Z",
    "lastRunVersion": "2026.2.6-3",
    "lastRunCommand": "configure",
    "lastRunMode": "local"
  },
  "auth": {
    "profiles": {
      "openrouter:default": {
        "provider": "openrouter",
        "mode": "api_key"
      }
    }
  },
  "models": {
    "providers": {
      "openrouter": {
        "baseUrl": "https://openrouter.ai/api/v1",
        "apiKey": "YOUR_OPEN_ROUTER_KEY",
        "api": "openai-responses",
        "models": [
          {
            "id": "deepseek/deepseek-r1",
            "name": "DeepSeek R1",
            "reasoning": false,
            "input": ["text"],
            "cost": {
              "input": 0,
              "output": 0,
              "cacheRead": 0,
              "cacheWrite": 0
            },
            "contextWindow": 164000,
            "maxTokens": 8192
          },
          {
            "id": "meta-llama/llama-3.3-70b-instruct",
            "name": "Llama 3.3 70B",
            "reasoning": false,
            "input": ["text"],
            "cost": {
              "input": 0,
              "output": 0,
              "cacheRead": 0,
              "cacheWrite": 0
            },
            "contextWindow": 131072,
            "maxTokens": 8192
          },
          {
            "id": "google/gemini-2.0-flash-exp:free",
            "name": "Gemini 2.0 Flash (Free)",
            "reasoning": false,
            "input": ["text"],
            "cost": {
              "input": 0,
              "output": 0,
              "cacheRead": 0,
              "cacheWrite": 0
            },
            "contextWindow": 1048576,
            "maxTokens": 8192
          }
        ]
      }
    }
  },
  "agents": {
    "defaults": {
      "model": {
        "primary": "openrouter/auto",
        "fallbacks": [
          "openrouter/meta-llama/llama-3.3-70b-instruct",
          "openrouter/google/gemini-2.0-flash-exp:free"
        ]
      },
      "models": {
        "openrouter/auto": {
          "alias": "OpenRouter"
        }
      },
      "memorySearch": {
        "provider": "openai",
        "remote": {
          "baseUrl": "https://openrouter.ai/api/v1",
          "apiKey": "YOUR_OPEN_ROUTER_KEY"
        },
        "model": "text-embedding-3-small"
      },
      "compaction": {
        "mode": "safeguard"
      },
      "maxConcurrent": 4,
      "subagents": {
        "maxConcurrent": 8
      }
    }
  },
  "tools": {
    "web": {
      "search": {
        "enabled": true,
        "provider": "perplexity",
        "perplexity": {
          "model": "perplexity/sonar-pro"
        }
      }
    }
  },
  "messages": {
    "ackReactionScope": "group-mentions"
  },
  "commands": {
    "native": "auto",
    "nativeSkills": "auto"
  },
  "channels": {
    "slack": {
      "mode": "socket",
      "webhookPath": "/slack/events",
      "enabled": true,
      "botToken": "botToken",
      "appToken": "appToken",
      "userTokenReadOnly": true,
      "requireMention": true,
      "groupPolicy": "allowlist",
      "channels": {
        "SOMECHANNEL": {
          "enabled": true,
          "allow": true
        }
      }
    }
  },
  "gateway": {
    "port": 18789,
    "mode": "local",
    "bind": "lan",
    "auth": {
      "mode": "token",
      "token": "MYGATEWAYTOKEN"
    },
    "tailscale": {
      "mode": "off",
      "resetOnExit": false
    }
  },
  "plugins": {
    "entries": {
      "slack": {
        "enabled": true
      }
    }
  }
}

The Openclaw Gateway Dashboard

I feel a little silly, because for much of my experimentation I just had 2 terminals panes. One with openclaw tui and the other with openclaw logs --follow. I should have gone through the effort to use the dashboard sooner, but it was configured for localhost only, and I didn’t want to port forward for some reason. I enabled it on the LAN, and then just ended up reverse proxying as an ExternalService via an ingress on my kubernetes cluster. It was pretty clean. I had a little trouble figuring out how to get the session token going and validating my device from the cli, but afterwards.. the UI at least seems cool. I didn’t try editing the configuration from it, but you can. Using the chat was more pleasant from a rendering perspective.

Poke around the UI! You can see all the pieces…. then you can ask question.

Doing stuff

I yelled at it in slack and got the bot to do some simple python and bash scripts, Run docker containers. Search the web. It was okay. I just couldn’t get it to really do any long standing work in the background as I had sort of thought would happen. Background activities don’t seem to follow-up right now without continue a conversation.. I was hoping I’d see more automatic messages in slack that weren’t direct responses. It could be a configuration issue of course.

Cron jobs have the same thing… They don’t update me unless I ask about them.

Moltbook

I tried twice to join moltbook. Openclaw did an okay job and researching and trying to apply. Although the first time time I think it hallucinated instead of doing the actual work. The second time, I got an actual link and posted on Xitter….. BUT the verification didn’t work, so I’m not a able contribute to that waste of energy yet….. It’s funny tho… I was looking for things to for Openclaw to do.. and was struggling to find something simple…. Moltbook suddenly made sense.

Takeways

The real winner here is the basic “plus” tier of Google Gemini. It continues to be really good… It helped me solve configuration issues for openclaw, generate some k8s manifests I needed, and research all sorts of things–at a low cost.

To get real value out of openclaw I think I’d need to try a lot harder… BUT it seems like you also need to really double-down on it. Like the guy that used the $200/mo OpenAI package as the “best value.” I believe them to, but I’m not in the mood to go all-in on this. I can’t pay $200/mo to advance hobby/science projects… it doesn’t make sense. I’m better off just continuing to use the basic AI accelerators already available to me…. And maybe one day….. I won’t feel this existential pressure to do technology in a hurry, and I can just go slow again…maybe.

So I spent over half my weekend and $25 in tokens tinkering with this, and if I’m smart… I’ll shut down the VM and let a different tier of technologist play with it. Basically I setup a slack proxy to let an LLM agent run a docker container.

Maybe next weekend I’ll go in the garage.

The blog’s new home falls under my “legacy” static hosting model. Aka content is stored in a share on my synology, I then expose the content to an NFS mounted folder from an Nginx pod. Traffic is back-hauled through haproxy on my Hetzner VPS, through Wireguard to my “external” ingress-nginx controller, which is configured to support the PROXY protocol.

When the blog was hosted in Lanecloud, it was stored in Minio, and reverse proxied via Nginx. At the time, I had spent a ton of time with the Nginx configuration so that it hid any hint of Minio. It was probably unnecessary, but it was a fun exercise.

Rclone supports SMB, which really simplified moving the blog. I just changed my rclone target in my blog script to use an SMB target on my Synology instead of the rclone job to Minio. Pretty rad.

Shout outs!

• My rackmate, @ssh – thanks for getting me into this and being an awesome network engineer.
• Data Center Dude Josh – thanks for all the service, support and friendship.
• Britt – thanks for your support, enthusiasm, and hardware!
• Lanecloud customers – thanks for being part of the project!

So why now?

A few factors… Mainly it’s been time for a while. I’ve been slowly working on v2, but it’s a lot of work. Building something that is serious and meant for other people to use is a large undertaking. It kind of stopped being fun, and all the prerequisite activities just keep taking me away from working on the things I’d rather work on. Sounds similar to TCO and Opportunity cost narratives we use to justify going to the cloud doesn’t it?

Another reason I had kept going was that we had an opportunity to host a big shiny ARM server. Unfortunately after months and months of correspondence, and 2 different vendors, we just can’t seem to get it over the line. It’s not worth waiting and keeping the lights on for it anymore.

Where am I gonna go?

Most of my footprint will probably move to home and Friendnet. NetCup ARM VPS’s are looking really appealing. I might even setup a full k8s cluster out there at some point. I also like HostHatch.

When are things going offline?

I hope to have things fully powered down Mid December 2025.

However, I still wanted to keep my default neovim for when I really wanted to get in and out of an editor fast. I decided to launch LazyVim by typing lvim and leave my standard vim and nvim.

Fortunately neovim has a concept called NVIM_APPNAME that lets you load the app from an alternate configuration directory.

Pretty simple stuff.. Checkout lazyvim into ~/.config/lvim and make a command alias for lvim setting the NVIM_APPNAME variable

Here’s the key snippet from my lazyvim github gist.

git clone https://github.com/LazyVim/starter ~/.config/lvim
### put this in your shell profile
### you need neovim 0.9.newish for the NVIM_APPNAME param to work..
alias lvim="NVIM_APPNAME=lvim nvim"

So this is a blog post I started in April… Now it’s nearly the end of September. I’ll keep it short and sweet just so I can close this out.

Contributing to mature open source projects isn’t a fast process. It requires diligence and patience on your part, and you may be surprised how much work is involved beyond the simple bug fix you actually implemented.

The important thing to remember is that what can feel like turnstiles, complexity, and unnecessary prerequisites are really there to protect the sanity of the under-compensated developers of the project. In the case of the contribution that inspired this post, it teaches you a lot about what good looks like for testing, changelogs, and managing releases.

At this point–about 18 months ago, I ran into a bug with the Ansible Netbox inventory plugin. I run my netbox server under subpath rather than at the root path. This IS a supported configuration with netbox itself, its just less supported. (even now the swagger UI for the netbox API generates the wrong urls). After an update, how the inventory plugin derived the api paths from OpenAPI broke my ability to use it. Borrowing from inspiration in one of the other modules in the collection, I was able to implement a simple fix in an evening and resume doing ansible+netbox things.

As mentioned the fix wasn’t too bad and my buddy encouraged me to upstream it. I remember seeing that open a PR was a bit involved, so I procrastinated–for about a year. Honestly I expected someone else would have probably resolved the issue already, but that wasn’t the case. I conceded, and put on my good open source citizen hat.

This was the part of the blog that was supposed to highlight in detail my colorful weekend-long journey to get my PR in the correct place, but I don’t remember it quite as well now. I’m just gonna make a simple list that might paint the picture:

• Find correct issue references
• Set up local netbox and local test environment
• Get test tools to pass in local test environment
• Hunt down issue references
• Figure out correct way to create a changelog fragment and commit
• Bang on my PR until the required tests in the pipeline passed

Each of those pieces proved to be a bit time consuming for me, but I got through it all bit by bit.

I’m happy to say my PR to the ansible-netbox inventory plugin was accepted and merged. As of this writing, there hasn’t been a new release, so you’ll have to install from the master branch.

I’m looking forward to seeing my fix listed in the next release changelog. It was a bit of a slow process, but that’s one of many things that makes open source successful. It’s great all the tooling is in place for thorough testing so we can confidently know my change doesn’t break things for others.

day 002 20250306

• made this blog post (baby steps okay?)
• logged into one of my ingress boxes and looked at logs. 99% bot traffic. no surprise.
• one of my lanecloud hypervisor hosts has 550 days of uptime… that’s something I guess.
• started on an ansible role to add custom startup scripts to clammy-ng
• got distracted with some hugo theme tweaks that I couldn’t quite make work

day 003 20250308

Ended up being kind of an Armbian Day

• submitted the removal of a RK3568 bugfix patch since it got merged into LTS kernels and was conflicting on build.
• Ran a bunch of glmark2 and sbcbench marks on boards comparing kernels. No real conclusions other than I think an RK3588 DMA patch someone added had a slight performance regression.

day 004 20250309

• My Synology backups that goto my eSata drive have been dead for a month. Got the drive online… now says its full. Really need to add back push notifications for that. More work to be done to get this resolved.

• Made a new playbook for my Provision KVM guest ansible role. It just uses vars_prompt to make it easy to spin up a few VM in a hurry. I like it.

• Wasted a lot of time trying to debug performance issues with my librespeed instance. Some of the performance problems are just VM performance sucks on Synology VMs, even when using appropriate acceleration. Other parts of it, is that it seems to be a known thing that nginx or ingress-nginx also seems to introduce some performance issues when it proxies librespeed. I managed to do some tuning that made it better, but it’s still unable to hit wirespeed on gigabit. I’m extra amused that librespeed is more performant on a VM running on my Rock 5B than it is running on my Synology DS1821+

  Here’s some settings for the service ingress that sort of helped with nginx.. but needs more TBH

  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "21M"
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "50"
  

day 005 20250310

Just kind of did my rounds and looked at some things.

• Verified Synology backups are happy now that I cleaned up my external drive
• Updated Netbox
  • deleted retired LET deal VMs
  • updated renewal date field on remaining LET deal VMs
• Verified db dumps and borgmatic backups are running on my Netbox server
• Looked at some healthchecks.io helm charts. Would be nice to deploy that in the future and integrate with borgmatic

day 006 20250311

Pretty beat today so just did some fiddling. I fired up some temporary debian pods on my synology moncluster VM, and one on my Rock 5B k3s cluster. I ran quick yabs.sh benchmarks on them. The pod on the 6 core ARM VM was faster in single and multithreaded than the 4 core VM on my Synology 1821+. Between this and the librespeed benchmarks, I think I’ve convinced myself the monocluster is moving off the Synology and onto a Rock 5B.

WTF is the monocluster

I have 2 k3s deployments at home. The first one is the “armlab” cluster. Its 6 VMs running on 3 Rock 5B boards. It’s meant to be torn down and recreated a lot. It has an HA control plane via kube-vip. It has metallb, cillium, and some basic service deployed via flux. I really wanted to figure out using BGP in the HA cluster before deploying to it, but I decided I should do that later and first just focus on getting my nomad cluster.

The monocluster was my KISS approach to just focus on a “best practices” IAC setup for k8s with fluxcd…and focus on bootstrapping all the extra things like external-dns, external-secrets, etc. One detail is that I was also trying to design it as a solution I could use to run on a single node VPS.. It took me a while to solve for getting nginx to listen on port 80,443 node ports but I did it. I’ve been redeploying it like crazy as well while I add functionality, but it’s time to set up it’s final (temporary) home.. and move stuff over it to it.

day 007 20250313

I found one of my rock-5b boards that I used as an armbian build node a year or so ago with the Googalator kernel. It has a nice NVME in it. It’s a great fit for the home of my monocluster. So I decided to build an image and get it installed… Naturally I hit some unexpected curveballs

• My other Synology cache drive alerted that it had 1% lifespan…… I had to replace it.. Now I’m running my cache on a pair of $20 nvmes… RIP crucial P3
• I booted edge kernel 6.14.0-rc4… it was weird… the nvme was intermittent. And eventually disappeared… I went back and forth a few times and then decided to build -current aka LTS kernel 6.12.y.
• worked better.. but something with armbian-install or uboot on SPI flash isn’t booting from NVME.. i don’t feel like debugging.
• just using sdcard for bootloader for now. seems to work.

day 008 20250314

I’m surprised about my burst of productivity given I’ve been on like 4 hours of sleep today.

• The nvme boot problem seem to partially have somethign to do with the nvme drive I was using.. it would init fine on a reboot, but not on a cold boot. Storage controllers and SBCs can be picky. I pulled a 512 Skyhinx out of an external enclosure and now it’s happy.
• I re-installed monocluster as bare metal k3s on the above mentioned Rock 5B.
• I also tested libre speed… looks good!

root@ronny-1:~# librespeed-cli --skip-cert-verify --local-json librespeed.json --concurrent 2 --server 1
Using local JSON server list: librespeed.json
Selected server: mtest [librespeed.mtest]
You're testing from: {"processedString":"172.17.20.115 - private IPv4 access","rawIspInfo":""}
Ping: 0.64 ms	Jitter: 0.38 ms
Download rate:	925.08 Mbps
Upload rate:	953.28 Mbps

day 009 20250315

Time to start getting some actual workloads moved over…. or in my case.. test some workloads.

I focused on some basic nginx servers that are just directory indexes for data on an NFS mount. Was a great starting point.

I lost a lot of time trying to use app-template to do the things. It’s cool, but also extremely overkill and ended up being a bad fit for my use case.

I dumbed things down to unified manifest file for each server + service. I took advantage of flux’s ConfigMapGenerator to make my configuration hashed and cause pods to redeploy when I update. Works well.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web-static
resources:
  - ./namespace.yaml
  - ./nginx-linuxmirror.yaml
  - ./nginx-otherserver.yaml
configMapGenerator:
  - name: nginx-linuxmirror
    files:
      - default.conf=nginx-configs/default-linuxmirror.conf
  - name: nginx-default
    files:
      - default.conf=nginx-configs/default.conf

day 010 20250316

• moved the monocluster rock5b into my rack
• wired the fan on the heatsink to 3.3v for now
• ordered some Molex Picoblade 1.25mm connector crimps so that I can wire the fan to the actual fan header later
• naturally the switch port i used was wrong and i had to fix the PVID before I was pulling the properly assigned static DHCP address.

day 011 20250317

A rude awaking

Checked my email in the morning and saw an abuse report from Hetzner in my inbox. Turns out people were exploiting an old pastebin server I’d setup a long time ago for Armbian-related stuff. My hetzner box has haproxy doing TCP reverse proxying over a wireguard tunnel to my fabio load balancer for my nomad cluster. I stupidly have a mixture of public and internal services running on it. Anyway.. I terminated the service and ripped out the DNS records… Nobody was using it–including myself. Not how I wanted to start my week, but I good reminder to spend some more energy on my security posture. Looks like a bit of extra bandwidth got burned on my hetzner account as well. They changed policies from 20TB included a month to 1TB :( Boo…. but lets be real, 20TB of bandwidth a month in Ashburn for $5 a month is too good to be true.

Hopefully a better homelab day tomorrow.

day 012 20250319

My rackmate is working on a new core router. We’re upgrading hardware and going from vyos 1.3 to 1.4. I had to re-write my NAT rules. Also trying to use a dummy interface for my NAT interface since sometimes we have multiple WAN links.. Hopefully this will simplify things. Regardless I put the translation interface in an interface group. Hopefully that will make it easy to move things around in a pinch

day 013 20250324

Mostly poked around k9s and grafana today. Just wanted keep up the rhythm at least. I upgraded k3s on my armcluster. It was a minor update or 2 behind. Ansible did all the work of course.

day 014 20250406

Had some other stuff going on the past 2 weeks, but I haven’t given up.

I did my normal kernel build and update routine for my router. I spent a little bit of time trying to see if I could speed up the reconciliation time of my fluxcd deployment, but no solutions yet. I have a dependency chain that get held up waiting on external-secrets to reconcile back into a ready state. The annoying this is external-secrets leaves a ready state on any update I push.

day 015 20250412

Busy day. Changed from NVME cached LVM to ZFS on one of my lanecloud servers. It was quite a bit to unwind.. had to move VMs. make checklists.. move filesytems, backup stuff. etc. The box has 256 gigs of RAM so ZFS cheats performance pretty well on top of the underlying 4 spindles. I made my 2 NVME drives its own mirror pool. The 4 rust drives I configured in the ZFS raid10 equivalent:

#!/usr/bin/env bash
SDA="wwn-0x5000c50091843eae"
SDB="wwn-0x5000c50091906b31"
SDC="wwn-0x5000c50092509d76"
SDD="wwn-0x5000c500926006cc"
zpool create rustpool \
    -o ashift=12 \
    -o autotrim=on \
    -O compression=zstd \
    -O dedup=off \
    -O xattr=sa \
    -O acltype=posixacl \
    -O atime=on \
    -O relatime=on \
    -O sync=standard \
    mirror ${SDA} ${SDB} mirror ${SDC} ${SDD}

day 016 20250413

Around 18 months ago I ran into a bug with the Ansible Netbox inventory plugin when querying my netbox server that runs on a subpath. I made a local fix, but never upstreamed it. This weekend I decided to upstream it. Rebasing my patch just worked. That was cool. Doing the actual dance to properly submit the patch to GitHub was more involved. I was going to write a dedicated blog post on it, but maybe later.. or not at all. I’m not complaining at all, it’s just part of open source… and projects need good PR hygiene and process so that devs can not be beat down.

day 017 20250415

Woohoo my bugfix for the Ansible Netbox Inventory Plugin PR was accepted. FOSS Works! :P

I haven’t redeployed the VMs on the server that got the ZFS switch. My nomad tasks I stopped disappeared. And the particular VMs running were from my oldest iteration of the lanecloud IAC. TLDR; I have to bugfix the playbooks to get them to redeploy….. or I can follow through with switching to libvirt deployments. I opted to modernize… Let’s just say I’m moving slow.

day 018 20250505

Man.. really spacing this stuff out…. in my defense last week I was on vacation and I made sure to avoid computering.

Just a basic maintenance night to kind of get the juices flowing.. Some apt-updates and built some fresh 6.14 kernels for my RK3588 cluster

Apt update on my poor helios4 trixie box took forever… but that’s normal. The Helios 4 is still a champ!

day ??? 2025 Q2-Q3 update

Clearly I lost track of things.. Daily routines are hard for me I’m bursty in my projects by default.

I did check off several milestones… each with their own deep rabbit holes.

• I finally fully migrated my home things to my k3s monocluster
  • powered down my nomad cluster! RIP: i think it was up for at least 7 years
  • Forked a certmanager-webhook that used DNS lexicon.
    • Reworked the pipeline to build multiarch containers – quickly.
• Added unbound dns with blocklist support to clammy-ng
• Finally updated my 4 year old homeassistant install
  • lots of breaking config tweaks i had to catch up
  • also some fiddling with some tasmota devices that had reset
• Lanecloud Modernization Prep-work
  • Created an ansible role for VM registration in netbox
    • lots of rabbit holes again here
    • this allows me to completely decouple VM registration / configuration properties from the actual provisioning code

That was quite a bit of work… I think i’ll allow my next day to be day 50

I figured interacting with Hugo was a good warm-up for a workday evening. I always have to kind of re-remember how to interact with it. I’m also not very good at image processing. So lets try that again.

Here’s a pic of the James River when it flooded earlier in the month.

Good job me! Finally used the short codes correctly.

Projects

Will try to just keep this high-level so the list doesn’t look so insurmountable. It’s also silly that I have 2 discrete environments to worry about, but that’s just how it is.

Homecloud - migrate my Nomad services to Kubernetes

I’ve basically done the hard part. I’ve got a single node x86 variant of opinionated armlab k3s build.

Core attributes of the monocluster:

• k3s
• cilium
• fluxcd
• secrets manager w/ 1password integration
• ingress-nginx
• external-dns
• certmanager

Over the holidays, I spent a zillion hours trying to automate all the weird little bootstrapping steps, and solving for some of the sequencing ex. 1password setup before external-secrets etc.

Also really the biggest pain in migrating is my current Nomad setup using fabio ingresses several different domains, and I’m having to suck it up and migrate things to cloudflare DNS for sanity. I still have a few more zones to move.

Once that’s done, then it’s just a matter if redeploying my existing services.

LaneCloud – move my VM deployment to libvirt

I’ve settled on just using ansible libvirt as the most sustainable way for me to manage VMs. I need to migrate my existing lanecloud deployments using my homegrown nomad solution for running VMs over to just basic libvirt.

Lanecloud – personal Kubernetes cluster

I’ll stick with providing friends VPS, but I would like to just get a Kubernetes cluster out there that uses Cilium BGP.

Lanecloud – network shuffling

I need some more VLANs and better east-west filtering. Also want to deploy an instance of clammy-ng to sit between my VLANs and our core router. It will be much easier for me to manage NAT rules there than on our core router. We have multiple WAN links, and that means I have to duplicate NAT rules for each path.. It’s annoying.

Get motivated n stuff

Alright… I made this blog post. So great start, me. Good job. Also I kind of feel better remembering all the work I already did. I just need to hammer away at it. Hence the spirit of this whole #100daysofhomelab thing. Thanks TechnoTim and friends.

Disclaimer: point of this post is just to share a few quick hints.

Using cilium with multus

I’m just putting this here first because this was my biggest pain point.

Cilium has a flag called cni-exclusive that has to be disabled. If you don’t multus won’t work, and you’ll waste a lot of time. I eventually came across the hint on a discord server thread where Techno Tim had been beating on multus for a non-kubevirt use case.

Here’s the magic command in my multus install script that runs after installing cilium

  cilium config set cni-exclusive false

Quick Overview

So the whole thing with kubevirt is to be able to manage your VMs like pods—which it does quite well. However, in that spirit, it also wants to manage networking the same way. AKA: exposing services and do port-forwarding. This of course is fine for many use cases, and cilium plays nice with that.

Another However: you might want to treat your VM like a normal VM and put it on a VLAN and have it be fully routable / accessible. That is what having multus is all about: it lets you have additional CNI drivers on your kubernetes cluster so you can give your pods different types of interface–such as a bridged interface.

In order to achieve that objective, there’s several paths, but I focused on 2:

• macvtap – This one is new, has heavy development from the kubevirt team. It has some advantages, such as not requiring a bridge interface, it just makes a new mac and hangs it off an existing interface. I’m sure there’s other advantages, but a disadvantage is it doesn’t work for hair-pinning traffic if you need to reach the control host. I have some scripts for setting it up, and probably could have made it work if I had found my cilium fix sooner.
• bridge – This is as tried-and-true as it gets for linux virtual machines. You make a bridge, and new VMs attach to it. The bridge networking CNI is part of the CNI reference suite of plugins. I ended up settling on this approach at the end.

Reminder this was still just kind of a quick R&D exercise, so I cut some corners and quickly re-used some other tooling to get the job done.

Again the basic formula was:

• bridged networking
• k3s
• cilium – although I’m not really taking advantage of it in this approach
• multus
• kubevirt

Speedrun

This is just stepping through parts of my kubevirt install scripts folder. Remember: there’s no substitute for hours of your own suffering, but I hope there’s a few hints for you here.

k3s base setup

1. Create a bridge and put your primary interface on it. Normally I’d do this with netplan, and frankly would have the bridge interface on a separate VLAN from my management interface, but I just quickly did this with network manager and created bridge0. Your IP may change after you do this… hence do it first.
2. Install K3s and assure flannel and servicelb aren’t configured.
3. Install bridge cni plugins – The bridge is part of a greater bundle of reference plugins, I just filtered for bridge on the tar extraction
4. Copy kubeconfig from your cluster to your workstation
5. install cilium – I followed easymode for that

installing kubevirt

1. install the virt plugin via krew – this plugin provides additional CLI comforts for interacting with the VMs
2. install the kubevirt operator
3. install kubevirt crds
4. install kubevirt cdi - containerized data importer – Using CDI is a big part of doing business with kubevirt. It takes VM images and stores them in an OCI format that’s easier for kubevirt to deploy without a custom storage class. You’ll want to read about it.
5. enable network bindings plugin feature gate – this is a kubevirt feature gate that was easiest when applied post-install.
6. install multus – again: take note of the cilium setting cni-exclusive: false.

deploying VMs

1. create a bridge attachment definition for bridge0 – See this manifest – the VM deployment will reference it.
2. create a data volume – CDI will import this image and create a datavolume that the VM will use
3. create a virtual machine – my example uses an armbian/debian cloud-init image and attaches is to the bridge network
4. check your dhcp server and see if your VM is online.

useful commands

kubectl get dv
kubectl get vmi
kubectl get vm

It’s nice to watch these assets from within k9s. remember if you do esc : in k9s you can do manual queries for things like above.

Watching the cdi importer for creating the dv is great for troubleshooting.

Craving a new router solution

I was a Ubiquiti EdgeRouter fan for years. The series provided Amazing capability for the price, and the OS being based on a fork of Vyatta made it all that much more appealing.

However, in recent years my opinion began to sour as it’s become evident that priority on this product line is going way. Ubiquiti’s focus is clearly oriented around their USG platform. I don’t blame them for it. It’s a better market alignment for them. The EdgeRouter series isn’t a good fit for SMB’s and they don’t provide enough support for enterprise.

Despite them being amazing capability for the price, they’re still pretty pricey and their CPUs are slow. Most performance relies on hardware offload capabilities, so CPU oriented tasks like VPN solutions, and bonding can have an impact. (Disclosure: LACP performance when I upgraded to an ER-6P was fine.) A cheap ARM SBC like the NanoPi R5S could deliver the same performance for my use case.

My goal was retire my ER-6P and replace it with low-cost Armbian powered SBC. The core requirements were simple:

• ansible managed
• use mainline kernel
• support zone a zone firewall configuration

Approach

Use modern components when possible

Basically I wanted to avoid getting bogged down in managing long orchestration scripts whenever possible. When Vyatta/VyOS was made, we didn’t have netplan, networkd, frrouting, nftables, etc. Consequently, it has a ton of code to directly manage interfaces, bird, and iptables. I wanted to offload those burdens as much as possible.

Another part of off-loading those burdens, was to try to find as many off-the-shelf ansible roles as possible to manage those modern services. Shout-out to mrlesmithjr as he had flexible roles for many of the components I needed.

Configuration

Since I’m using ansible (the point of this project), this means managing the configuration in a sane vars structure. The biggest part of this was creating a structure that could describe most aspects of an interface and its networks in a single place. I could then just reference components of that again later in the configuration as needed.

I decided to use my foomuuri_networks object as my main source of truth.

In the example snippets below you can see how I referenced other attributes of the network later in the configuration

foomuuri_networks:
  test1:
    interface: test1
    description: test network 1
    zone: test1z
    address: "192.168.101.1"
    netmask: "255.255.255.0"
    address_summarized: "192.168.101.1/24"
    dhcp:
      start: 192.168.101.100
      end: 192.168.101.150

lan_netplan_configuration:
  vlans:
    test1:
      id: 1001
      link: bond0
      addresses:
        - "{{ foomuuri_networks.test1.address_summarized }}"

dnsmasq_dhcp_scopes:
  - interface: "{{ foomuuri_networks.test1.interface }}"
    netmask: "{{ foomuuri_networks.test1.netmask }}"
    start: "{{ foomuuri_networks.test1.dhcp.start }}"
    end: "{{ foomuuri_networks.test1.dhcp.end }}"
    lease_time: "4h"

Challenges

The firewall

Solving the firewall the way I wanted it to be solved was the biggest part of the project.
Most existing tools and scripts in the wild were interacting with iptables, and sometimes even required legacy mode. I really wanted something strictly nftables native if possible.

To further my pickiness–I also really really really wanted a zone firewall.

Firewalld

I had originally assumed firewalld was going to be my path since it’s a “zone firewall” and has ansible modules to support it. The reality is, its a host firewall and not a router firewall. To be a router firewall I need firewall policies that are oriented around traffic traverse between 2 interfaces, rather than traffic between an interface and the host. It is possible to do it with firewalld, but it requires writing lots of custom rules in XML and managing that. No thanks.

Foomuuri

Foomuuri is the salvation of this project. If you take away anything from this blog, it’s please check out Foomuuri. Timing wise I was really lucky. Foomuuri is quite new, originating around Feb 2023… and I happened to come across it a few months later where it had just been added into debian unstable packaging. You don’t need a debian package in reality. Foomuuri is an extremely elegant python script with minimal dependencies.

Foomuuri can function as a host firewall, and most importantly for this project… It can function as a router firewall.

Its configuration language is flexible, and easy to template. Like a good *nix tool, the configuration files can be broken up and arranged in anyway the user sees fit.

I made a foomuuri ansible role for managing it’s installation and configuration.

The code

It consists of the clammy-ng ansible project reference implementation repo and the clammy-ng ansible collection containing a filter plugin, roles such as my foomuuri role to implement the solution.

Components used

My primary components right now include:

• dnsmasq
• foomuuri
• frrouting
• inadyn
• netplan
• networkd
• nftables
• wireguard

Check it out!

See the readme for demos, screenshots, and notes about other quality of life packages and scripts provided.

PRs Welcome

This was targeted for my use case, but I think it’s a flexible start for others. Feel free to fork or PR enhancements. Share your crazy configurations to inspire others!

There’s some truth in all those conclusions. To be clear, the official tagline is:

The premiere not-for-hire cloud provider serving niche tech communities and driving innovation

Organic Origins

My participation in the Armbian project led me down the path of placing a 40 thread server in my rack at home. This REALLY sped up kernel compilation–which is something you do constantly if your hobby is oriented around the Armbian Build Framework to make Linux images for weird SBCs.

I built the server to be pretty flexible. It could run docker natively as well as run VMs with libvirt. Although that sounds trivial, it’s actually quite a pain, as docker is really aggressive in inserting itself at the end of the firewall chain. Custom rules have to be added for network traffic to flow properly on other bridge interfaces.

Serving Others

Is it really a server if there’s just one user? Obviously I’m not compiling kernels all day long, so it was very reasonable to want to give more people access to the speed I had. I provided a few very large build VMs to some people in the ARM SBC community so they could enjoy the same time-saving performance I did when building kernels. Additionally, a few GitHub Actions runners for Armbian were also added to share the pipeline load.

Late one afternoon a co-worker pinged me. They were working on an internal project, and their GitLab CI pipeline was giving weird errors. IT WAS DNS and our European counterparts that maintain those runners would be unavailable until the next business day. There’s nothing worse than being in the flow and then being obstructed by somebody else’s IT problem.

Never to leave a comrade behind when they’re down, I took action. I configured a Gitlab Runner agent with DIND on my big iron, registered it to our GitLab server, added some labels, and assigned it to his project.

“Try Now.”

Productivity resumed…and all those threads really made the pipeline much faster. We kept the runner bound to the project indefinitely.

Increasing Criticality

True to Cloud TCO rhetoric, you will occasionally encounter hardware problems when running your own hardware. My experience is no exception. My OS drive was a first gen SSD which had been fine, but as this server’s responsibilities had increased, something was occasionally causing the SSD firmware to glitch and cause the server to misbehave.

Glitches like that are quite insightful. They remind you that your are providing services that other people rely on. The DM’s come fast. “I can’t reach my VM.” “Is the server down?” “The CI Runner is hung.” “There’s something wrong with your network.”

After one such issue resolution I jokingly replied: “The issue has been resolved. Sorry for the inconvenience. Thank you for your patience and thanks for choosing LaneCloud.”

Black Friday Deals

It doesn’t take a QuinnyPig hot-take to figure out that some customers don’t need the level of service and associated costs that come with using a top-tier hyperscaler. You’re likely aware of the next tier down–Linode, Digital Ocean, Vultr, Hetzner, etc. …but you can keep going down and start to find specialty hosting companies, and indie operations.

Maybe you’ve heard of LowEndBox, a fairly organized index of low-cost hosting options. Now we’re getting close with LowEndBox to what we’re seeking. We’ve left the business district and gone to the commercial district, but still entering through the store-front.

It’s time to go to the flea market. The LowEndTalk forums are the definitive flea market for cloud services, colocation, and hosting. Like a true flea market, there’s strange etiquette, questionable vendors, reputable vendors, amazing deals, haggling, and plenty of buyer-be-ware scenarios. There’s a lot of overlap between LowEndBox and LowEndTalk vendors, but the LowEndTalk community gives you a lot of perspective–including realizing there’s plenty of organizations still happily running LAMP stacks and simple services.

Between Black Friday and March Madness deals where you pre-pay for 1 or 2 years of hosting for pennies on the dollar, I found myself with an inventory at least 10 VPS instances distributed across the globe. The compute capacity of LaneCloud was expanding.

Branding Spiral

By this point, the “LaneCloud” references were common Slack jokes, and we’re in peak covid lockdown so I decided it was time to seek out a professional and obtain an official Logo. Fiverr.com to the rescue.

I found an artist that specialized in minimalist 2d logos and enlisted their services. They gave me 4 choices… I selected 2 of them. One which I thought was more practical and I consider to be the official logo and another one that was a little more retro and technical which I opted to keep as an unofficial logo for backend resources or testing.

“Would you also like the social media kit?” …uh sure.

The results were complete and definitely worth the modest rate the artist charged. I had PNGS, SVGS, EPS as well as Adobe Illustrator original files. The social media kit educated me on the existence of custom headers and footers you can apply to LinkedIn, Xitter, and Facebook. I started feathering some of these new graphics into my profiles.

I had such a good experience, I wanted to see what else I could get on Fiverr.
Well, If you write your own script–with a little bit of luck you can hire a professional actor for $60 and get it back and 4 hours.

Since I had invested in branding and marketing, it felt prudent to commit to the name and purchase some domains. I secured lanecloud.cloud as my official public-facing domain. The sharks at afternic, GoDaddy’s domain squatting division, rejected my generous $200 counter-offer for lanecloud.com. They periodically send insulting emails to see if that will motivate me to buy.

Hi Lane,

Have you given up on this project? Is it safe to assume that you are no longer interested?

Let me know if you would like to make an offer.

I always make the same offer and they don’t respond.

Being lanecloud.com-less and having $200 remaining in my “marketing budget,” I clearly needed to solve this problem with swag. StickerMule is absolutely amazing. Their UI is easy, and their design team responds back quickly with proofs. They’ll gladly make kerning adjustments and other reasonable modifications without additional fees. Armed with die cut LaneCloud stickers, static clings, and even branded packing tape, it was time to define a product.

Product Vision Attempt

I found myself with a problem in search of a problem. I had all these cheap VPSs doing nothing–“Idlers” in LowEndTalk parlance. Servers with nobody to serve is a sad problem.

I concluded that meshing my pool of random VPS nodes into a unified Nomad Cluster would be a great way to leverage the resources. That solved my first problem. The answer to my second problem was revealed through the process of writing my fiverr commercial.

Pilot your new applications in LaneCloud

My general idea evolved into this basic thinking:

As an SRE-oriented person, I felt like it was still tragically too common that indie developers are still spinning up VPS instances and manually operating their apps for POCs and pilots. IMHO this means they’re missing out on a good metrics and alerting solution, good log aggregation and dashboard solution, and a sane solutions for secrets management. I felt like I could package up an opinionated solution that made sure developers had these things challenges solved…and strongly encouraged them to use it.

Clearly, this was somewhat inspired by Heroku, but I wanted this to be more container oriented–perhaps even exposing some aspects of Nomad to the users as well. The boundaries of LaneCloud would be crisp. Ephemeral service deployment, with monitoring, logging, and secrets management provided. For any other persistent storage, the customer would need to use a cloud solution such as object storage or a managed database.

The purpose of LaneCloud was to provide a better starting point for developers that want to build an app and take it through a pilot stage. Besides the observability and secrets features, LaneCloud would be able to provide moderate scaling across it’s pool of inexpensive compute resources. This allows a project to grow organically for a longer period of time, before needing to shift to a higher SLA cloud provider, or needing to rush and refactor their stack away from a single ec2 or VPS instance.

Following the same formula, I realized in addition to developers, LaneCloud could help hobbyist run popular “self-hosted” genre of applications such as the ones packaged on linuxserver.io. Having two viable use cases with the same base solution gave me confidence and a clear-enough vision that I could start planning.

Architecture Brainstorm

I started sketching down all the core infrastructure needs and tasks I’d ultimately need to tackle. It consumed several pages in my field notes and decided it was worth hanging onto in a more legible medium. I typed it up in a document as a basic “How to build a cloud” framework.

In reality, it’s just things you need to consider for any “production” greenfield environment. Even with managed services, these things still matter.

Here’s the raw headings from the outline:

My “how to build a cloud” rough outline

## Deploy an Inventory IPAM tool

## Define a basic taxonomy

## Make an out-of-band control Plane

## Build Monitoring Stack

## Internal Network

## Documentation and Version Control System

## Choose a secrets manager

## Solve for DNS

## Plan for User Management and Access Control

## Pick a task scheduler

## Persistent Storage Plan

## Worker Instances

## Service Discovery

## Ingress Control

## Edge Security

## Artifact Management

Home Base Required

Although the core model behind LaneCloud was distributed low-cost VPS instances, I intended to treat them like spot instances and assume a lack of reliability. I’d need a reliable hub to operate my nomad control plane, coordinate mesh networking, etc.

As this central location idea was brewing, I had summer with several power outages. On several occasions I found myself sprinting to plug in the generator before my UPS ran out. I was seldom fast enough.

Familiar messages would appear: “I can’t reach my VM.” “Is the server down?” “The CI Runner is hung.”

Signs were pointing me into a single direction. It was becoming evident that my server needed to go into colocation.

Texas Hospitality

I got a few quotes locally, but they weren’t quite in the price range I was looking for.
My search expanded to posting on LowEndTalk for more hosting options in my area. Not to many leads, but Josh from Data Ideas replied to my post and invited me to “come down to Texas.”

I hadn’t considered Texas. I’m spoiled with my sub 10ms ping times from Richmond to Ashburn–how could I ever consider something 40ms away? I began considering that I would live and popped onto the Data Ideas Discord server to learn more about this small provider.

Data Ideas is a small business that specializes in providing Colocation, VPS, and even RPI hosting to tech-enthusiasts like me. This means they’re much more willing to take a customer like me seriously, help get my equipment online, deal with any irregular non-enterprise hardware I might have, and just generally be very accommodating. The pricing was also fantastic.

I began working out an arrangement to send 2 servers, a switch, a router and an Odroid HC4. Sending an ARM powered driver-toaster is a great example of that irregular hardware case.

Before finalizing my arrangements, an existing customer on the Data Ideas Discord pinged me. He was wanting to expand his footprint and had a proposal for me to split 10U, a lot of network bandwidth and a generous amount of IPv4 addresses. I probed a little more to make sure I wasn’t setting myself up for trouble. The guy wasn’t doing anything nefarious with his half the resources, was very talented at BGP routing, and generally was a good match.

That’s how I met my rackmate, and how I also invented the word rackmate.

I started boxing up my iron, and thanks to Pirate Ship, LaneCloud went to Texas.

Control Plane V1

Although the long-term vision was certainly containers, I felt it was important to be able to provision VMs gracefully as a first step. My need was legitimate, but I was partially inspired by the fact that many VPS providers in the LowEndTalk space were using primitive solutions. I wanted to demonstrate a “simplified” approach using modern tooling.

The provisioning tooling for LaneCloud Alpha could easily be it’s own blog post. I’ll just enumerate some highlights:

• automation is driven by Ansible
• IP is assigned from Netbox
• VM scheduled by Nomad’s qemu driver
• custom cloud-init config generated by Ansible and Nomad template
• additional user customization available as a cloud-init include stored in consul
• Public DNS entry updated via Ansible module for dns-lexicon
• VM fully provisioned and available on internet with public IP in under 2 minutes.

Check out my LaneCloud snippets repo for more details.

Re-thinking Objectives

As time has progressed, my intentions around LaneCloud have changed a few times.

The lack of native ARM runners with GitHub Actions (now resolved) inspired me to pursue ARM bare-metal hosting using Raxda Rock 5B boards. Sadly I underestimated how long until the RK3588 would actually become viable to reliable run VM’s and containers. I should certainly know better by now.

Fast forward 2 years and the RK3588 is generally viable for such a use, but as time progressed I realized this wasn’t a path I wanted to go down. Low-cost hosting is a race to the bottom. Observing all the dialog on LowEndTalk made me realize that public customers in this space are brutal, prone to fraud, and expect a lot for a little. This was not the direction I wanted to go.

I’ve also began reconsidering my entire nomad-based architecture. It’s clever, but really isn’t made for persistent VMs. It’s difficult to manage console connections and maintain state. Between the recent HashiCorp License Changes and Kubernetes tooling at critical mass adoption, I’m ready to switch to Kubernetes and also use KubeVirt for any VM needs. Maybe one-day I’ll even get to MicroVMs.

What is probably next

As far as getting LaneCloud into the marketplace, I’m not sure I’m really going to pursue anything directly right now. It’s probably best at doing what it already does, which is serve the needs of myself, friends, and some of the tech community. I like the idea a “core services” platform-engineering style offering that I could make available to some folks. Perhaps operating as a co-op to let a small group folks take advantage of pre-configured tooling and resources is the most practical path.

Regarding the architecture, it’s definitely time to replace my Hashistack components with Kubernetes. As previously mentioned, it’s reached a state where I’m sold on it. The operator model is great, and there’s plenty of ways for me to easily integrate things. I really hope I can make it to MicroVMs.

So what is LaneCloud Really

Mainly LaneCloud is a way to keep me grounded in technology and remind me what it means to run things in the wild. With my $dayjob in consulting, I operate in a lot of spaces, but don’t always get to observe them long term. It helps me maintain several perspectives, including why most organizations should just use the cloud instead of running their own stuff.

I do enjoy having all the resource capability at a fixed cost. That’s not a financial argument tho, LaneCloud is definitely still a loss-leader.

Many may still just view it as a glorified homelab that escaped the lab, but I much prefer to view it as the strangest experiment my friend has ever seen.

Digging slightly deeper

I got addicted to just sharing my projects/thoughts etc in real-time over “big chat” (just made that up), and social media. As we know–social media takes more than it gives these days. Sharing on chat is fun, but even that can be information overload for the friends you’re trying to share with… Especially if half of its really notes. I’m very prone to stream-of-conciousness types of messaging. I’m sure its fatiguing for others.

beyond note taking

Hedgedoc has been a blessing for keeping notes in markdown. I really love it.. but I don’t like it much as a medium for sharing, even though it has a concept of publishing. Needs more decoupling. Also it’s theme is sometimes too narrow of a fixed with for snippets.

the right filing system

Lately I’ve been reaaally craving a blog to share my more recent science projects.. It’s been eating at me.. so that’s a sign. I have to say even going through this exercise of writing this first-post that it’s therapeutic to know I’m filing this content in an appropriate location.

The Eternal turmoil of momentum vs principled computing

“Just setup hugo to deploy in github pages” is the 2024 equivalent of “just scp files to an nginx server you manually configured”.

Naturally if I’d done either of those things, this would have been online 10 months ago… but I can’t. I’d been trying to hold out until I have a public-facing k8s tooled correctly with external-dns and cert-manager, but my compromise for now is to just deploy onto lanecloud object storage and front-end with an nginx i already have deployed and managed via ansible.

testing code snippets

yaml

- name: Alice
  age: 30
  city: New York
- name: Bob
  age: 25
  city: London
- name: Charlie
  age: 32
  city: Paris

bash

export HELLO=world

if ${VAR}; then
  echo "${SOMETHING}"
  printf("done\n")
fi

python

message = """
Hello, World!

This is a multi-line string
with multiple lines and spaces.
"""

print(message)

golang

package main

import "fmt"

func main() {
  fmt.Println("Hello, World!")
}