I kept my conversation with Amp beside a live preview of the blog. I could read a post, point at a sentence and ask for a change without explaining where to find it. That changed editing from describing a page to pointing at it. A small difference, but a very useful one!

Comment on the page, not around it

The blog uses Hugo, and the repository tells Amp how to start its preview server inside an Orb. I open Blog Preview in the thread’s Portal and read the actual page, including drafts and future-dated posts. The typography, images and captions are all there. I do not have to imagine what the Markdown will look like.

The Portal review widget lets me select something on the page and attach a comment. Amp receives the selected text and its location along with my feedback. It edits the Markdown, Hugo reloads the page, and I read the result in place.

In Part II, I mentioned one of those comments: “Assume one? It doesn’t feel direct and clear.” That was enough to ask for the wording change. I did not need to copy the surrounding paragraph into the conversation first.

Amp’s writing still needed editing. I decided what the posts should say, challenged claims that sounded stronger than the evidence and rejected wording that did not sound like me. It helped trace technical claims back to the repository and the checks we had run. However, a sentence can be technically correct and still not be something I would write.

There is also a rule in AGENTS.md: commenting on a draft does not give Amp permission to publish it. It must ask me to confirm the exact post title before starting publication. Asking for a better caption should not accidentally become a release!

One source, two publications

I also wanted these Amp posts to reach Substack readers. What I did not want was to prepare each post twice, copying paragraphs and uploading the same images again.

So the blog stays the original. My publication workflow requires the post to pass beta and production checks, followed by clearing the changed pages from the cache, before Substack distribution begins.

A small command built around a pinned version of python-substack prepares the Substack copy from the local Markdown. It handles the blog’s formatting and images and adds a link back to the original article. The library’s publishing flow needs a draft record, so the command creates one and then publishes it with email delivery enabled. There is no separate Substack editing stage, and later blog edits do not automatically sync across.

The command does not fetch the live blog or prove that its release checks passed. Amp must complete those checks first and use the reviewed source for that release. The repository also provides an Amp-post feed, but that is a separate output, not the input to this command or an automatic publishing connection to Substack.

Before sending anything, Amp must ask me a second question: whether to publish and email that exact title to Substack subscribers. Permission to update my blog is not permission to send an email.

The command keeps a local record of its progress. If a create or publish attempt has an uncertain result, it stops rather than blindly trying again. I would rather inspect Substack than send everyone the same post twice. The integration uses undocumented endpoints, so I do not want to assume they will always behave as expected.

Back to the blog

I started with a homelab that needed years of catching up. I am ending with a better way to look after it, try new ideas and write about what I learn. There is still work to do, but it feels much easier to begin.

Thank you for following along. I hope you found something useful for your own projects, or simply enjoyed seeing this one come together. The series ends here. The tinkering, of course, does not!

Revisit the series

If you want to dive back into any part of the rebuild, all ten posts are below.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation
4. Proving the entire platform with one blog
5. Handing PostgreSQL over to Amp
6. Modernizing an application, not merely moving it
7. Replacing a surveillance media chain, not just a recorder
8. Proving Frigate before the cutover
9. Upgrading observability beyond the old cluster
10. Building a platform agents can keep improving

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation
4. Proving the entire platform with one blog
5. Handing PostgreSQL over to Amp
6. Modernizing an application, not merely moving it
7. Replacing a surveillance media chain, not just a recorder
8. Proving Frigate before the cutover
9. Upgrading observability beyond the old cluster
10. Building a platform agents can keep improving (this post)

Why did I spend so much time rebuilding a homelab that was still mostly working? Newer software was one reason, but it was not the main one. I wanted the next upgrade, security fix or half-finished idea to be easier than this one. Future Vincent had already inherited enough jobs from past Vincent!

So this project had two goals. The first was the long-overdue upgrade and rewrite. The second was to build a platform that agents can safely improve later. The previous posts tell the individual stories. This final part is about what that changes for me.

The LLM is only part of it

Better LLMs obviously made this possible, but an LLM on its own is still just a very clever chat window. It can explain Kubernetes all day and my cluster will remain exactly as old as before.

What made the difference was the agent setup around it. Amp could read the real repository, change it and run the same checks I would run. Each Orb had its own clean environment, so one task did not inherit whatever mess happened to be on my laptop. Repository instructions explained the safe way to work, and restricted runners reached the homelab only when a real test needed them.

The model did the reasoning. Amp gave it the tools, context and a safe place to work. Both mattered.

The real gain was async work

Watching an agent type quickly is impressive for about five minutes. The much bigger gain was that I did not need to sit beside it.

I could give an Orb a clear objective, let it investigate or build, and come back when it had evidence for me. Several independent jobs could run in separate sessions without fighting over one workspace. I could review a finding, answer a question or redirect a session from my phone. Production decisions still needed my attention, but the investigation did not stop just because I left my desk.

The screenshot below gives a small idea of that pace. Home Portal alone had reached 38 Amp threads and was still moving that day. Each thread kept its own task and history, instead of everything being squeezed into one heroic terminal session. My terminal has suffered enough already.

Without hosted agents working in the background, I would not have found enough time for this rebuild. I certainly would not have had time to fix the bugs and security problems found along the way. Speed helped, but being able to work asynchronously was the game changer.

Can agents be trusted?

People often say LLMs are unreliable. They can be. An agent can misunderstand the task, trust the wrong health check or continue very confidently from a bad assumption. Confidence, unfortunately, is not a test result.

My experience is that agents can also be extremely diligent when I give them a clear goal and a solid way to check their own work. Telling an agent to “be careful” is not enough. I want it to inspect the right environment, test the real route, compare the data, open the page in a browser and stop if the result is wrong.

That happened several times during this rebuild. A check caught a result that only looked successful, and a release stopped instead of moving to production. This does not make the agent infallible. It means the usual engineering practices still work: keep the change small, test the real thing, record the result and require approval before doing anything risky. No green test, no Ship button. Simple!

What this scale cost me

There is no honest “API-equivalent” price for this work. My linked subscriptions covered a lot of the inference, including 656.6M tokens recorded at USD 0. Nobody can tell me what exactly the same work would have cost through model APIs, so I will not invent a number.

What I can show is what I actually paid Amp: USD 392.23 in total. That includes USD 40 of Amp subscription credits from two months, leaving USD 352.23 of extra token usage. Orb usage was only a tiny part of the total.

The usage is still useful for showing the scale. Across my account, I recorded 1.27B tokens: 656.6M through linked ChatGPT and 613.3M from Amp. I also found 180 threads linked to this repository. The thread chart and model charts cover different scopes, and seven older or plugin-created threads did not record a mode.

These are large numbers, but they are not a benchmark. Your homelab does not suddenly need a billion tokens to be respectable! They simply show how much work my subscriptions, prepaid credits and hosted environments supported.

What comes next?

The external watchdog from Part IX runs on the same free-tier Oracle VM as my OpenClaw. The obvious next step is to let Amp reach further: keep OpenClaw updated, harden the setup and build my own knowledge base for an always-on personal agent that I host myself.

That work is still ahead of me. I want to approach it in the same way as the homelab: clear ownership, repeatable changes, proper checks and a human decision before anything risky. An always-on personal agent sounds great. An always-on personal agent with no guardrails sounds like the beginning of a different kind of blog series.

A much better starting point

My old homelab depended too much on spare weekends, commands I was supposed to remember and my willingness to disturb something that was just about working. That is why so many upgrades waited for years.

The rebuilt platform is different. More of its design now lives in the repository. Agents have isolated places to work, real ways to test their changes and clear boundaries around production. LLMs made the work faster, but remote agents made it possible for me to fit the work into my life. Verification made that speed useful rather than scary.

I started this project with five years of jobs waiting for future Vincent. At least the next future Vincent will have some help.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation
4. Proving the entire platform with one blog
5. Handing PostgreSQL over to Amp
6. Modernizing an application, not merely moving it
7. Replacing a surveillance media chain, not just a recorder
8. Proving Frigate before the cutover
9. Upgrading observability beyond the old cluster (this post)

I first set up monitoring for my homelab about five years ago. It could still draw graphs and send alerts, but the stack had become old together with the cluster it watched. Five years is a long time in software. Some of the components were almost ready for their own retirement post!

The replacement itself was not very exciting. I now use VictoriaMetrics, Grafana, Blackbox Exporter, vmalert, Alertmanager, Loki and Alloy. It is a modern, fairly standard observability stack, and there is little value in explaining every component yet again.

The interesting part is what happens after the stack finds a problem. In the old setup, an alert waited for me to notice it, understand it and find enough time to fix it. Alerts are very patient. Production problems are usually less so.

Now a selected alert can wake an Amp Orb and start the investigation by itself. The homelab is no longer only monitored continuously. It can also begin triaging a problem, preparing a fix and moving it towards deployment while I am away.

Let an alert wake an Orb

The setup is quite small. An Amp plugin gives me a private webhook URL. I store it in a Kubernetes Secret, then configure Alertmanager to send selected critical alerts to it. There is no extra incident platform in the middle. Alertmanager simply talks to Amp.

When a webhook arrives, the plugin checks the payload, keeps only the useful facts and ignores duplicate retries. It wakes a medium-mode Orb for the incident. If the same problem returns, the new alert goes into the same thread, so the Orb keeps the earlier context instead of starting from zero. I fired test alerts through the complete path to make sure this actually worked.

The handoff is fully agentic. Nobody has to notice the alert and copy it into a prompt. The Orb begins with read-only checks, looks at the evidence and decides whether the problem needs a code or configuration change. If it does, the work can continue through the same repository, beta checks and deployment path from the earlier posts.

This does not mean an alert has permission to change production. That still stops for my approval, which is probably for the best. The game changer is that the alert can start the work, keep the context and move towards a checked fix before I even open my laptop.

Keep one witness outside Kubernetes

An external watchdog is not a new idea. I always knew I should have one, but I could never be bothered to set it up. There was usually something more interesting to build than a script that checked whether the other scripts were alive.

Amp finally took care of it for me. The watchdog runs on the free-tier Oracle VM where I already run OpenClaw and checks the public paths every minute from outside my homelab. It is simple, independent and no longer sitting on my to-do list.

A rebuilt and agent-managed cluster

After all this work, I now have a rebuilt cluster with old services replaced by newer stacks and components. More importantly, the whole platform is agent-friendly, agent-assisted and increasingly agent-managed.

This is not vibe coding. I did not ask an LLM to make the homelab “modern” and hope for the best. A lot of thought went into where things belong, how changes are tested, how beta stays separate from production, how deployments stop when checks fail, how the platform is watched and how the next service can be added without making a mess.

These are simply good engineering practices. The difference is that agents can now follow them even when I am not sitting at my desk. The cluster is not fully autonomous, nor do I want it to be. It can investigate, fix and deploy through a path designed for agents, while the important decisions still come back to me. That is a much more useful kind of automation.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation
4. Proving the entire platform with one blog
5. Handing PostgreSQL over to Amp
6. Modernizing an application, not merely moving it
7. Replacing a surveillance media chain, not just a recorder
8. Proving Frigate before the cutover (this post)

Part VII ended with a working Frigate beta. Four cameras were streaming and recording, the storage projection made sense and the container ran with a small, documented security exception. That proved the design. It did not prove that I could point my family at it, because the browser had no proper sign-in yet, and it did not prove that the recorder would still be recording next week.

So I needed two more things before I could even think about a cutover: sign-in that I trusted, and a test that lasted longer than a demo. A working beta is not the same as a safe cutover. Surveillance is the one application where I care most about the boring hours.

Put Auth0 in front of Frigate

The identity provider was never in question. Back in the original surveillance post I had already decided that I did not want to run a user database at home, and Auth0 has protected the old feed since then. Baby Tracker joined it in Part VI. The question this time was how to put it in front of an application I did not write.

Frigate has its own login, but I did not want yet another password to look after. It can instead trust a proxy to tell it who the user is, so Amp gave Frigate a dedicated oauth2-proxy and configured the Traefik route to ask that proxy before forwarding anything. Auth0 puts the user’s roles into the ID token. oauth2-proxy refuses any session that carries none of the approved roles. Frigate then maps the roles it does receive onto its own built-in admin and read-only viewer roles. If a request somehow reaches Frigate with no usable role at all, it lands in a deliberately useless default role whose only camera does not exist, so it can see precisely nothing. Failing closed twice felt about right for a camera system.

One detail matters more than it looks. Before the route asks oauth2-proxy anything, it strips every identity header a browser could have supplied, and afterwards it copies only the two headers from oauth2-proxy itself through to Frigate. Without that, a request could simply declare its own role and Frigate would have no way to tell the difference.

Frigate is also not a single web page. Live video, recording playback, the API and the WebSocket traffic behind the live view all had to go through the same boundary, and then straight to Frigate’s authenticated port rather than being relayed byte by byte through the proxy. The internal health port from Part VII stayed Pod-local. A neat login page in front of the UI is worth very little if the video endpoints answer without it.

I did find one bug myself. Signing out of the beta site sent me to the wrong place, so I told Amp: “Redirection should follow the beta and non-beta split.” Then, because a login page that looks right is exactly the kind of thing that makes me nervous, I followed up with: “Make a thorough check that frigate won’t expose recordings or live streams to unauthorized persons.”

Amp tested the role mapping in the pinned Frigate image, checked that a synthetic viewer could reach a media route but not an admin route, and attacked the live beta without a session. Requests for the UI, API, recordings, live media and WebSockets all went to Auth0 instead of returning content. Forged identity headers changed nothing. It also verified the corrected sign-out path, which returned beta to the beta Home Portal rather than production.

To be clear about what this is and is not: the beta boundary behaves this way today. Production still needs one final pass with disposable test accounts for every role, including authenticated live view and playback. The configuration exists; that last production check does not.

Keep the first checks quick

I did not want to wait 24 hours to find out that I had broken something small. So Amp built a short loop first: deploy to beta, wait for Frigate to become ready, confirm that every configured stream is producing frames, confirm that recording files exist, and request the browser route without a session to make sure it still redirects to Auth0. It lives in the Justfile now, so deploying and checking the beta is one command rather than a list I would forget. If something basic was wrong, we knew within minutes rather than the following day.

None of this required looking at private footage. Amp used Frigate’s stream counters, the timestamps on the recording files and FFmpeg’s description of a few sample segments. The four available feeds ran at about five frames per second, new files kept appearing on the QNAP, and the sample segments contained H.264 video with AAC audio, exactly as designed.

That loop proves the path from camera to recording to browser. It does not prove the recorder can stay healthy, which is a different question entirely. Once the short checks passed, the same build could move on to the long test without anyone touching it.

Let 24 hours decide

At this point the quick checks looked good. However, I was not going to replace the old recorder after only a few minutes, so I asked Amp to keep the same beta running and check it over 24 hours.

The test covered only the four active cameras, as I had already told Amp to ignore the one I removed. It still failed because Frigate restarted, recordings had gaps and RTSP errors kept returning. If I had opened the beta during one of the good hours, I would have said it worked and missed those problems.

Now I need Amp to find out why it restarted and why the recordings had gaps. Then I will run the same test again with all four active cameras. Object detection can wait until this basic job works properly.

The old recorder keeps running until then. I am not cutting over until the 24-hour check passes.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation
4. Proving the entire platform with one blog
5. Handing PostgreSQL over to Amp
6. Modernizing an application, not merely moving it
7. Replacing a surveillance media chain, not just a recorder (this post)

Home surveillance was the first application I deployed in the homelab. I wrote about it in my original homelab post and again in a post of its own: cameras locked behind the firewall with no way to phone home, FFmpeg runners saving footage to the NAS in five-minute chunks, and later a small Node.js relay so that Home Portal could show a live feed. It worked for years, and I was quite proud of it.

It was also, if I am honest, a mess. Feeds on the old cluster failed from time to time, and NodeMediaServer frequently ran out of memory. Every camera stream passed through a custom RTSP reader, then RTMP, then HLS and MP4, while separate FFmpeg processes wrote the recordings to network storage. With so many stages, even finding which part had failed took more work than it should have.

I did not want to carry that whole chain into the new cluster. As with Baby Tracker in Part VI, moving it as it was would only have given the old design a new home. What I actually wanted was one service that receives the streams, records them and shows them to me.

Frigate looked like it could be that service. It is an NVR built around FFmpeg, and it ships with go2rtc, a restreamer that holds a single connection to each camera and shares it with whoever needs it. Between them they cover exactly the jobs my relay and my recorders had been doing, without me maintaining any of the code.

So I gave Amp a short prompt: “Look at the plan to rebuild the surveillance platform, start the work”. The plan described the outcome I wanted. It said nothing about the cameras.

Let Amp discover the cameras

That omission was deliberate. I could have typed out the camera list, the stream paths and the codecs from memory, but my memory of that system was several years old. I wanted to see whether Amp could find out the truth on its own rather than trusting what I remembered. If it could, I would also get an up-to-date inventory for free.

It could. Amp found the cameras on their isolated network, confirmed which of them answered on RTSP and ONVIF, and then probed the actual streams instead of reading the spec sheet. Each of the four reachable cameras offered two streams: a full-resolution H.264 main stream for recording and a small substream that is perfect for the live grid. Both carried G.711 audio, which browsers do not play natively. Amp even noticed that the camera’s own metadata described the H.264 profile differently from what the bitstream actually contained. That is the kind of detail you only learn by testing the real thing, and exactly why I did not want to work from memory.

Those findings shaped the design rather than the other way around. Frigate could copy the main video straight into recordings without transcoding it, which matters a great deal on a Raspberry Pi. go2rtc could keep one connection per camera stream and share it with the recorder and every browser tab, so opening the live grid no longer meant opening yet another RTSP session against each camera. Only the audio needed converting, from G.711 to AAC, and that is cheap.

Work out the storage first

The recordings still belonged on my QNAP, not on a Raspberry Pi’s local disk. Beta and production also needed separate directories on the NAS. A test run should never be able to write into the real recordings, let alone delete them.

Before building anything, Amp worked out the storage for my original five-camera plan. Five main streams plus audio would use about 5.44 Mb/s, or 58.8 GB a day. Ninety days would need roughly 5.3 TB for the camera data, so Amp recommended at least 6.1 TB after allowing for overhead. I later retired one camera permanently, which brought the active system down to four and reduced the real requirement.

I liked having this number before a single recording was written. The old system never had anything like it written down anywhere. It is still an estimate, not a promise. A ten-second sample of the four reachable cameras showed rates between 330 and 832 Kb/s per camera, depending on what was happening in front of the lens. That is far too short for retention planning. I still want a representative full day before I trust the final allocation.

The replacement is much simpler to draw. Each camera connects to Frigate and go2rtc. Frigate writes the recordings to the QNAP and serves both live and recorded video to the browser. It does not need my custom relay or per-camera recorders.

Keep camera access private

The cameras require a login for RTSP, and in the old system that login lived in a plaintext ConfigMap. Amp moved it into a Kubernetes Secret, and Frigate now reads it from the environment when it builds the stream URLs. The credentials stay on the server side of everything; the browser only ever talks to Frigate.

Amp also kept the workload’s network access to the minimum. A NetworkPolicy lets it reach only the camera addresses, and only on the RTSP and ONVIF ports. Frigate has a second, unauthenticated port for its own health checks, and that port stays inside the Pod with no Service in front of it, so it cannot quietly become a way around the front door.

Frigate’s official container could not run under my strictest security policy. Amp followed the startup errors and added back only the permissions it needed. I verified that Frigate started while the other limits stayed in place. The small exception is written down in the repository.

Start with a small beta

Object detection stayed off, deliberately. Detection is the feature Frigate is best known for, but the first beta only had to prove three things on the ARM64 workers: streams come in, recordings land on the QNAP, and both keep going. Every extra feature would only have made a failure harder to explain.

Amp deployed it to the LAN-only beta environment from Part IV and checked it the way I wanted it checked, which is to say without watching the footage. Stream counters showed frames advancing on every configured feed, recording segments appeared on the NAS with sensible timestamps, and a probe of sample files confirmed H.264 video with AAC audio inside. All four active cameras were live and recording. The fifth camera had been switched off permanently, so four was the complete setup rather than a partial result.

The beta showed that the design could work, and I verified it myself. Should I just push Ship and replace the old recorder? No. The live feeds and recordings still needed to be protected behind sign-in, and a short beta did not prove that the recorder would keep working. I would tackle both next.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation
4. Proving the entire platform with one blog
5. Handing PostgreSQL over to Amp
6. Modernizing an application, not merely moving it (this post)

At the end of Part V, the next job was to bring Baby Tracker into the new cluster. However, simply moving the existing application was not enough. It was still running on React 16 with an old Create React App build, so I would have ended up with a new Kubernetes cluster running the same old application. That was exactly the kind of like-for-like upgrade I wanted to avoid in Part I.

I asked Amp what was possible if we modernized the application properly. The answer was not just a new frontend. I also needed somewhere private to test it, a better authentication setup and a faster way to deploy changes to beta.

Rewriting Baby Tracker

I first mentioned Baby Tracker in my original homelab post. It was an attempt to keep family data within my home without using any external service. It was also a project for me to learn how to build a single-page application. I built the whole thing manually, long before I started using AI coding agents.

The application was still useful, but the technology behind it had become difficult to maintain. Putting the same build into a new container would not fix that, so Amp rebuilt the frontend with React 19, Vite and TypeScript.

This was not about using the newest framework just for the sake of it. The new code is easier to understand, test and change. More importantly, the next update should not require me to first remember how an old ejected Create React App build works!

Now I had a rewritten application, but code sitting in a repository does not prove that it works. I needed to deploy it to the real cluster and test it without making an unfinished family application public.

Testing it without making it public

In Part IV, I created a LAN-only beta route while testing the new platform with this blog. Baby Tracker was where this beta setup became really useful.

Beta has its own workloads, configuration and data boundary. Its route is only available from my home network, so I can build and deploy the application from a machine at home, open the beta site over HTTPS and test it on the real cluster. Nothing needs to be exposed to the public internet.

This gives me a safe place to find a broken frontend, a wrong environment variable or a bad authorization rule before production is involved. It is much more useful than testing the application only on my development machine because it uses the same Kubernetes platform as production.

The first important test was authentication. The old application did not have its own sign-in flow, so I did not want to copy that design into the rewrite.

Rewiring authentication

The new frontend signs users in with Auth0. Requests to the API then pass through Oathkeeper, which checks the Auth0 token before the request reaches the API. If there is no valid token, the GraphQL request is rejected with a 401 response.

I am also using Auth0 for other applications in the homelab. Home Portal signs users in directly with Auth0, Matrix uses Matrix Authentication Service, Frigate sits behind oauth2-proxy and Grafana uses generic OAuth with a local login kept for emergencies. The blog remains public. They share the same identity provider, but each application still has its own access rules.

The Auth0 CLI is also a surprisingly good match for working with an agent. Amp can start the device authorization flow from the terminal, then I approve the displayed code in my browser. The code expires if I do nothing, so it is much better than giving an agent a long-lived client secret. It is both easy to use and time-limited, which is a good balance for this kind of work. The CLI session created afterwards is separate, so I still only use it on a trusted machine.

At this point the rewritten application and authentication were working in beta. However, deploying every small change was still too slow.

Speeding up beta deployment

Initially, every change had to go through GitHub Actions to build an image, then Flux had to deploy it to beta. The median CI time to publish an image was 225 seconds, and that did not include the rollout or any testing. Waiting almost four minutes for an image makes a small frontend change feel very expensive.

I asked Amp what could make beta development faster while keeping production deployment in Flux. I did not ask for a particular tool. Amp suggested Skaffold, which rebuilds only what changed and deploys it directly to the LAN-only beta environment.

This faster path does not create a separate version of the application. Beta and production use the same shared Kubernetes manifests with separate overlays, so the two environments stay aligned. Skaffold handles beta feedback while Flux remains the only production deployment path.

The full beta deployment and verification took 78 seconds. The applications became Ready on the Raspberry Pi workers, the real beta pages loaded, authentication was checked and production remained unchanged.

The timings cover different work, so they are not a direct benchmark. What matters is that I can now make a change, deploy it to the real beta hardware and test it while I still remember what I changed!

Production deployment remains deliberately slower. It still needs an immutable image verified in beta, my approval and a reviewed Flux change. The faster feedback loop does not weaken that process.

Baby Tracker is now more than an old application moved into a new container: its frontend, authentication and beta workflow were modernized together, and Amp’s Skaffold suggestion made that workflow fast enough to use. It was a good first test because it is still a fairly simple CRUD SPA. The next application is much harder! Part VII takes the same approach to my home surveillance platform, with multiple live camera streams, recording and network storage.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation
4. Proving the entire platform with one blog
5. Handing PostgreSQL over to Amp (this post)

A stateless blog was a safe first test. PostgreSQL was different. If I broke it, rebuilding the container would not bring the data back.

The database holds my public IP history, written by the simple script that updates my public IP at Cloudflare DNS every 15 minutes. Every run since January 2021 has added a row: more than five years of history. No single row matters much, but I like having the whole timeline.

In the old homelab this would have been a weekend of checklists and nervous copy-pasting. This time I typed one sentence to Amp:

look into that control plane, migrate the old dns history table to the new Postgres instance, and also stand up the cloud flare dns updater job in the new cluster

Three jobs in one breath, with no plan and no mention of how to check anything. I did not care how the rows got from one cluster to the other, as long as all of them arrived and new ones kept coming.

What Amp did with one sentence

Once I had given it SSH access to the old Raspberry Pi control plane, it went looking. It found the table, counted 215,089 rows and noticed two old updater jobs still writing every 15 minutes. Then it came up with the plan I would have wanted if I had bothered to think about it: back up first, rehearse the copy while the old jobs kept running, then freeze the writers, copy the last few rows and compare both sides properly. I never described any of that.

The comparison is where it earned my trust. Its first check failed, and for a moment it looked as if five years of history had been damaged between two terminal windows. Amp exported both sides the same way, found them byte-for-byte identical and traced the mismatch to its own checking query. The data was fine. The check was wrong.

Before suspending the old jobs and letting the new updater touch a real DNS record, it listed what it was about to do and asked. I typed “yes”. It copied the final four rows and confirmed that both databases held 215,093 rows with identical checksums. New rows started arriving on schedule. Along the way it also fixed two bugs I had not asked about, one of which could have lost history while Kubernetes reported the job green. I learned about both from the summary.

Then I asked for the upgrade

The next day the new cluster still had one old part in it: PostgreSQL 15. I typed:

upgrade the deployed Postgres instance to the latest stable version

I did not say 18, I did not say how, and I half expected it to change 15 to 18 in the manifest and tell me it was done. Instead it checked the running server, worked out that the current release was PostgreSQL 18.6, and came back with the reason a tag swap would not work: a major release can change the on-disk format, and the official image had moved its data path too. Kubernetes would have rolled out a perfect image and left me with a database that refused to start. So it planned a dump and restore, and asked once whether it could push the change and run it on the live cluster. Thirty-four seconds later I replied “yes do that”. That was my last contribution.

Two things happened in the next few minutes that I would have found hard on my own. Flux fought back: Amp had paused reconciliation so that GitOps could not recreate a writer mid-move, but a parent reconciliation restored it and the writers came back to life. Amp noticed, stopped them again and paused both layers. GitOps was doing exactly what I had configured it to do, at the worst possible moment!

After the restore the new server held 215,174 rows, the previous day’s total plus a day of new ones. Another Amp thread was already waiting to put Baby Tracker into the same database; the two agreed between themselves who owned it and who would wait. I was not part of that conversation.

From my one-line prompt to the final report took under fourteen minutes.

What actually changed

The database moved, it was upgraded and no rows are missing. What stayed with me is that I did not do it, and I barely steered it. Apart from lending it an SSH key, I wrote one sentence for each job, said yes twice and read two reports. Amp asked before touching DNS and before pushing to master, and that is the boundary I want to keep. Everything in between was decided by Amp, and decided better than I would have in a hurry.

To be fair, this was a small database. One table, a couple of hundred thousand rows; the move did not even need pg_dump. The job was small. The range Amp covered was not. It read the old cluster before touching anything, knew that a PostgreSQL major version and a Docker image layout are different kinds of change, noticed Flux undoing its work and understood why, fixed bugs in code I had not pointed it at, and negotiated with another agent over who owned the database. That is a wider range than I would cover on a good day, and nobody told it which of those things mattered.

That, I think, is the actual shift. My job was to describe the outcome and to know which two questions I wanted to be asked. The rest was somebody else’s problem, and for once that somebody was not future me.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation
4. Proving the entire platform with one blog (this post)

After building the K3s control plane, I needed to find out whether the rest of the platform actually worked. I could have moved several applications at once and spent days wondering which layer was broken. Instead, I chose the simplest workload I had: this blog.

It is stateless, quick to rebuild and very easy to recognise when it works. If you can read the page, the test passed! That made it a perfect signal for the complete path: Git → Flux → MetalLB → Traefik → cert-manager → TLS.

One small blog, one very big test

Each project had one clear job. Flux applied the desired state from Git. MetalLB gave the cluster an address on my home network. Traefik received requests, matched each hostname to the correct service and terminated TLS. cert-manager obtained certificates from Let’s Encrypt through Cloudflare DNS.

That sounds like a lot of moving parts for one little blog. That was exactly the point! A familiar page turned the whole platform into a simple yes-or-no question: did a change in Git arrive correctly over HTTPS?

The feedback loop was fast. I could make one change, let Flux apply it and check the result in the browser. There was no database to migrate and no user data to protect. The blog gave me confidence in the shared networking before I put anything more complicated behind it.

Kubernetes was healthy, but Chrome disagreed

The best example came when Kubernetes looked healthy but Chrome still rejected the TLS connection. My split DNS was combining local and upstream records for the same hostname, so the browser received two incompatible descriptions of one service. Making the local answer authoritative fixed it.

No amount of restarting pods would have helped. The platform was not healthy until I could use it from the browser. That small test checked the path I actually cared about, not merely the individual pieces inside Kubernetes.

Pave the way for beta

Once the page loaded through Traefik with a valid certificate, I had more than a working blog. I had a baseline networking path that every future application could reuse. Traefik became the front door for ingress and TLS termination instead of every application inventing its own way in.

That simple baseline later made a much safer workflow possible. An application can have a separate beta deployment with its own hostname while still using the same Traefik and TLS path as production. Local DNS points the beta hostname at the cluster, while a Traefik allow-list keeps it accessible only from my home network.

This gives me a realistic but LAN-only beta environment. I can test an application over HTTPS without exposing unfinished work to the public internet. The beta setup is more advanced, but it rests on the same path first proven by one very simple blog.

Of course, the blog did not prove data recovery or any stateful migration. It did prove that the platform had a working front door and a fast feedback loop. With that baseline in place, I was ready for the much scarier PostgreSQL migration.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly
3. Building the new K3s foundation (this post)

K3s itself was not new to me. For my old cluster, I followed Jeff Geerling’s Raspberry Pi cluster guide and customized the K3s Ansible playbooks it recommended. That got me running quickly, but over time my version of the setup and the system it created became harder to reason about.

This time I wanted to try something different. Instead of finding another setup and slowly turning it into my setup, I described the outcome and let Amp work out how to get there.

The hardware did not change. My four Raspberry Pis stayed exactly where they were; I only reinstalled Raspberry Pi OS from scratch. The new part was one Ubuntu Server VM running in Hyper-V on Windows 11. It would run the K3s control plane, work as an AMD64 node and welcome the Pis back as ARM64 workers.

Keep the control plane boring

A single server is not highly available. I knew that. However, extra control-plane nodes would add more moving parts while I was still trying to understand the basic setup. For now, I wanted the control plane to be simple, reproducible and boring—and I mean boring as a compliment!

K3s uses its default SQLite datastore. I added resource reservations to protect the control plane, then disabled the bundled Traefik and ServiceLB because I wanted GitOps to manage those parts later. This gave me a small foundation without pretending it was already the finished platform.

Turn the plan into something repeatable

I had already written the rebuild plan, so I told Amp: “Create scripts to prepare this host as the kube control plane according to the rebuild plan.” On my local Amp runner, it turned that plan into scripts for networking, security updates, SSH, firewall rules, secrets encryption and K3s. I ran those scripts manually on the real VM and pasted the failures back into the thread for Amp to fix.

Before I ran anything, I asked Oracle to review the changes. This was foundation code. Every application would eventually depend on it, so an extra review felt essential rather than excessive. The review pushed me towards the official K3s installer and CLI wherever possible. Amp could still customize the official script with my pinned version and options; it did not need to reinvent the installer in shell. K3s already knows how to install K3s (and future Vincent does not need another clever script to debug!).

The scripts also test key-based SSH before disabling password access. That is a small check, but a very useful one. Automation should prove the replacement works before removing the old access path. Opening the Hyper-V console because I locked myself out would not be a fun start.

The first run broke DNS!

The first run quickly found two bad assumptions. One firewall command tried to insert rule number one into an empty firewall, where rule number one did not exist. More importantly, the default-deny firewall blocked the Pi-hole resolver already running on the VM. Kubernetes looked healthy, but household DNS was broken!

I told Amp exactly what I could see: “Now the Pi-hole instance on this host is not responding to DNS queries.” It traced the problem, made the firewall commands safe to run repeatedly and restored DNS from the LAN. Then it added DNS to the verification so the same mistake could not quietly return.

This felt very similar to the time Amp followed one wrong DHCP lease all the way to a powerline extender. I did not need to guess the fix. I described the symptom, gave it the context and let it follow the evidence. A healthy Kubernetes node was not enough. The same VM also served my home network.

Add backups before moving on

Because this cluster uses SQLite, K3s’s etcd snapshot command does not apply. My backup briefly stops K3s, creates a consistent archive, starts K3s again and transfers the compressed result to the NAS.

There is one important trap here. If the NAS is not mounted, its usual path still looks like an ordinary directory. A careless backup job could happily report success while filling the VM’s own disk! Mine stops unless it can prove the NAS is mounted, then writes a checksum beside every archive and removes old backups according to the retention policy.

I did not test a restore yet because there was still nothing meaningful to restore. The cluster had no workloads or important state, so restoring an empty foundation would prove very little. That test will matter once real applications arrive.

For now, I had a clean K3s cluster, repeatable setup and backups ready before there was anything precious to lose. Better still, the first failure had already made the checks stronger. It was time to build the rest of the platform.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it
2. Making a homelab repository agent-friendly (this post)

In Part I, I decided to rebuild the homelab with Amp. However, before I trusted it with Kubernetes, I wanted a much safer place to start: this blog.

My prompt was: “Design an orb based blog writing and preview system which I can see posts live on the right hand side in the Portal, and publish when I say yes.” I wanted Amp to set everything up, show me the result and wait before publishing. That sounded simple. It turns out the repository first needed to explain how I actually work.

Start with a repeatable setup

Amp works in an isolated Orb, not on my laptop. Therefore, I pinned the tools and kept dependencies in lockfiles so every new Orb could build the blog the same way. I also put the common commands in the root Justfile. Now “run the checks” means just check, instead of a collection of commands I may or may not remember after six months.

This helped me too! A setup that is easy for Amp is usually easier for future Vincent.

Tell Amp when to stop

Next I added AGENTS.md files beside the code. They contain practical rules: beta is not production, a deployment does not include a DNS change, and inspecting a database does not mean writing to it. Checks that only need the repository run in the normal sandbox. Tasks that need my real cluster use a constrained runner inside my home network and an explicit Kubernetes context.

Those rules matter because this is not a demo cluster. It contains services and family data that I do not want an enthusiastic agent to “fix” unexpectedly.

The funniest test came when I told Amp to ship the blog and then wondered why nothing happened. It was waiting for the exact publication confirmation that I had required. I was annoyed for about ten seconds. Then I was very happy that it had listened!

Keep the useful procedures

Some jobs have too many important steps to remember, especially beta verification and production promotion. I recorded those as Agent Skills. Amp can follow the same reviewed sequence each time and stop if the image, health check or data check is wrong. The Skill describes the job; it does not give permission to run it.

I still steer the work as it happens. While reviewing the first post in the live Portal, I commented: “Assume one? It doesn’t feel direct and clear.” Amp rewrote the sentence. Later, a LAN runner could safely perform read-only beta checks, but the repository rules did not permit it. I changed that specific rule rather than asking Amp to ignore all the rules.

There are still limits. Production needs my approval, secrets stay outside Git and not every application has equal test coverage. However, Amp can now investigate a problem, make a change and show me the checks without guessing what command to run next. More importantly, it knows when it must wait for me. With that in place, I was ready to let it help rebuild K3s.

Originally published on V for Vincent.

1. Why I modernized my homelab instead of upgrading it (this post)

This is the start of a new series about rebuilding my homelab from the ground up. Later posts will cover the platform, applications, data and lessons from the journey. This first part is about the decision that shaped everything afterwards: I was not looking for a like-for-like upgrade. I wanted a full-scale modernization.

I have written about my tiny yet powerful homelab before. It started with four Raspberry Pis, a QNAP NAS and a small Lenovo PC. It grew into a useful collection of home applications and a playground where I could learn Kubernetes, networking and storage.

It also grew one experiment at a time. I would get an application running and then move on to the next idea. After a few years, some applications were stuck on old dependencies, deployment instructions existed only in my head, and documentation had drifted away from reality. A change in one place could have consequences somewhere completely unexpected.

My Dependabot page told the same story. I had accumulated update pull requests that I simply had no idea how to resolve safely. The backlog had grown to the point where it became easier to ignore the list than to work out where to begin.

A small but satisfying footnote: I asked Amp to close the entire legacy batch in one go. When Dependabot opened another batch, I came back today and asked Amp to decline all of those too—the modernization meant I no longer needed them.

A power trip that took down my homelab network provided another warning. Amp helped trace the failure across the network and Kubernetes, but the incident reminded me how difficult an organically grown system can be to reason about when something goes wrong.

I could have installed a newer Kubernetes version, restored all the old workloads and called the job complete. That would have been faster at the beginning, but it would also have preserved every outdated dependency, hidden assumption and awkward workflow. I would have ended up with a newer cluster carrying exactly the same baggage.

So I decided to treat the old homelab as a source of requirements, not as a template.

The useful applications and important data would stay. Everything else had to justify its place. Old dependencies would be upgraded or removed, applications could be rebuilt when incremental upgrades no longer made sense, and important operational knowledge would move out of my head and into the repository.

This finally gave me a way to clear the legacy Dependabot backlog. Closing that old batch was a satisfying sign that this was real modernization rather than a cosmetic upgrade. New updates will always arrive, but now I have a much better foundation for dealing with them.

I also wanted the repository to be friendly to AI coding agents. That does not mean allowing an AI to make uncontrolled infrastructure changes. It means creating an environment where an agent can understand the system, follow paved paths, run the right checks and know when it must stop and ask me.

This closely matches Amp’s article about putting an agent in an Orb: an effective development environment should be designed for agents from the start, not adapted as an afterthought. Agents in Orbs and agent-to-agent communication arrived at the right time. They made it practical to divide the rebuild into focused pieces and involve the real homelab only when verification required it.

Amp did much of the heavy lifting, but it did not replace engineering judgment. I still chose the direction, decided which risks were acceptable and approved operations affecting real services or data. The agents investigated, implemented, tested and kept detailed records. That combination let me move faster without lowering the quality bar.

More importantly, it changed what felt possible. Instead of spending my time keeping old components alive, I could improve existing applications and try new ideas. The homelab was becoming a platform for experimentation again, rather than a collection of dependencies I was afraid to touch.

In the next post I will make that agent-friendly repository concrete before using it to build the new Kubernetes control plane safely.

Originally published on V for Vincent.

This time the trouble started with an unexpected power trip. After I restored the power, the homelab network did not come back. There was no obvious error anywhere. My routers were online, the modem had reconnected and all the usual lights looked normal. But services were still unreachable and my Kubernetes control plane could not start properly.

I started spot-checking the network configuration and noticed that the DHCP advertisement was wrong. One of my Ubuntu VMs had received a completely different IP address, gateway and DNS server. That was a useful clue, but I still had no idea which device was sending it or why.

So I asked Amp to investigate.

Amp connected to the VM over SSH and checked its addresses, routes and DHCP lease. It then went through the old system logs and found a very clear before and after. Before the power trip, the VM always received its reserved address from my main router. After the power came back, another device started giving it two-minute leases with the wrong gateway and DNS server.

At first I suspected Pi-hole, because it can also run a DHCP server. Amp checked its configuration and backups and ruled that out. It then identified the unexpected DHCP server as one of my TP-Link powerline extenders.

It turned out that the extender has a feature called Smart DHCP. If it cannot find another DHCP server, it starts one by itself to make initial setup easier. During the recovery from the power trip it could not see the router’s DHCP service, so it began handing out its own network configuration. Everything looked online, but devices were quietly being told to use the wrong gateway and DNS server.

Amp also worked out that the Kubernetes failure was not a separate problem. The control plane was configured with the VM’s reserved address, but that address no longer existed on the VM. Kubernetes quite rightly refused to start its networking.

After I disabled Smart DHCP on the powerline extender and rebooted the VM, Amp checked everything again. The lease was coming from the correct router, the VM had its reserved address and gateway, Pi-hole was resolving DNS, and every Kubernetes node was healthy again.

Without Amp, I would probably have spent hours rebooting devices and changing settings one by one. Instead, it followed the evidence from DHCP to Linux networking, Pi-hole and finally Kubernetes, then made sure the fix really worked.

What a gem! This is exactly the sort of problem where an AI coding agent is surprisingly useful. It did not just give me a list of commands to try. It investigated the actual machines, connected all the symptoms and stayed with the problem until my homelab was back.

Also, stay tuned! A version 2 build of my homelab is already in progress. I will write more about the new setup, what I am changing and everything I learn from rebuilding it from scratch.

Originally published on V for Vincent.