Building with AI

AI-driven coding is still software engineering. That's my hot take. And even though I'm the one saying it, I'm still coming around to it.

Lately, I've been building an application at work to streamline our AppSec and ProdSec work. There are a couple other quality of life features, too. But it's largely an exercise in some deep vulnerability management. So, lots of data.

But...on the topic of AI-driven coding...I don't have the time to code the whole thing by hand. Also, this thing can't be a year-long project. I have a day job. And side projects. And this is a side project at the day job. So, naturally I started leaning heavily on AI to do the coding.

At first, I thought, "Sure. It's vibe coding, but I need this to work not be pretty." Then I tried the Codex app. It's a new type of "editor" where the focus is on threads, git worktrees, managing your models, and not really seeing the code at all. I'll admit, this idea of an "agentic coding workflow" was jarring at first. But, considering I was already vibe coding the app because I needed it yesterday...it helped me justify leaning into this type of thing.

I quickly found, the problem is maintaining reliability and repeatable behavior.

Finding Guardrails

This application is a data-driven approach to AppSec and Product Security. So far, it's about 8GB of vuln findings, commit history, CVE data, etc. As expected with any application like this: I've had a few issues with data ingestion and other features along the way. While having an LLM handy to troubleshoot and fix things certainly made it quicker--it didn't remove the need for systems-level thinking.

Here are a few things I find myself considering:

  • Full test suites: integration tests, unit tests
  • API consumption practices: rate limiting, retry logic.
  • Graceful errors: Errors in long-running jobs don't mean "failure". Log and continue.
  • Data isolation.
  • Data modeling.
  • Security best practices.
  • Database backup and restore procedure. (And testing it!)
  • Introducing threat modeling.

I've found trying to think through these concerns, I've sort of naturally found myself looking at things like: skills, standard instructions (i.e. AGENTS.md), hooks, and even tools like Graphify.

Where to from here?

I finally feel like I have some momentum with this new idea of AI-driven coding. It's weird, but it's forcing me to think in terms of systems than I have before.

Systems, standards, "best practices", guidelines...all written for LLM ingestion, and standing as guardrails for the generated code. I still don't like having less control over the code, but I'm coming around to what this change actually means.