Secure Design in an AI Enabled SDLC

Why Design Intent needs to become part of the Inner Development Loop?

I have written previously about how I think AI changes the shape of software development and one of the ideas I keep coming back to is the Inner Development Loop, or the Agentic Development Loop...call it what you wish 🙂

As coding agents become more capable, more of the development process is happening before we reach the traditional checkpoints of commit, pull request and CI/CD.

An agent can interpret a requirement, create a plan, write code, test it, inspect the result, remediate problems and repeat that process several times before a human sees the outcome.

Therefore, this also extends to where Secure Design needs to operate. Secure Design should still start before implementation, but it cannot stop there.

My view today is that security context needs to influence agents whilst design and implementation decisions are being made.

This is where my earlier thinking around Design Intent fits into the Inner Development Loop.

Originally, I'd proposed Design Intent primarily as something that happened at the start of the development process, and I still believe that should happen. What has changed in my thinking is that the resulting context should also remain available throughout the Inner Development Loop.

The problem starts before insecure code exists.

A lot of the discussion around AI-assisted development focuses on whether models generate vulnerable code. This is obviously very important, but it is only part of the problem.

Imagine asking an agent to:

Build an API that allows customer records to be shared with an external user.

The agent will more than likely produce a technically competent implementation. But there are more important questions sitting underneath that requirement.

Who should be allowed to share the record? Which parts can the external user see? Does access expire? Can the link be reused? What happens if the user's permissions change? How is tenant separation maintained? What needs to be recorded for audit?

These are not primarily coding questions...they are design decisions with potential security consequences.

If those questions have not been answered, the agent still needs to build something aligned to the provided intent. As a result, it will more than likely infer the missing pieces from the information available to it.

This is why I think saying that "AI writes insecure code" is too simplistic. AI can also very efficiently implement an ambiguous or insecure design.

Perfectly reasonable code can still implement the wrong security model and introduce business logic issues and other vulnerabilities.

Design Intent is more than a better prompt!

Instead, it is the structured expression of what we are building, how it should behave and the boundaries it must operate within.

This might include functional requirements, architecture context, trust boundaries, data sensitivity, authentication and authorisation expectations, known threat scenarios and specific security outcomes that the implementation must preserve.

This is not saying every change needs a heavyweight security design document. Quite the opposite.

The objective should be to give the agent only the minimum relevant context needed to make good decisions about the change it is implementing.

Let's say an organisation has a general security principle stating:

Access to customer data must follow least privilege.

Whilst useful, this is still very open to interpretation, especially by an agent. A much more complete Design Intent specific to the Product might instead say:

External collaborators may only access records explicitly shared with them. Access must be revalidated against the current grant on every request and must never allow access to resources belonging to another tenant.

This gives the agent something much more useful to design and test against.

If we were really good at this, we could also build in reference architectures and technical implementation patterns.

From security documentation to working context.

Naturally, this leads into how we should think about traditional security documentation and standards.

Don't get me wrong, we still need secure coding standards, architecture principles, approved technologies and engineering guardrails. But simply giving an agent access to everything the security function has ever written is unlikely to be the answer.

The context needs to become progressively more specific to the Product the agent is working on.

At the broadest level, we have organisational security requirements and engineering standards.

Below this sits product and system knowledge. This is likely to be how identity works, where the trust boundaries are, how tenancy is implemented, which data is sensitive and which architectural patterns we have chosen.

Then we have change-specific Design Intent. These are the threats, security requirements and expected outcomes that matter for the specific feature being built.

That final set of information becomes part of the agent's working context inside the Inner Development Loop.

I think this is an important distinction for security teams. The objective is not to produce an enormous security prompt.

But rather it is to make sure the agent has the right security knowledge at the point it needs to make a decision.

Security context cannot be static!

This is another part of this that I think becomes increasingly important. Why?

• Our understanding of a product changes.
• Threat models evolve as new functionality is introduced.
• Architecture changes.
• We discover new abuse cases.
• Security reviews identify assumptions we had not previously considered.
• Incidents and vulnerabilities teach us how the product actually behaves.
• Engineering teams establish better design patterns and retire older ones.

As a result, the security context available to an agent should evolve alongside that knowledge. This is why I do not see Design Intent being implemented simply through a static security.md file sitting in every code repo.

There will undoubtedly be relatively static guidance, but the more valuable context will come from current understanding of the product and the change being made.

Over time, security knowledge should become a living part of the development orchestration system. As the product teaches us more about its security requirements, that knowledge can influence the next change and those that come after.

Where I think this all leads?

I do not think the end state is developers manually writing security instructions every time they start an agent.

Instead, a development orchestration environment can progressively assemble much of this context itself, or reach out to something that can provide it as and when needed. Wit this, the developer describes what they want to build and the system understands which product components are changing.

Relevant architecture and security knowledge is identified, with only the applicable threats and design patterns brought into the working context for the agent to iterate against.

The resulting Design Intent shapes how the agent plans, builds, tests and validates the change.

The developer still owns the outcome and CI/CD still supports assurance, but security guides the implementation throughout the agentic Inner Development Loop rather than appearing for the first time afterwards.

Closing.

I still believe the central idea I started with behind Design Intent stands up. What has changed in my thinking compared with several months ago is where it lives and how it is consumed.

I originally thought about Design Intent primarily as an activity before code generation...like how we would threat model and define security requirements based on the outputs of that work. Now I think it is better understood as part of the Inner Development Loop itself.

Agents are inevitably going to make more design and implementation decisions throughout the development loop. If we want those decisions to be better, we need to make sure the right security context is available at the point it is needed.

It does not mean giving the agent every security document we have, or simply creating a bigger prompt at the start and hoping for the best. It means providing the relevant architecture, threats, trust boundaries, design patterns and required security outcomes for the product and feature it is actually working on.

Importantly, this context cannot remain static as our understanding of a product naturally evolves over time. This will happen through new threat models, architecture changes, security reviews, incidents and continuously improved design patterns.

For me, this is what Secure by Design starts to look like in an AI enabled SDLC.