Inside the Build: A Foundry-aware AI agent powered by an open-source LLM and OSDK-backed tools

A field guide for Foundry developers, FDEs, and ML engineers shipping agents that live outside the platform but think with it.

Who this is for

Foundry developers, FDEs, and ML engineers who already work inside AIP and Agent Studio but are being pushed by a specific model requirement, a custom weights file, or GPU economics into hosting the agent somewhere else. This is also for the platform owner who's been asked "why can't we just run this in Foundry?" and needs a defensible answer either direction.

If you're standing up your first agent and don't have a specific model constraint, this isn't your post yet. Start in AIP Agent Studio and come back here only when you hit a wall you can name.

The pattern

The pattern shows up the same way across our builds. A customer wants an agent. The agent has a real job, not a chat demo, so it has to read and write through the Ontology. We start the scoping conversation inside AIP, where the agent should be. Two weeks later somebody on the call says one of three sentences and the architecture changes:

"We've fine-tuned a Llama variant on our own corpus and we need to use it."

"The model has to run in this specific quantization to fit on the GPU we already own."

"We've been waiting on H100 capacity in our Compute Module queue for three weeks."

That's it. That's the whole reason this guide exists. The agent still belongs to the Foundry workflow. The Ontology is still the system of record. The write-back, the audit trail, the human-in-the-loop review, all of that stays inside the platform. Only the model and the agent loop move outside. The tools the agent calls reach back into the Ontology over OSDK.

What follows is what we've learned doing this on real engagements, with the parts you can copy.

1. When to host an agent outside Foundry

Default position: don't. AIP and Compute Modules cover more agent work than most teams realize, and every layer of platform you give up is a layer you have to rebuild yourself. Governance, RBAC tied to the Ontology, write-back audit, eval harnesses, the AIP Logic surface where business users can read the rules. All of that you keep for free if the agent stays inside.

You leave the platform for one of two reasons.

Model control

AIP's catalog covers most agent work. The moment you need a specific open-source model, custom-trained or fine-tuned weights, or a particular quantization, self-hosting starts to look reasonable. Compute Modules can run any container, which technically gets you there, but GPU availability inside a shared platform queue is the part that bites. We've watched a queue back up for days during a tenant-wide busy period and the agent we were trying to ship just sat there.

If your team has a Llama variant they've fine-tuned on internal docs, or a quantized Qwen they need at a specific INT4 precision to fit the memory budget, or a model that simply isn't in the AIP catalog yet, that's a real model-control problem. Don't fight it. Host the model where you can control the runtime and reach back into Foundry over OSDK.

GPU economics

This one is more nuanced than it looks. Compute Modules share platform-level resource pools. That works fine for CPU-bound workloads and modest inference. The moment you need a specific GPU SKU, or an H100 for throughput, or steady access to a 4-GPU node for a 70B model, you're competing with every other workload on the platform. It can get slow. It can also get expensive in ways that are hard to forecast because the cost shape lives inside platform usage, not a per-hour bill.

Outside Foundry, GPU economics get more transparent but not necessarily cheaper. Lambda Labs, Modal, Replicate, Runpod, the on-prem cluster your IT team set up two years ago, all have their own cost shapes, cold-start behavior, and scale-down knobs. The trade is: you get predictability and control over GPU choice, you give up the integration tax of doing your own networking, secret management, and observability outside the platform.

The deciding question is rarely cost in isolation. It's can we get the GPU we need, when we need it, with the model we need. If yes inside Foundry, stay inside. If no, leave with a plan.

2. Defining the agent and choosing the model

This part gets skipped more often than it should. Teams jump from "we need an agent" to "which model do we use" without writing down what the agent is actually allowed to do.

Before you touch a model, write down three things.

1. What is the agent allowed to do. Read the Ontology? Write back? Call which functions? Hit which compute modules? Trigger which Actions? If you can't enumerate the allowed surface, you can't scope a tool layer.
2. What tools does it have. One per Ontology query you want to expose. One per function. Plus any side tools (search, math, format helpers) that don't touch Foundry. A flowchart helps more than a paragraph here. Draw it.
3. What shape are inputs and outputs. Free text in, structured JSON out, going where. The agent's outputs are the contract with the rest of the system.

The scope is part of the design. Don't let scope drift into "the agent can do anything." That's how you ship a thing that's confidently wrong on 12% of the cases and nobody notices.

Choosing the model

Once the scope is locked, the model choice has three real axes.

Tool-calling ability. A model that can write paragraphs and a model that can reliably emit tool calling syntax in the exact format your dispatcher expects are not the same model. Look up the tool-calling benchmark for your candidate before you commit. Open-source models with strong tool-calling priors save you weeks of dispatcher-tuning later.

Context window. Be honest about what the agent actually reads in a turn. If you're feeding it the last 5 Ontology object summaries plus the user message, 32k context is fine. If you're stuffing in a full document plus tool outputs from a multi-step chain, you'll feel the cost of 128k context fast. This is also highly relevant for the inference infrastructure you choose which is discussed below.

Latency under your expected load. A 70B model running at 4 tokens/sec on a single H100 is unusable for a chat agent and fine for an overnight batch agent. Profile under realistic concurrency before you sign off.

Deployment framework

For the runtime, the practical shortlist:

• llama.cpp and llama-cpp-python for GGUF models, excellent for custom deployment on on demand GPU platforms.
• vLLM for high-throughput GPU serving. PagedAttention plus continuous batching matters at scale.
• LM Studio for the laptop iteration loop before you commit to a server. With recent versions adding MLX support, it's also a credible deployment target on its own: Mac or server.

Memory math, briefly

Model size in VRAM is rarely just the parameter footprint. KV cache grows with context window. If you allow 128k context on a 70B INT8 model, the cache eats serious VRAM before the first request lands. Balance:

• GPU memory budget (the actual SKU you have)
• Model size and quantization
• Maximum context window you'll allow
• Concurrency target (each in-flight request has its own KV cache)
• Reasoning depth you need from the model

Also pay attention to GPU memory bandwidth, not just capacity. Throughput-bound workloads on a slower memory subsystem hit a ceiling you can't optimize past in software.

Respect the model's default config

This is the part that quietly destroys self-hosted agent quality. Some open-source models ship with a recommended default config: temperature, top_p, repeat penalty, sometimes a specific system prompt template. The model card on Hugging Face usually has it. Many open models are tuned to perform optimally under strict configs. If you copy a temperature setting from a different model because somebody on the team thinks 0.7 is the "creative" setting, you get sub-optimal outputs that look fine in dev and drift in production.

Below is the structural pattern you'll see across open models, treat the values as placeholders and pull the real ones from the model card:

self.llm = Llama(
    model_path=MODEL_PATH,
    n_ctx=131072,
    n_gpu_layers=-1,
    verbose=True,
)
response = self.llm(
    prompt,
    max_tokens=16384,
    temperature=0.0,
    top_p=0.95,
    repeat_penalty=1.1,
)

Three things to notice. n_ctx=131072 is the full context the runtime allocates per session, which is also what sets the KV cache ceiling. n_gpu_layers=-1 pushes every layer to GPU; if you're memory-constrained, this gets reduced and inference slows. temperature=0.0 plus top_p=0.95 plus repeat_penalty=1.1 is the recommended config from the model's own card. Use the card. Don't guess.

3. Implementing the OSDK-backed tool layer

A Foundry-aware agent that lives outside Foundry is, structurally, a reasoning loop with hands. The hands are the tool layer. The tool layer talks to the Ontology over OSDK.

There are several tool frameworks: LangChain, Haystack, LlamaIndex, the OpenAI Agents SDK, the Anthropic SDK's tool use loop. They all roughly do the same thing: take a model's tool-call output, dispatch to a Python function, return the result back into context.

Before you pick a framework, do the two prerequisite steps.

Expose the access points as OSDK first

If you did sections 1 and 2 right, you already know what the agent needs to access. Whether that's an Ontology object set, a Function, a Compute Module, an Action, or a dataset, expose each one through the Ontology SDK and test it from a plain Python script before the agent ever runs. Use the Python OSDK.

Two reasons. First, if the OSDK call fails standalone, debugging it from inside a reasoning loop is miserable. The model's tool-call output, the dispatcher, the framework, and the OSDK call all become suspect at once. Second, the OSDK contract you build here is what the rest of the agent code stabilizes against. Get it right early.

Create a long-lived Foundry token

Create a service-account-style token with scope limited to the Ontology entities the agent needs. The OSDK uses a token that is scoped only to the ontological entities your application wants to access, in addition to the user's own permissions. Tighten that scope. The model is going to call these tools without a human in the loop; the token's permissions are the safety boundary.

Rotate the token on a schedule. Store it in your hosting platform's secret manager, not in the container image. If your hosting platform doesn't have a secret manager, you picked the wrong hosting platform.

Write the tools narrow

Each tool is a function. It takes typed arguments. It calls OSDK. It returns JSON the model can read. You can add complex logic inside the function before returning, that's fine, but the surface the model sees is narrow.

One tool, one purpose. A single tool with twenty optional arguments confuses the model. Two narrow tools the model can pick between work better than one fat tool with a mode parameter, except when you genuinely have one operation with three minor variants. The example below is the borderline case where a mode parameter is reasonable because the underlying queries differ only by which Ontology query they hit.

def my_tool_name(self, arg1, arg2=None, mode="default"):
    """
    Wraps an OSDK Ontology query as an agent-callable tool.
    Returns a JSON string the model can read.
    """
    try:
        from foundry_sdk_runtime import AllowBetaFeatures
        client = self.get_osdk_client()

        with AllowBetaFeatures():
            if mode == "option_a":
                result = client.ontology.queries.my_query_a(
                    param_1=arg1,
                    param_2=arg2 or "",
                )
            elif mode == "option_b":
                result = client.ontology.queries.my_query_b(
                    param_1=arg1,
                )
            else:
                result = client.ontology.queries.my_query_default(
                    param_1=arg1,
                    param_2=arg2 or "",
                )

        # Flatten Ontology objects into plain JSON-safe dicts
        results = [{
            "id":    str(getattr(r, "id", "")),
            "label": str(getattr(r, "label", "")),
            "score": round(float(getattr(r, "score", 0)), 4),
        } for r in result]

        return json.dumps({
            "total_found": len(results),
            "mode": mode,
            "results": results,
        })

    except Exception as e:
        return json.dumps({"error": str(e), "total_found": 0, "results": []})

Three things worth pointing out. The flattening step matters: Ontology objects don't always serialize directly, and a tool that returns a TypeError mid-loop kills the agent. The error path returns a structured payload, not a stack trace, so the model can read the failure and decide whether to retry or escalate.

The description matters as much as the wrapper

This is where most self-hosted agents are weakest, and where AKOS's LLM Context Filter module came out of repeated tuning passes.

The tool's docstring or schema description is what the model reads to decide whether to call it. Lead with when to use the tool, not what it does. Name parameters the way the model would naturally describe them, not the way your database schema does. If you're using an open-source model that doesn't have strong tool-calling priors, include an exact call example inside the description. The model will pattern-match on it.

Concretely, a tool description like "Looks up part inventory by part_id" underperforms "Use this tool when the user is asking about whether a specific part is in stock or how many units are available. Example: lookup_part(part_id='X-4421'). Returns count, location, and last-updated timestamp."

Pick the framework based on the model

Hosted models with native tool-calling (GPT-4 class, Claude, Gemini) work well with LangChain or LlamaIndex out of the box, because the framework's parser is built around their output format.

Open-source models with custom output formats often fight the framework. Tool-call parsing fails silently, retries cascade, and you end up debugging escape characters in a chat template at 11pm. In that case, rolling your own dispatch loop is cleaner. It's 200 lines of code. You own the parsing logic. You can tune it to whatever output shape your model actually emits, including the malformed shapes it emits 3% of the time.

For chat-app deployment of the agent inside the customer's Foundry workflow, our AI Chatbot SDK App handles the OSDK-side wiring so the agent surfaces inside a Workshop module without rebuilding the chat UI.

4. Network, ingress, and deployment

The agent works in dev. Now you have to ship it.

Network policy

Wherever you host the model, the egress and ingress policies have to be set up and approved by your administrator. This is the step that derails timelines because nobody scopes it during architecture review. Two specific items:

• Egress from Foundry to your hosting platform. If the agent dispatches from inside Foundry (e.g., a Workshop module or a Compute Module makes the API call to your hosted endpoint), the platform needs an outbound network policy allowing your hosting platform's domain. Get the security team involved early.
• Ingress to your hosting platform. Most hosting platforms expose an HTTPS endpoint. Restrict by IP allowlist or token, ideally both. The Foundry token isn't a substitute; it's an OSDK-scope token, not an inbound auth token for your model endpoint.

Where the call originates

You have the agent. You know where the agent fits in the workflow. Now you trigger the call. Most commonly, the trigger is one of:

• A Workshop module calls the hosted endpoint via an AIP Logic function or a Compute Module-hosted dispatcher.
• An OSDK-backed external app (built on Developer Console) calls the endpoint from a server-side handler.
• A scheduled job inside Foundry kicks the agent on a cadence.

Whichever it is, the endpoint URL, the auth token, and the timeout settings live in Foundry as a Source or in the calling Compute Module's config, not hard-coded.

Scale-down and concurrency

Hosted GPU platforms (Lambda Labs, Modal, Replicate, Runpod, your own k8s cluster) all have cold-start behavior and scale-down knobs. Configure them deliberately:

• Cold-start tolerance. First call after scale-down might take 30 seconds or more depending on model size. If your agent is user-facing, that's a UX problem. If it's batch, it's fine.
• Scale-down threshold. How many seconds of idle before the platform shuts down the GPU. Too aggressive and you pay cold-start tax constantly. Too lenient and your bill grows.
• Concurrent request capacity. How many in-flight inferences per replica. Too high and the GPU thrashes its KV cache, latency spikes. Too low and you spin up unnecessary replicas.
• Maximum replicas. Set a ceiling so a bug in a calling loop can't autoscale you into a thousand-dollar surprise.

Document these settings somewhere the rest of the team can read. We've inherited self-hosted agents from previous vendors where the scale-down was set to 5 seconds and every cold start hit the user. Nobody knew why the agent felt slow.

For the Compute Modules scaling docs if you're keeping the dispatcher inside Foundry. For external hosting, your platform's docs are the source of truth.

The honest moment

Three things we've burned ourselves on running this pattern.

Open-source models fight the framework. We spent days on a build where the model's tool-call output was almost valid JSON but with a trailing comma the LangChain parser refused to forgive. We tried a system-prompt patch. We tried a model swap. We ended up writing our own dispatcher, which we should have done on day one. If your open-source model emits a non-standard tool-call format, accept it early. Don't fight the framework into accepting it; replace the framework with a 200-line dispatcher and move on.

GPU economics are not free, and the bill shape is unintuitive. We had a project where the scale-down was tuned for a single-user dev workload. The customer's pilot rolled out to 40 users. The bill in week one was 6x what we forecasted because cold-start tax compounded and the platform was keeping a warm pool the entire workday. We fixed it in an evening, but the customer saw the first bill before the fix. That was on us. Now we tune scale-down and concurrency against a realistic traffic shape before go-live, not after.

We've shipped this architecture when AIP would have done the job. One build, the customer asked for a self-hosted agent because somebody on their team had a strong preference for an open-source model they'd seen at a conference. We agreed too quickly. Eight weeks later we rebuilt the agent inside AIP Agent Studio because the model in question had been added to the catalog, the team didn't actually need the custom weights they thought they did, and the maintenance overhead of the external infra was eating an FDE's afternoon every week. We lost time. The customer didn't lose anything they were paying for because we ate the rebuild, but the right call would have been "let's hold for two weeks and check the AIP catalog first." We say it now. Sometimes it costs us a phase. It's still the right call.

What this looks like end to end

The shape of a real deployment:

1. Foundry hosts the Ontology, the workflow, the Workshop UI, the AIP Logic surface, the HITL review interface (we use the Human Validation Station module here), the audit log.
2. The agent loop runs on Lambda Labs (or Modal, or Runpod, or your on-prem cluster). The model is whatever the constraint forced you to: a fine-tuned Llama, a quantized Qwen, a custom-trained classifier-plus-LLM combo.
3. The tool layer is Python. Each tool wraps an OSDK call, returns JSON. A long-lived service token, scoped tight, lives in your hosting platform's secret manager.
4. Foundry calls out to the hosted endpoint when the workflow needs the agent. The agent reasons, calls tools, the tools reach back into the Ontology over OSDK. The agent's final output goes back into Foundry as an Action or a write-back to an object.
5. Every step is logged. Every write is reviewable. Every model and prompt version is tagged.

That's the architecture. The hard part is none of the boxes. The hard part is the model config, the tool descriptions, and the scale-down threshold. That's where the post should be useful.

TL;DR

• AIP's catalog covers most agent work cleanly. The reason to host an agent outside Foundry is almost always one of three: a model that isn't in the catalog, custom-trained or fine-tuned weights, or a specific GPU SKU you can't get reliable access to through Compute Modules.
• If you go outside, the agent still has to be Foundry-aware. That means an OSDK-backed tool layer, a long-lived Foundry token, and tools that return JSON the model can actually parse.
• Three things separate a working self-hosted agent from one that quietly fails in production: matching the model's default config (temperature, top_p, repeat penalty, context window), keeping each tool narrow with a description that leads with when not what, and tuning scale-down and concurrency for your real traffic shape.
• We've shipped this pattern in multiple builds. We've also rebuilt one of them back inside AIP because we didn't need to leave in the first place. The decision tree below is the post-mortem.

If you're standing one of these up and want a second pair of eyes on the architecture, talk to us. We'll send you the same deploy checklist we use internally. It's the one-page PDF that goes with this post.