AWS SageMaker Org Template · Service Catalog
CDK · GitHub Actions OIDC · Multi‑environment MLOps platform
Enterprise‑ready AWS AI platform provisioning using CDK, Service Catalog, and SageMaker Organization Templates with secure CI/CD (GitHub Actions + OIDC). Enables governed, repeatable ML workspace setup across Dev/Pre‑Prod/Prod.
Project summary
Pipeline‑1: IaC foundation
Category
Cloud + MLOps (AI Platform / Infrastructure as Code)
Industry
Cross‑Industry · Enterprise AI Platform
Domain
AI Platform Engineering / MLOps Infrastructure
Keywords
Problem & objective
Problem
- Inconsistent SageMaker environments across teams
- Security gaps (IAM/KMS) and non‑reproducible infra
- Slow onboarding and brittle CI/CD wiring for ML
Objective
- Automate AWS AI platform with CDK + Service Catalog
- Secure CI/CD (GitHub OIDC) + least‑privilege IAM
- Repeatable ML workspace setup across environments
Solution & architecture
Overview
Provision enterprise‑grade SageMaker Domain, Organization Templates (Service Catalog), IAM/KMS, S3, and environment stacks via AWS CDK. CI/CD with GitHub Actions + OIDC ensures secure, repeatable deployments. Foundation for training (Pipeline‑2) and deployment (Pipeline‑3).
Key components
- GitHub Actions
- AWS CDK (Python)
- SageMaker Domain, Projects, Org Templates
- Service Catalog product registration
- IAM OIDC roles (GitHub)
- KMS encryption, S3 assets, SSM config
Scalability & reliability
- IaC multi‑account/env ready
- Stateless CI/CD (no static creds)
- Quota‑aware provisioning
- CloudFormation rollback
- Least‑privilege IAM + KMS reduce blast radius
AI/ML & DevOps details
AI/ML type or DevOps focus :
DevOps / MLOps Platform Engineering (Infrastructure as Code for AI Platforms)
Models, pipeline, or automation implemented :
Infrastructure automation only — AWS CDK–based provisioning of SageMaker Domains, Organization Templates (Service Catalog products), IAM/KMS, and CI/CD wiring (GitHub Actions + OIDC).
(No ML models or training pipelines are implemented in Pipeline‑1; those are handled in Pipelines 2 & 3.)
CI/CD , containerisation or orchestration tools used
- GitHub Actions – CI/CD orchestration
- AWS CDK (Python) – Infrastructure pipeline as code
- OIDC (GitHub → AWS) – Secure CI/CD authentication
- AWS CloudFormation – IaC execution backend (via CDK synth/deploy)
- Docker – Asset bundling for CDK (where applicable)
(No Kubernetes/EKS or runtime container orchestration in Pipeline‑1; orchestration is infra provisioning only.)
Monitoring, Logging, or optimisation practices
- CI/CD Observability: GitHub Actions logs for build/deploy visibility and failure diagnosis.
- IaC Execution Logs: CloudFormation stack events for deployment status and rollback tracing.
- Security Auditing: IAM policy scoping and change tracking via CDK diffs.
- Cost Optimisation Guardrails: Instance defaults + AWS Budgets and quota checks to prevent overspend.
Skills & technologies
Primary (advanced)
- AWS CDK (Python)
- AWS IAM, OIDC, KMS, least‑privilege
- SageMaker platform setup (Domain, Org Templates)
- GitHub Actions CI/CD
- CloudFormation / IaC
Secondary tools
- S3 · SSM Parameter Store · KMS
- Docker (CDK asset bundling)
- Node.js / npm · Python virtualenv
- Git, YAML, Bash
Cloud & DevOps
Challenges & Outcomes
Key technical challenges faced:
- Designing reusable SageMaker Organization Templates that work across multiple environments and teams.
- Secure CI/CD authentication between GitHub Actions and AWS using OIDC without long-lived credentials.
- IAM least-privilege policy design for CDK deployments and Service Catalog products (avoiding over-permissioning).
- Service Quota constraints for SageMaker instance types during initial provisioning.
- Config management across environments (Dev/Pre-Prod/Prod) without hardcoding account/region values.
How they were resolved:
- Modular CDK stacks & constructs were used to make SageMaker Org Templates reusable and environment-agnostic.
- OIDC-based IAM roles were configured for GitHub Actions to enable secure, keyless CI/CD authentication.
- Least-privilege IAM policies were iteratively refined using CDK diff and deployment logs to scope permissions correctly.
- Service quota planning was documented and pre-approved to avoid provisioning failures during scale-up.
- Centralized configuration via SSM Parameter Store + env variables removed hardcoded values and enabled clean multi-env deployments.
CI/CD Architecture & YAML mapping
| Architecture Block | AWS CI/CD Construct |
|---|---|
| Source Repository | GitHub (IaC repo) |
| Source Trigger | GitHub Actions trigger (push / workflow_dispatch) |
| CI Runner | GitHub Actions Linux Runner (ubuntu-latest) |
| Build / IaC Execution | AWS CDK (cdk bootstrap / synth / deploy) |
| Infrastructure Backend | AWS CloudFormation (via CDK synth) |
| Artifact Storage | Amazon S3 (CDK assets / templates) |
| Platform Provisioning Target | Amazon SageMaker (Domain, Org Templates via Service Catalog) |
| Template Registry | AWS Service Catalog (Org Templates as Products) |
| Security & Auth | OIDC (GitHub → AWS) + IAM Roles & Policies |
| Secrets / Config | AWS SSM Parameter Store (+ env vars) |
| Encryption | AWS KMS (platform resources) |
| Logs & Monitoring | GitHub Actions Logs + CloudFormation Stack Events |
AWS DevOps CI/CD – Reference Architecture
Pipeline-1: IaC / Platform Provisioning
Architecture Flow:
- Developer pushes changes to GitHub (IaC repository).
- GitHub Actions workflow is triggered (push / manual dispatch).
- GitHub Actions assumes AWS IAM Role via OIDC (no static credentials).
- CDK bootstrap initializes target AWS account/region (one-time).
- CDK synth generates CloudFormation templates.
- CDK deploy provisions:
- SageMaker Domain & Organization Templates (Service Catalog Product)
- IAM Roles & Policies
- KMS Keys
- S3 Buckets (CDK assets)
- CloudFormation executes stacks with rollback on failure.
- Logs and status are available in GitHub Actions + CloudFormation events.
YAML Mapping (GitHub Actions → CDK):
- Trigger: on push / workflow_dispatch
- Auth: aws-actions/configure-aws-credentials (OIDC → IAM Role)
- Runtime: setup-python (for CDK app) + Node.js (CDK CLI)
- Steps:
- cdk bootstrap (one-time per account/region)
- cdk synth (template generation)
- cdk deploy (Service Catalog product + SageMaker Org Template)
Security & Guardrails:
- OIDC-based IAM roles (no secrets in CI)
- Least-privilege IAM policies for CDK execution
- KMS encryption for platform resources
- Environment isolation (Dev / Pre-Prod / Prod)
Assets & references
MLOps CDK GitHub Action
Infrastructure-as-code and GitHub Actions workflow for Pipeline‑1 platform provisioning.
View GitHub repoPsitron ML Build
Build repository for the ML pipeline stage and related implementation assets.
View GitHub repoPsitron ML Deploy
Deployment repository for ML release workflows and production delivery assets.
View GitHub repoOutputs of Pipeline‑1
- Service Catalog product registered (SageMaker Org Template)
- SageMaker Domain + Studio available
- Seed repos created (train/deploy)
- IAM roles + OIDC wired
Study material resources
CDK templates, IAM policies, architecture diagrams (restricted access)
Request Study Material