Skip to content

Caster Logo Caster: Coding a Topology

Overview

Caster is the deployment component of the Crucible framework. It uses Terraform, an open source Infrastructure-as-Code tool, to define and apply infrastructure configurations.

Caster provides a web interface where exercise developers create, share, and manage topology configurations.

Early versions of Caster exposed Terraform configurations and outputs directly through the UI and primarily supported advanced users. Ongoing development adds higher-level features that reduce the need to write Terraform manually while still allowing direct configuration when users need it.

Terraform Integration

For more information on native Terraform constructs used in Caster, please refer to the Terraform documentation.

The Crucible Terraform Provider enables programmatic management of Crucible resources through Terraform configurations.

Configuration

Configure and deploy Caster using the Caster Helm Chart. The Helm Chart README provides detailed instructions for all deployment settings.

Classification Banner

Caster UI supports an optional, customizable classification banner that displays persistently at the top of the application. The banner can show classification labels (such as "UNCLASSIFIED" or "SECRET"), maintenance messages, or any other persistent notification. Configure the banner through HeaderBarSettings in the Helm chart. See the Classification Banner section of the Caster Helm Chart README for configuration details.

Example classification banner with an example message

Terraform Versions

The Terraform Version dropdowns in Caster list only the versions installed in your instance. Which versions are available, and which one is the default for new workspaces, are deployment settings rather than UI settings:

  • Default version: The version Caster assigns when neither the workspace nor any of its parent directories specifies one. Caster marks it (Default) in the dropdown.
  • Maximum parallelism: The Parallelism field only accepts values below this limit.

Configure both through the Caster Helm Chart. To choose a version for a directory or a workspace, see Directory Settings and Workspace Settings.

Note

The Crucible Terraform Provider requires Terraform 1.0 or later. Configurations that use it need a workspace set to a 1.x version.

Run Environment and Secrets

Terraform runs don't get the Caster API's environment variables unless the deployment lists them. Provider credentials reach runs through Environment Sets instead. The deployment gives the API each credential as an environment variable whose name marks it as a secret, by default one starting with CASTER_SECRET_. Administrators then decide which projects get it.

  • Caster never shows a secret's value, and masks secret values in run output.
  • A run whose secret is missing, or too short to mask safely, fails before Terraform starts.

Configure the secrets, and the variables every run inherits, through the Caster Helm Chart. The chart README explains how to move a deployment that relied on runs inheriting every variable.

For the rules a set's entries follow, how runs get them, and the security model, see Environment Sets in the Caster API repository.

Infrastructure Connections

A connection is a hypervisor Caster reads from: a vSphere vCenter or a Proxmox VE cluster. Caster lists each connection's VM templates, ISOs, networks, and datastores in the Inventory panel of the projects you assign it to. An Environment Set can read a connection's endpoint and credentials into a run, so you configure each credential once.

Connections are deployment settings: add them through the Caster Helm Chart. Caster never shows a connection's address or credentials, and a connection's settings reach a run only through an Environment Set.

For connection settings, see the Caster API repository: Infrastructure Inventory covers each provider's settings, which objects Caster reads, and filtering them by location, name, ISO location, and datacenter.

Permissions and Roles

Sets of permissions control access to features in Caster. Permissions can apply globally or per Project.

Examples of global permissions include:

  • CreateProjects: Create new Projects
  • ViewProjects: View all Projects and their Users and Groups
  • ManageUsers: Make changes to Users

Users with View or Manage permissions for an administration function (for example, ViewVLANs or ManageWorkspaces) can open the Administration area. However, they see only the sections they have permission to access in the sidebar menu.

These permissions control the infrastructure features:

  • ViewInfrastructure: View every connection and its inventory
  • ManageInfrastructure: Assign connections to projects and project groups, and create Environment Set entries that read a connection
  • ViewEnvironmentSets: View every Environment Set, never secret values
  • ManageEnvironmentSets: Create, edit, and delete Environment Sets, and assign them to projects and project groups
  • ManageProjectGroups: Create, edit, and delete project groups, and manage any group

You can view all available permissions in the Roles section of the Administration area.

Grant ManageProjectGroups With Care

A holder of ManageProjectGroups can make themselves a manager of any group, and so reach every project in a group and the credentials its Environment Sets give. Grant it as carefully as ManageEnvironmentSets.

Roles

You apply permissions to users by grouping them into roles. Caster supports two types of roles: System Roles and Project Roles.

System Roles

Each user can have one system role that provides global permissions across all of Caster.

Default system roles:

  • Administrator: All permissions within the system.
  • Content Developer: Has the CreateProjects permission. Users with this role can create and manage their own Projects, but cannot change global settings or other users' Projects.
  • Observer: Has all view permissions. Users with this role can view everything in the system, but cannot make changes.

Users with the ManageRoles permission can create custom system roles in the Roles section of the Administration area.

Project Roles

When you add a user to a Project, you assign a project role that defines what they can do within that specific Project.

Available project roles:

  • Manager: Perform all Project actions, including managing user access. When someone creates a new Project, they automatically become the Manager for that Project.
  • Member: View and edit all objects within the Project.
  • Observer: View all objects within the Project, but cannot make changes.

You can't create custom project roles.

Administrator Guide

Caster administrators use the Administration View to manage projects, workspaces, VLANs, modules, users, roles, connections, Environment Sets, project groups, and groups.

Administration View

Across the Crucible exercise applications, the Administration View is where privileged users configure the platform and control access. It includes user and team management, role and permission assignment, and setup and maintenance of app-specific templates and content. It is where admins prepare and manage the environment so events run smoothly for participants.

Accessing the Administration View is the same in all Crucible exercise applications: expand the dropdown next to your username in the top-right corner and select Administration.

The Administration dropdown in the top right-corner

Projects

The Projects pane in the Administration View displays all projects in the system. Administrators can search for projects, open them, rename them, and delete them.

The table lists each project's Name, Description, and Created date. Turn on My Projects to limit the list to projects you are a member of.

The screen shot below shows the Projects pane in Caster's Administration View.

Caster Administration View, Projects pane

Add a New Project

  1. In the Projects pane, click +.
  2. Enter a Name and click Save.

Search for a Project

In the Search Projects field, type all or part of a project name.

Edit an Existing Project

Click the project name to open the Users/Groups and Project Members panes. From here, you can add users and groups to the project as members, assign their roles, and remove them.

  • To add a member, find the user or group in the Users/Groups pane and click Add.
  • To change a member's role, select a new role from the Role dropdown in the Project Members pane.
  • To remove a member, click Remove in the Project Members pane.

For a description of each project role, see Project Roles.

Open a Project

Click the Open icon to open the project in a new browser tab, where you can browse and manage the project's directories, workspaces, files, and designs.

Rename a Project

  1. In the project row, click the Pencil icon.
  2. Edit the Name and click Save.

Delete a Project

  1. In the project row, click the trash can icon.
  2. Confirm Delete.

Workspaces

The Workspaces pane in the Administration View lets administrators monitor active runs and control workspace operations system-wide.

The screen shot below shows the Workspaces pane in Caster's Administration View.

Caster Administration View, Workspaces pane

  • Disable Workspace Operations: Toggle to prevent workspace operations that require locking. Use this before restarting the Caster.Api service to avoid corrupting Terraform state. The setting resets when the API restarts.
  • Active Runs: Displays all currently running workspace operations. Use the Search field to filter by workspace. The table shows Created At, Destroy, Status, Workspace ID, and Actions.

VLANs

Caster Administration View, VLANs pane

Caster manages VLAN IDs by creating pools of 4096 VLANs and dividing them into partitions. When a user requests a VLAN from a partition, Caster assigns an unused ID and marks it in use until the user releases it.

An administrator assigns a partition to a project or a project group, or makes it the system default. A project's VLAN requests use its own partition, then its group's, then the default partition.

  • VLANs can include tags for organization, and users can request a VLAN by tag
  • Users can request a specific VLAN ID within a partition
  • VLANs marked as reserved (including 0, 1, and 4095, reserved by default) are never assigned

Caster doesn't configure any hypervisor or switch. It only hands out VLAN numbers and tracks which are in use, so the same pool works for vSphere, Proxmox, software-defined network segments, or a physical switch. Your Terraform puts the number where that platform expects it.

Caster prevents collisions only between Caster runs. TopoMojo, Player networks, and networks built by hand choose their own VLANs. Give a pool a range nothing else uses, or mark the VLANs that other systems use as reserved.

Use a VLAN in Terraform

Request a VLAN with the crucible_vlan resource from the Crucible Terraform provider. Caster marks the VLAN in use when the run creates the resource, and releases it when the run destroys it.

terraform {
  required_providers {
    crucible = {
      source = "cmu-sei/crucible"
    }
  }
}

variable "project_id" {
  description = "The Caster project whose partition supplies the VLAN"
  type        = string
}

# A VLAN from the project's partition
resource "crucible_vlan" "blue_net" {
  project_id = var.project_id
}

Where the VLAN comes from:

  • project_id: the partition assigned to that project. The project's ID is the last part of its address in Caster, .../projects/<id>.
  • partition_id: that partition. Copy the ID with the copy button on the partition in the VLANs pane.
  • Neither: the default partition. If no partition is the default, the request fails.

To ask for a particular VLAN, add tag (a VLAN with that tag in the partition) or vlan_id (that number in the partition). The request fails if that VLAN is in use or isn't in the partition.

Use crucible_vlan.<name>.vlan_id wherever your provider takes a VLAN ID:

# vSphere: a distributed port group on that VLAN
resource "vsphere_distributed_port_group" "blue_net" {
  name                            = "blue-net"
  distributed_virtual_switch_uuid = data.vsphere_distributed_virtual_switch.dvs.id
  vlan_id                         = crucible_vlan.blue_net.vlan_id
}

# Proxmox (bpg/proxmox): a VM NIC tagged with that VLAN
resource "proxmox_virtual_environment_vm" "workstation" {
  # ...
  network_device {
    bridge  = "vmbr0"
    vlan_id = crucible_vlan.blue_net.vlan_id
  }
}

With the Telmate/proxmox provider, set tag = crucible_vlan.blue_net.vlan_id in the VM's network block.

Reserved and Tagged VLANs

Caster hands out VLANs only from a partition, so where a VLAN sits decides what its settings do:

Setting On a VLAN in a partition On a VLAN not in any partition
Reserved Stays in the partition, but is never handed out Is never handed out or added to a partition by count
Tag Lets Terraform request it with tag Does nothing until the VLAN is moved into a partition

Adding VLANs to a partition by count takes the lowest-numbered free ones, not tagged ones. To put a tagged VLAN in a partition, select it in the pool's unassigned VLANs and reassign it.

The VLANs pane has two tabs:

  • Pools: Create and manage VLAN pools and the partitions within them.
  • Projects: Review which partition each project draws VLAN IDs from.

Add a VLAN Pool

  1. In the VLANs pane, on the Pools tab, click New Pool.
  2. Enter a Name and click Save.
  3. Expand the new pool to add partitions to it.

Modules

In Caster, a module is a Terraform construct. A module is a container for related resources used together. It lets you describe infrastructure at the architecture level instead of defining each individual object.

A module packages one or more resources into a single block and exposes required or optional variables. For example:

  • A generic virtual machine module that exposes values such as TeamId, Networks, and ExerciseId.

    • TeamId: Sets guestinfo.teamId in extra_config.
    • Networks: Creates a Network Interface Card (NIC) for each specified network and assigns it to the specified network VLAN.
    • ExerciseId: Appends the exerciseId to the name of the VM for use with on-demand exercises that require unique naming.

  • A domain controller module that requires values such as domain name, IP address, and administrator credentials.

  • A module that defines an entire Cyber Flag enclave.
  • A module that defines a configurable GreySpace environment.

Caster discovers modules from two kinds of source, configured in the Caster API settings. Either, both, or neither can be configured:

  • Git module sources (see Git Module Sources in the Helm chart README): any git repository Caster can clone, such as GitLab, GitHub, or a self-hosted git server. Every git tag is a module version. A repository is either one module, or (with the Subdirectories layout) a set of modules, one per top-level directory. A tag such as network-segment/v1.2.0 versions only that module.
  • GitLab group (legacy): every GitLab project in the configured Terraform-Modules group and its subgroups that is visible to the API GitLab token and has at least one version.

Note

The module inputs and outputs files must be JSON (variables.tf.json and outputs.tf.json). Caster does not read modules written only in HCL. An optional caster.json sets a module's display name and description.

Private repositories need git credentials both where Caster discovers modules and where Terraform runs, because terraform init downloads each module from its git address. When Terraform runs as Kubernetes jobs, the job pods need them too. See docs/Modules.md in the Caster.Api repository and the Caster Helm chart's gitcredentials value.

Caster can add or refresh modules in three ways:

  • Each module list request checks every source for new tags. Refresh the browser if a new module or version does not appear. If a source cannot be reached, Caster keeps showing the modules it already has and shows the source's error on the Modules page.
  • An administrator can update the modules from every source at once.
  • An administrator can add or refresh the modules of one source, or one module, by its ID. This re-reads it even if no tag changed.

While working on Caster Modules:

  • Search: Type all or part of a module name to filter the list.
  • External Module ID: Enter a module source name, a source name and subdirectory (crucible-modules/network-segment) for one module in it, or for a GitLab source, a GitLab project ID. Then click the Add/Update the Modules from this source icon to add or refresh it.
  • Add/Update the Modules from every source: Click the cloud icon to check every configured source at once.
  • Copy: Click the copy icon to copy the module's ID to the clipboard.
  • Versions: Displays the number of versions available for the module in the git repository. A module must have at least one version before you can add it to a Design.
  • Path: Displays the module's path. For a git module source, the path starts with the source name.
  • Date Loaded: Displays when Caster last loaded the module from the repository.
  • Delete: Click the trash can icon to delete a module.

Selecting a module opens a form where you choose a version and enter the required variable values. After you select Submit, Caster generates the Terraform module block, which you can copy into the configuration file.

Users

Users appear in Caster after they authenticate through the Identity provider and open Caster in a browser.

The screen shot below shows the Users pane in Caster's Administration View.

Caster's Administration View, Users screen

Add a New User

  1. In the Users pane, click Add User.
  2. Enter the user's User ID and Name.
  3. Role: (Optional) Select a system role for the user.
  4. Click Create to save, or Cancel to discard.

Assign a Role to a User

In the user row, in the Role column, select a role from the dropdown (None Locally, Observer, Content Developer, or Administrator). For a description of each role, see System Roles. None Locally means the user has no system role assigned in Caster. They can still receive permissions through a group or a project role.

Roles

The Roles pane in the Administration View is where administrators review and build roles. It has two tabs:

  • Roles: System roles and the permissions assigned to each. The three default roles--Administrator, Content Developer, and Observer--appear here alongside any custom roles. For a description of each, see System Roles.
  • Project Roles: The fixed set of project roles and their permissions. You can't add or edit project roles. For a description of each, see Project Roles.

Add a Custom System Role

  1. In the Roles pane, on the Roles tab, click +.
  2. Enter a Name.
  3. Select each permission the role grants. Selecting All grants every permission.
  4. Click Save.

Connections

The Connections pane lists the infrastructure connections the deployment configures, with each one's hosts and storage use, and the projects and project groups that see it. Viewing it requires ViewInfrastructure.

A connection's inventory appears in a project's Inventory panel only when the connection is assigned to the project or its project group. Assigning a connection gives the project its inventory, not its credentials; runs get credentials only from Environment Sets.

Assign a Connection

Assigning requires ManageInfrastructure.

  1. In the Connections pane, click Assign on the connection.
  2. On the Projects tab, select the projects that should see the connection, and clear the ones that shouldn't.
  3. Use Search to find a project.
  4. On the Project Groups tab, select or clear groups the same way.
  5. Click Save.

To refresh a connection's inventory now rather than at its next scheduled read, click the Refresh icon on the connection.

Environment Sets

An Environment Set is a named group of variables that Caster gives to every Terraform run in the projects you assign it to. Use sets to give projects provider credentials and settings without writing them in Terraform files.

The Environment Sets pane has two tabs:

  • Sets: Every set, its entries, and the projects and project groups that get it.
  • Secrets: The secrets the deployment gives the API, whether each one is available, and the sets that use it. Caster never shows a secret's value.

Each entry in a set has:

  • Kind: Environment variable, set in the run under its name, or Terraform variable, set as TF_VAR_<name>.
  • Name: The variable's name in the run.
  • Value source:
  • Secret: One of the API's secrets, chosen by name.
  • Plain value: A value stored in Caster. Plain values are not secret: members of every project that gets the set can read them.
  • Connection: A field of a connection, such as its URL or API token, read when each run starts.

A variable name can come from only one set on a project. Caster refuses an assignment or edit that would give a project the same variable from two sets.

Note

A secret can't be a Terraform variable. Terraform saves variable values in plans and state, so give credentials to the provider as environment variables.

Add an Environment Set

  1. In the Environment Sets pane, click +.
  2. Enter a Name and, optionally, a Description, then click Save.
  3. Expand the set and click Add Entry.
  4. Choose the Kind, enter the variable Name, choose the Value source, and supply the secret, value, or connection field.
  5. Click Save. Repeat for each entry.

Create a Set From a Connection

Create Set from Connection builds a set holding a Terraform provider's environment variables for one connection, read from the connection's settings. Creating one requires ManageInfrastructure as well as ManageEnvironmentSets.

  1. In the Environment Sets pane, click Create Set from Connection.
  2. Select the Connection and the Terraform provider.
  3. Optionally, enter a Name. Without one, Caster names the set after the connection and provider.
  4. Click Create.

Assign an Environment Set

  1. Expand the set and click Assign.
  2. On the Projects tab, select the projects that should get the set, and clear the ones that shouldn't.
  3. On the Project Groups tab, select or clear groups the same way.
  4. Click Save. Caster checks every change together, and saves all of them or none.

A set's variables reach every workspace in its projects at their next run.

Copy an Environment Set

  1. In the set's row, click the Copy icon.
  2. Click Copy to confirm.

Caster names the copy "name (copy)" and assigns it to no project.

Edit Several Sets Together

  1. Select two or more sets with the checkboxes in the Select column.
  2. Click Edit Selected.
  3. For each variable, edit its value, delete it, or select Add where missing to add it to the selected sets that don't have it.
  4. When the sets give a variable different values, Caster asks for the value to add first.
  5. Click Save.

Delete an Environment Set

  1. In the set's row, click the trash can icon.
  2. Review the projects that get the set, then click Delete.

Those projects' next runs don't get the set's variables.

Project Groups

A project group is a set of projects that share memberships, Environment Sets, connections, and a VLAN partition. What a group carries adds to what each project has of its own. A project belongs to at most one group.

  • Memberships: A user or group with a role in the project group has that role in every project of the group.
  • Environment Sets: Every project in the group gets the group's sets as well as its own.
  • Connections: Every project in the group sees the group's connections' inventory as well as its own.
  • VLANs: A project uses its own partition first, then its group's.

Group managers don't need the Administration View. A group member whose role is Manager, or has the Manage Project permission, can open Project Groups from the dropdown next to their username.

Add a Project Group

  1. In the Project Groups pane, click Add Project Group.
  2. Enter a Name and, optionally, a Description, then click Save.

Manage a Project Group

Click a group to expand it. Each tab saves on its own.

  • Projects: Select the projects to include, then click Save. A project already in another group moves out of it. Click New Project in Group to create a project inside the group.
  • Memberships: Add users and groups and choose their roles, as for a project.
  • Environment Sets: Select the sets the group's projects get. Requires ManageEnvironmentSets.
  • Connections: Select the connections whose inventory the group's projects see. Requires ManageInfrastructure.
  • VLANs: Choose the group's partition. Visible only with ManageVLANs.

Groups

Groups are collections of users that administrators assign roles to or add to projects as a single unit instead of managing users individually. Administrators use groups to simplify access management for teams. For example, an administrator can add a group such as White Cell to multiple projects. Any user placed into the matching group in the identity provider automatically gains the same project access in Caster. This allows administrators to manage group membership in one place while Caster applies the access across all of the applications.

Administrators can create new groups, search for and edit existing groups, and delete groups here. Administrators add users to groups manually or, if enabled in the environment, match them to group names from the Identity provider token.

Creating a New Group

  1. In the Caster Administration View, select Groups.
  2. Click the + sign to add a new group.
  3. Type in a Name and click Save.
  4. Expand the new group.
  5. Under Users, search for a specific user and click Add to place that user into the Group Members column.

Renaming an Existing Group

  1. In the Caster Administration View, select Groups.
  2. Click the pencil icon and edit the group Name.
  3. Click Save.

Deleting a Group

  1. In the Caster Administration View, select Groups.
  2. Click the trash can icon then Delete.

User Guide

Users access assigned projects, create and edit Terraform files, and deploy configurations to workspaces.

Project

The top-level construct in Caster is a project. The project is a way to organize and categorize similar environments for multiple files and directories within Caster. The main screen of Caster displays a list of the projects available and allows a user to create a new one.

A project can:

  • Categorize large events
  • House directories, workspaces, and subdirectories

Users can add new projects, rename projects, save projects, and also import and export projects. A project's landing page in Caster has a navigation panel for easy movement within the project's files, workspaces, and directories.

Add Directory

Add Directory lets you create a new directory at the same level as the above projects.

Export Project

Export Project lets you export the project as an archive file. Select the Archive Type (default: Zip), then click Export. Enable Include Ids to include directory IDs in the export.

Import Project

Import Project lets you import a project from a zip file. Click Choose File to select the zip file, then click Import. Enable Preserve Ids to append IDs to directory names during the import.

Inventory Panel

The Inventory panel in a project's navigation lists what is on the hypervisors assigned to the project: VM templates, ISOs, networks, datastores, and vSphere content library items.

  • If the project has more than one connection, choose one from the connection list at the top of the panel.
  • Type in Search inventory to filter the list.
  • Click Copy Terraform on an item to copy a Terraform snippet that refers to it, or copy its name.
  • Click Refresh inventory to read the hypervisor again.

If the panel lists nothing, ask an administrator to assign a connection to the project or its project group.

Environment

Click Environment in a project's navigation to see what the project's runs get:

  • Environment Sets: The sets assigned to the project, each variable they set, and whether each secret is available. Caster never shows secret values.
  • Connections: The connections whose inventory the project's Inventory panel lists.

Caster marks what the project gets through its project group "via group name".

Administrators with ManageEnvironmentSets or ManageInfrastructure can click Change Assignments to add and remove the project's sets and connections in one save. When a chosen set reads a connection the project doesn't show yet, Caster offers to add the connection too. Change what comes through the group on the group itself.

Files

Files represent text files to eventually put onto a file system and use with the Terraform command line tool. You can name and edit files through Caster, but file extensions are important and have specific meaning to Terraform.

  • .tf A configuration file that defines resources, variables, etc., in a Terraform configuration.
  • .auto.tfvars Contains the values used for variables defined in .tf files.

Locking and Unlocking Files

When working with files in Caster, CTRL+L locks/unlocks a file to prevent others from editing that file simultaneously. When locked, the file icon appears as a dashed line. When unlocked, the file icon appears solid. Administrators can also lock files. A file is administratively locked to prevent anyone from changing that file. A lock icon in the top right corner of the file edit screen denotes that the file is administratively locked. CTRL+S saves a file.

When another user has a file open for editing, a red person-with-pencil icon appears in the top right corner of the file edit screen. Hover over it to see who has the file: This file is being edited by name. Project Managers, and users with the global ManageProjects permission, can release that user's lock: click the icon (Force release the file from name), then click Force Release in the confirmation. The other user loses any unsaved changes, so contact them first. Force release clears only the edit lock; it does not remove an administrative lock.

See the official Terraform Documentation for more details on supported file types and extensions. In the future, Caster may provide more guidance on allowed types and content of files.

Workspaces

Caster workspaces

A workspace represents a specific instance of a deployed Terraform configuration. Use the same configuration to deploy virtual machines to multiple workspaces that differ only by the values set to certain variables. For example, define a configuration once for an enclave in a Cyber Flag exercise, and then deploy to flag00 through flag30 workspaces - each creating a copy of the enclave.

Workspaces can contain files, which extend the configuration of the directory for that specific workspace. This might include files specifying values for variables defined in the directory, or additional resources deployed only for that workspace.

A workspace is where users:

  • Create an instance of a Terraform configuration
  • Run their plans. (Runs are a specific instance of a Terraform plan)
  • Manage the differences and the variables in their environments

Users can access workspaces from a project's navigation pane in Caster. They can add additional files—but not directories—to a workspace.

The workspace view displays all planned and applied runs. Runs shaded in red represent destroy operations, while runs in white indicate other status classifications.

Users can Plan, Destroy, Apply, Taint, and Reject operations in real time in the workspace view.

Workspace Settings

Each workspace has its own Terraform settings. Caster applies them to every run in that workspace.

  1. In the project navigation panel, right-click the workspace and select Edit.
  2. Change any of the following:
  3. Name: 90 characters or fewer. Use only letters, numbers, -, _, and ..
  4. Terraform Version: The Terraform version this workspace runs. The dropdown lists every version installed in your Caster instance, newest first, with the instance default marked (Default).
  5. Parallelism: Limits the number of concurrent Terraform operations. Defaults to 10 when empty. Only set this to troubleshoot provider errors.
  6. Azure Threshold: The number of consecutive failed destroys in an Azure workspace before Caster tries to recover by removing azurerm_resource_group children from the state.
  7. Click Save.

Note

Save stays disabled until you change a value. Leave Parallelism and Azure Threshold empty to use your instance's defaults.

The workspace view displays the workspace's current Terraform version next to the Plan and Destroy buttons.

Creating a workspace prompts only for a name. Caster sets the new workspace's Terraform version, parallelism, and Azure threshold from its directory at that moment. See Directory Settings. To change them afterward, use Edit.

Terraform Execution

Caster runs Terraform in one of two modes, selected in the Caster API settings (see Kubernetes Jobs in the Helm chart README):

  • In-process (default): Caster.Api runs a Terraform binary inside its own container. Each available version lives in its own directory under the configured binary path.
  • Kubernetes jobs: Caster.Api runs each operation as a Kubernetes job, using a Terraform container image tagged with the selected version.

In either mode, stopping or restarting Caster.Api while a Terraform operation is in progress can corrupt the Terraform state.

In order to avoid this, a System Administrator should follow these steps in the Caster UI before stopping the Caster.Api container:

  • Navigate to Administration, Workspaces
  • Disable Workspace Operations by clicking the toggle button
  • Wait until all Active Runs complete

Directories

The top-level construct within a project is a directory. A project can contain many directories. Directories contain files that make up a particular Terraform configuration, workspaces that represent a specific instance of that configuration, and subdirectories. Directories are primarily for organization and shared resources.

Directory Hierarchies

Directories can contain subdirectories to create a hierarchy of directories and the configuration files contained therein. When creating a run, the files in the workspace, the workspace's directory, and all parent directories merged and pass to Terraform as a single configuration. This eliminates redundancy when developing many environments that are similar or share a set of common variables or data across many configurations. For example, a large deployment might have a top-level directory defining global variables vlan ids and team ids, and subdirectories defining resources using those variables.

Right-click a directory in the navigation panel on a project's main Caster page to Edit, Delete, Export, or Import it. Renaming a directory is part of Edit. See Directory Settings.

Peer directories (directories that fall outside a parent directory) are not included in a run.

Directory Settings

Directory settings are the defaults Caster hands to workspaces created in that directory.

  1. In the project navigation panel, right-click the directory and select Edit.
  2. Change any of the following:
  3. Name: Rename the directory.
  4. Terraform Version: The version Caster assigns to new workspaces created in this directory.
  5. Parallelism: Limits the number of concurrent Terraform operations. Caster sets this value on new workspaces created in this directory.
  6. Azure Threshold: The number of consecutive failed destroys in an Azure workspace before Caster tries to recover by removing azurerm_resource_group children from the state. Clear the Enabled checkbox to have workspaces ignore the threshold.
  7. Click Save.

Edit Directory dialog with the Name, Terraform Version, Parallelism, and Azure Threshold fields

Directory Settings Apply at Workspace Creation

Caster reads directory settings only when it creates a workspace. It uses the nearest value set on the workspace's directory, then its parent directories in turn, and falls back to the version configured for your Caster instance. Changing a directory setting later does not change workspaces that already exist. Edit those workspaces directly. See Workspace Settings.

To set the Terraform version for a whole branch of a project, set it on the parent directory before creating the workspaces under it.

Designs

Designer provides a graphical user interface for creating and editing terraform deployments through the use of modules.

When you open a project, you can create a design and add modules backed by Git to that design. You can also create variables for use in the modules settings.

Caster Tips

The following tips help you get the most out of Caster.

Crafting Terraform Code

This topic is for anyone who manages a Crucible instance who wants to configure their Terraform provider installation for Caster. You can configure Terraform to only download certain providers from the Internet and use them from a local File store.

Refer to HashiCorp's Terraform documentation: Command Line Interface (CLI) Configuration File Provider Installation.

For your reference, below is the .terraformrc file currently implemented in the SEI's CyberForce instance of Caster.

In the SEI's instance, we want to use any locally downloaded plugins in the sei or mastercard namespace. In addition, we can download any of the hashicorp namespace providers in the direct section directly from the Internet without any operator intervention.

These plugins are then cached in the plugin_cache_dir section, to save from downloading the providers during every Terraform plan and apply.

Example

plugin_cache_dir = "/terraform/plugin-cache"
provider_installation {
    filesystem_mirror {
        path = "/terraform/plugins/linux_amd64"
        include = [
            "registry.terraform.local/sei/*",
            "registry.terraform.local/mastercard/*"
        ]
    }
     direct {
        include = [
        "hashicorp/vsphere",
        "hashicorp/aws",
        "hashicorp/azurerm",
        "hashicorp/azuread",
        "hashicorp/null",
        "hashicorp/random",
        "hashicorp/template",
        "hashicorp/cloudinit"
        ]
    }
    }

Hosts

A host consists of a name, datastore, and maximum number of virtual machines that it can support. The API creates and manages hosts, then assigns them to exercises. An exercise can have many hosts.

Workspaces include an additional property, DynamicHost, which usually defaults to false. When Alloy creates a workspace, it sets DynamicHost to true and changes the behavior of a run. When DynamicHost equals true, Caster examines all of the hosts assigned to the current exercise. It chooses the host with the least usage (the number of machines to deploy/maximum machines) and assigns that host to the workspace.

In addition to the normal run files, Caster creates a generated_host_values.auto.tfvars containing two variable values: vsphere_host_name and vsphere_datastore, set to the name and datastore of the selected host. Upon applying the run, Caster tracks how many VMs deployed to the host, and uses this for future calculations.

After an on-demand exercise (ODX) finishes, Caster deletes the workspace and releases the host's resources. If a run attempts to deploy more virtual machines than there is capacity for in the available hosts, the run will fail.

On-Demand Exercise Functionality

Alloy calls Caster in order to deploy resources for lab or ODX-style functionality. Caster itself does not differ in its main functionality of deploying workspaces and lets Alloy handle most of the ODX functionality.

However, in order to support this functionality Caster dynamically selects a host to deploy to.

Normally, the configuration embeds the cluster or host to deploy to, either directly or as a variable, and Caster doesn't concern itself with this. For ODX's, Caster does need to concern itself with:

  • ensuring that the system deploys resources evenly to the available hosts, and
  • preventing the hardware from attempting to deploy more ODXs than it can support.

To address these concerns the team added the concept of a host to Caster.

Run, Plan, and Apply

A run is a specific instance of the Terraform plan and apply process for a workspace. The run instantiates the configuration in a directory as deployed resources within a workspace. Upon opening a workspace, the interface displays a list of runs. Users start plan or destroy operations there.

Plan and apply are specific Terraform terminologies. Every run includes a plan step and an apply step.

Plan

Clicking plan creates a new Run and executes the terraform plan command on the given configuration. This raw Terraform output, visible to the user, describes:

  • What actions Terraform will take
  • What resources to create
  • What resources to change
  • What resources to destroy

A plan shows the user what will deploy.

This output always ends with a summary of the form Plan: x to add, y to change, z to destroy. The user reviews this and chooses to apply or reject the plan.

  • Choosing apply creates an apply for the run and executes terraform apply on the previously generated plan. This causes Terraform to make the changes described.
  • Choosing reject invalidates the plan. It leaves the infrastructure unchanged.

Apply

Apply takes a run, executes terraform apply on the previously generated plan and deploys the resources for a workspace. The Apply command:

  • Deploys a workspace run
  • Releases plan tools such as network resources and virtual machines into vCenter

Within the workspace view users can see every run they have planned and applied.

Destroy

Selecting destroy instead of plan is similar, but the generated plan will destroy all previously deployed resources in the workspace, rather than matching the infrastructure to the current configuration. That is, Destroy creates a plan that will destroy all of the previously deployed resources in a workspace.

If the configuration defines a resource, a run creates it, and you later delete that resource from the configuration, the next plan or destroy run removes it. Terraform always tries to match the infrastructure to the current configuration.

There is only one run in progress at a time per workspace. Terraform locks the state of the workspace and only performs a single operation at a time. Developers may wish to break up large deployments into multiple directories and workspaces to operate on different parts of the deployments simultaneously. For example, break out user enclaves so developers can perform actions on other parts of a network without (potentially) waiting a long time to redeploy user machines.

The workspace view shows a table with all the runs the directory has planned and applied. Caster highlights destroy operations in red.

Within the workspace view users can click Destroy to destroy live Terraform applications.

This Infrastructure-as-Code approach is unfamiliar to some developers. It defines a configuration and applies it in its entirety, rather than selecting individual pieces for deployment. There are some ways to target individual pieces of a configuration, but Terraform recommends this as the exception rather than the rule, and Caster does not fully implement this.

Taint

Taint is a flag on a resource that tells Terraform to destroy and recreate a new instance on the next plan-and-apply cycle.

Taint allows users to redeploy resources. For example, if a user needs to redeploy a series of virtual machines, the user can:

  1. Taint these resources
  2. Run another plan-and-apply cycle that will redeploy the instance as if new from a template

Some resources are not in scope for taint, however.

Users can taint resources within the workspace view. A tainted resource will display in red shading. Users can easily access the Untaint command while in workspace view before running another plan-and-apply cycle if they decide to keep the resource.

Glossary

This glossary defines key terms and concepts used in the Caster application.

Connection: A vSphere vCenter or Proxmox VE cluster that Caster reads inventory from, configured by the deployment.

Designer: A graphical user interface for creating and editing Terraform deployments through the use of modules.

Directory: The outline of a project and the modules it contains.

Environment Set: A named group of variables, including secrets, that Caster gives to every Terraform run in the projects it is assigned to.

File: Text files to eventually put onto a file system and use with the Terraform command line tool.

Host: Consists of a name, datastore, and maximum number of virtual machines that it can support.

Inventory: The VM templates, ISOs, networks, datastores, and content library items on a connection, listed in a project's Inventory panel.

Module: A container for multiple resources used together. Modules create lightweight abstractions, so you can describe your infrastructure in terms of its architecture, rather than directly in terms of physical objects.

Parallelism: The number of concurrent operations Terraform performs during a run.

Project: A way to organize and categorize similar environments for multiple workspaces and directories within Caster.

Project Group: A set of projects that share memberships, Environment Sets, connections, and a VLAN partition.

Run: A specific instance of the Terraform plan and apply process for a workspace.

Terraform: An open source Infrastructure-as-Code tool.

Topology: The physical and logical arrangement of nodes and connections in a network.

Workspace: A specific instance of a deployed Terraform configuration.