Upgrades
Admin guide
Greenlight keeps itself current. Updates apply automatically inside your own cloud during the maintenance window you choose — including the ones that change infrastructure — and Shift never reaches into your cloud to do it. The work is done by a component running inside your own cluster, under an identity you own that cannot act outside the boundary Greenlight was installed into.
Delete protection is built into the infrastructure templates themselves: your databases, storage, DNS, and the cluster and its node pools cannot be destroyed or replaced by an automatic update, whatever a release contains. A release that changes no infrastructure simply updates the application.
What you can see
Section titled “What you can see”The Updater panel, on Internal Health’s Updater health tab, opens on one line: whether you are current or a newer release is waiting, and when the next run is due. An Update settings and last run toggle underneath holds the rest — whether automatic updates are on, the schedule and time zone they run in, your prerelease posture, the updater’s own image, and the last check and apply. Anything the install could not report says so, and says why, rather than showing a blank. The installed version itself is stated once, on Internal Health’s Versions card.
Below that is the run history, newest first. By default it leaves out the checks that found nothing new to take. A nightly install records a check every night, and on most nights there is nothing new to take — left in, those checks are almost every row and the runs that actually did something sit a long way down. Show up-to-date checks brings them back when you want the full record. Either way it shows eight runs at a time, with Previous and Next to walk the rest — the list pages instead of growing, so reading further back never pushes the rest of the page out from under you.
Open a run and you get the whole story: each phase with how long it took and why it ended, the exact non-secret inputs the run replayed, the release versions it moved between, and the full log the run produced — the infrastructure plan, the application rollout, and the installer’s own output — readable and downloadable long after the run’s pod is gone. Runs are kept for 90 days and their logs for the newest 30 runs.
While a run is in progress the history says so and refreshes itself every few seconds, so its phases appear as it records them; Refresh re-reads it on demand. The run’s log is the exception — it is buffered inside the updater’s own pod and written in one piece when the run ends, so an open run shows its phases rather than a log that fills in line by line.
A run that applied its release but failed one of the installer’s closing checks is labelled as exactly that, never as a failed update: the release is running, and one check did not pass. A run stopped by its time limit is recorded as failed, with that as its reason. A run whose Job finished without recording anything reads as “never reported” — which is a different fact from a failure.
When a run fails, or fails a closing check, Greenlight sends its log, redacted, to Shift’s fleet monitor so Shift can see what went wrong without asking you for it (Failure logs sent to Shift). Each send is recorded in your audit log.
If a run fails, the panel carries its reason and the exact installer command to apply that release by hand, and keeps them there until a later update succeeds. Once one does, the failure moves into the run history: it is something that happened rather than something you are still sitting on.
Settings → Maintenance shows the same schedule read-only.
What you control
Section titled “What you control”You choose the maintenance window — the time of day and the time zone updates run in. That is the extent of it: there is no approval queue, no version pinning, and no pause. Eligible releases apply on their own once their preflight checks pass, and an install that has fallen behind moves straight to the newest release rather than working through the intermediate ones.
To change the window, rerun the installer with GREENLIGHT_UPDATER_SCHEDULE (a five-field cron expression, default 0 3 * * *) and GREENLIGHT_UPDATER_TIMEZONE (an IANA zone name, default Etc/UTC) set in its environment. Values you do not restate keep their last setting, so a later rerun for anything else leaves the window alone.
One rerun has a desktop path too. On a private Azure install whose egress goes through your firewall, the desktop installer’s Connect your network screen runs the egress switch for you: it reruns the installer with --egress-next-hop set to your firewall’s private address, and you confirm its review as on any run. It then shows each destination the egress check reaches or finds blocked, and reruns the check with --egress-preflight after you change the firewall (Connect your network to a private install).
Staying current is what keeps one supported version to reason about and lets a security fix reach your install the night it ships, so it is not an opt-in setting. If you need a change freeze for an audit or a peak season, talk to us rather than looking for a switch.
Apply an upgrade
Section titled “Apply an upgrade”- Read the release notes. They cover migrations, rollback limits, and any policy changes.
- Let it run. The update applies at your configured maintenance time. Nothing is required of you.
- Check the result. Post-upgrade checks confirm the new version responds, audit writes work, and policy checks still pass. Internal Health shows the new version, how the run finished, and the run’s full log if you want to read what it did.
Kubernetes version upgrades
Section titled “Kubernetes version upgrades”Some releases move your cluster to a newer Kubernetes version. Your cloud provider carries that out by replacing every node in the cluster, one at a time, including the node the update itself is running on. The update is built for this: when its node is taken away it starts again on another node, waits for your cloud to finish replacing nodes, and completes in the same maintenance window. Internal Health shows it as one run, and its log shows where it moved to a new node.
Each release pins the Kubernetes version your cluster runs. Patch updates within that version arrive from your cloud provider on its own schedule. A cluster more than one version behind moves one version per update until it catches up, and a cluster already ahead of the release’s version stays where it is.
Everything else moves to the new nodes too: the dashboard restarts during the upgrade and is briefly unavailable, and each app restarts once as its node is replaced.
Update now, without waiting for the window
Section titled “Update now, without waiting for the window”Internal Health carries two actions for organization administrators:
- Preview update checks for a newer release, verifies it, runs the pre-checks, and produces the infrastructure plan — then stops. Nothing is applied. It is the safe way to see what an update would do.
- Apply now performs the update immediately, running exactly the same steps the scheduled run would.
Both do the same work as the nightly run, under the same customer-owned identity, with the same audit trail — the only difference is who decided the timing. Only one update runs at a time: while one is in progress the actions are unavailable and the page says so. Only one run ever does work at a time, however it was started. A scheduled run that lands during an update you started this way waits for it, for up to ten minutes. If yours is still going after that, the scheduled run appears in the history as Skipped, naming the run it waited for, changes nothing, and stores no log; the next scheduled run carries on as normal. Both are recorded in the audit log with who asked, which action they chose, and whether it started; a refused attempt is recorded too.
Neither action lets you change what gets applied. There is no way to pick a different version or skip a step from the dashboard, and the only install settings it changes are the node counts under Capacity below; any other setting is still a rerun of the installer. And there is no pause: staying current is not an opt-in setting.
Rollback
Section titled “Rollback”Helm uses an atomic rollout, so a failed application rollout keeps the previous Greenlight workloads running. Infrastructure changes are applied separately and may already have completed or still be converging; the dashboard reports that state and a later idempotent run reconciles it.
Some successful releases cannot be rolled back afterwards, because a database migration has already moved forward. The release notes say so before the release applies, and recovery past that point is a forward fix rather than a rollback.
Capacity
Section titled “Capacity”App capacity autoscales within its configured range, so ordinary demand needs no attention. If your apps are pressing against the top of that range, Internal Health says so and identifies it as a platform-capacity issue rather than an application bug.
The cluster adds a node only when an app’s pod fits on none of the nodes already running, not when a percentage is crossed. Internal Health’s Cluster capacity card says how many more small apps (ones that declare no size) still fit before that happens, and the most room any one node has left; an app that needs more than that adds a node as it deploys.
The range is changed from Internal Health, not from a terminal. Under the nodes on the Cluster capacity card, on the Cluster health tab, Can grow to shows how many nodes the cluster may grow to; organization administrators change it with − and +, then choose Apply now or At the next update window. More options holds two more counts: how many nodes stay on even when apps are idle, and how many nodes Greenlight itself runs on (which can be raised here, not lowered). When apps are waiting because the cluster is at its limit, the card asks in one line to raise it; you approve or choose Not now, which keeps it quiet for a week. When the cluster has kept more always-on nodes than it needed for 30 days, the card suggests keeping fewer. Nothing shrinks unless you accept. The card shows no prices: what a node costs depends on your machine size, region, and agreement with your cloud provider.
A change is applied by the same updater that applies releases, under the same identity and with the same safety checks: at the next maintenance window, or straight away with Apply now. Raising a count, or lowering the smallest, stops nothing that is running. If Apply now cannot start the updater, because a run is already going or the cluster does not answer, the change is still recorded and waits for the next scheduled run. If the apply fails, your install keeps the counts it had and the card says so. If someone reruns the installer with different counts while a change is waiting, the waiting change is not applied, since it was checked against counts the install no longer has; record it again from the card. Every change, and every declined ask, is in the audit log.
Lowering the largest count below the app nodes running now removes nodes your apps run on, so Greenlight checks first that their apps fit on the nodes that stay, with room left for one more copy of each app so its next deploy can still start a new copy beside the old ones. If they fit, the change is confirmed with how many nodes go, and it applies straight away with Apply now; the apps on the removed nodes restart on the others. If an update is already running, the change is not kept; try again once it finishes. If they do not, the card says there is not enough room and to remove some apps first; then try again. A largest count that is still at or above the nodes running now removes nothing.
Some settings are not changed here yet: machine sizes, which would replace the node pool; removing a node from Greenlight’s own pool, which can stop the update applying it; and the database size, which restarts the database.
Permissions and limits
Section titled “Permissions and limits”| Cloud | Permission boundary |
|---|---|
| Azure | Dedicated Greenlight resource group |
| GCP | Dedicated Greenlight project |
| AWS | Dedicated Greenlight member account |
Automatic updates depend on the cluster they run in, so they cannot recover a cluster that has been deleted or made unreachable. Those situations are genuine disaster recovery, and the runbook covers them.
Rarely, an update needs something only you can supply — usually a new setting no release can pick a sensible default for. When that happens the update stops with nothing changed, and the dashboard tells you what is needed and what to do next.
Agent plugin releases
Section titled “Agent plugin releases”Plugin releases update the Skills body, MCP configuration, and the agent-specific marketplace wrappers. Your own Knowledge is never touched, and Greenlight updates the plugin content in your marketplace repositories for you.
The plugin’s version is the Greenlight release that generated it. Greenlight only recommits your marketplace repositories when an upgrade changes the plugin, so agent clients are not prompted to update for releases that leave it alone. The plugin’s version can therefore be older than the Greenlight release you are running.