Choosing Software and Running Jobs

import csv
from pathlib import Path

from IPython.display import HTML

nodes = list(csv.DictReader(Path("_data/jetstream2-nodes.csv").open()))
disk_gb = next(n for n in nodes if n["default"] == "true")["disk_gb"]

L, R = 'style="text-align:left"', 'style="text-align:right"'
table = (
    "<table>\n<thead><tr>"
    f"<th {L}>Size</th><th {R}>Cores</th><th {R}>Available to your analysis</th>"
    f"<th {R}>Disk</th><th {L}>Relative cost</th></tr></thead>\n<tbody>\n"
)
for n in nodes:
    cost = f"{n['relative_cost']}×" + (" — by request" if n["by_request"] == "true" else "")
    table += (
        f"<tr><td {L}>{n['memory_gib']} GiB</td><td {R}>{n['cores']}</td>"
        f"<td {R}>≈{n['memory_available_gib']} GiB</td><td {R}>{n['disk_gb']} GB</td>"
        f"<td {L}>{cost}</td></tr>\n"
    )
table += "</tbody>\n</table>"
size_table = HTML(table)

If the upload was successful, scroll down.

Choose software and version¶

Choose first the software and version from the curated list (see container images). You can also select an image tag (sub version), but generally, the latest version should work.

Identify the main file¶

Identify the name of the main file. This is the file that will be executed by SIVACOR. Include the extension (.R, .do, .jl).

The file does not have to sit at the top of your package: SIVACOR searches the whole package for that name, and runs it from the directory it was found in.

# Render RMarkdown
# Assert that rmarkdown is available
if (!requireNamespace("rmarkdown", quietly = TRUE)) {
  stop("rmarkdown package is required but not installed")
}
rmarkdown::render("main.Rmd")

Network isolation¶

Each step carries a Net Isolation toggle, beside that step’s image and main file. It is off by default, and it is per step, not per submission: one step may be isolated and the next not.

When it is on, the container has no network access at all for the whole of that step. Nothing can be downloaded, no API can be called, and no result can depend on something fetched at run time.

This is the reason to split dependency installation into its own step. Install packages in a first step with isolation off, then run the analysis in a second step with isolation on: the analysis — the part being certified — is then isolated, even though the package downloads were not. See Step 0.

Optional chained runs (steps)¶

You can chain multiple runs together, by selecting the + ADD STEP button. The runs will be run in separate containers. Each run inherits the workspace modified by the previous run, so the output of one run will be made available as input to the next run.

stages:
  - image_name: dataeditors/stata15
    image_tag: "2023-01-27"
    main_file: main_step1.do
    network_isolation: true
  - image_name: rocker/tidyverse
    image_tag: "4.6.1"
    main_file: main_step2.R
    network_isolation: false
env_secrets:
  - key: API_TOKEN
    value: s3cret

resources:
  memory_gb: 60
stages:
  - image_name: rocker/tidyverse
    image_tag: "4.6.1"
    main_file: main.R

Advanced settings¶

Several advanced parameters are in a dropdown menu.

• Worker Size: the type of machine your submission runs on

• Extra Scratch Disk: a temporary disk in addition to the machine’s own disk space

• Environment Secrets: values passed to your code as environment variables.

Choose the machine size¶

Under Advanced, Worker Size sets the machine your submission runs on. It applies to the whole submission: every step runs on the same machine.

size_table

Submissions default to the smallest size. Only request more if you know that you need more. The output from a run shows what your last run actually used, as a share of what it was allowed.

Where to find run statistics

A finished run reports its peak memory and peak disk use on the submission page, beside the download links, as a share of what the machine allowed. Those two figures are what to size the next run on: a run that peaked at 40 % of a 30 GiB machine has no reason to ask for 60 GiB.

Extra scratch disk¶

Extra Scratch Disk asks for a temporary disk in addition to the machine’s primary 60 GB disk. It is enabled only upon request, see Requesting additional resources.

Once your account has a scratch disk allowance:

• enter the number of gigabytes you want for this submission, up to your allowance. Requests are rounded up to the nearest 10 GB;

• no changes are needed for your analysis. Your code sees a single filesystem.

• the requested number is not preserved from one run to the next - it must be re-entered every time you submit a job.

Requesting additional resources¶

To request additional resources, send an email to support@sivacor.org.

Environment variables¶

You can set environment variables for your job by using the env_secrets block in a workflow definition file, or by entering them in the submission form. These variables are available to your code during execution.

env_secrets:
  - key: API_TOKEN
    value: s3cret

Submitting jobs¶

Then click on the Run Replication Workflow button.

ℹ️ FAQ¶

See the FAQ.