Preparing a Compatible Replication Package

You should not include any data that you are not allowed to upload to third-party systems¶

While SIVACOR does not publish data or replication packages, and deletes completed jobs after a short period of time, it is not a designated secure computing system.^[1] You should not upload controlled data, and all uploads should be compatible with any data use agreement you signed.

If you have data that you are allowed to upload, but not publish, see the next point on how to exclude files from the final replication package.

Excluding files from final package¶

The final digitally signed replication package contains all data as originally included. If you need to remove files because you do not have redistribution rights, or large intermediate files, you can include a file named .sivacorignore (note the leading dot!) at the root of your project to exclude files or directories before the final replicated package is created. It follows the same pattern rules as .gitignore, so you can use glob patterns, negations, and directory-specific rules.

data/raw/
*.tmp

data/raw/
  file1.csv
code/
  main.R
  ...
.sivacorignore

Your replication package should be portable.¶

Code must run without manual intervention, use a single controller script (e.g., main.do or master.R), and avoid hard-coded absolute paths. You can only upload the package, not edit it on the site. It should also not have inconsistently used case-sensitive file or directory names.

All dependencies must either be included or installed automatically.¶

If your code uses libraries or packages, you must ensure that they are installed automatically (for Stata, we suggest you include them). We strongly encourage packages that use “environments”, and packages to manage dependencies.

Your replication package only uses a single software application per step¶

Each step of a SIVACOR submission only supports a single software application (e.g., Stata, R, Python), as encapsulated by containers. If your replication requires multiple applications, you will need to configure separate runs. However, your package itself can include the code for multiple applications, and you can chain them together in a highly simplified workflow system at submission, see instructions in Step 2.

Size considerations¶

The size available to run your code depends on the software being used, and how you manage files within your replication package. A complete run of your code needs room for more than one copy of itself: the archive you upload, the workspace it is extracted into, and anything your code writes all need to be accommodated.

The base system has about 55.5GB of space available. Software sizes differ. The table below lists free space after accounting for typical software versions. Specific sizes might differ slightly. See Available Software for the full, curated list of software.

import yaml
import requests
from IPython.display import HTML, display

DISK_GB = 60  # total disk on the machine, in decimal GB
OVERHEAD_GB = 4.5  # operating system + SIVACOR harness

# Docker Hub reports the *compressed* size of the layers; `docker pull` unpacks them, and the
# compressed copies are not kept. Measured unpacked/compressed ratios for the images below
# range from 2.05x (dataeditors/stata19_5-mp) to 2.95x (rocker/geospatial), so 3x is used as a
# deliberately conservative ceiling rather than a best estimate.
UNPACK_FACTOR = 3

# A handful of representative images, not the full curated list (see images.md for that).
SAMPLE_IMAGES = [
    ("rocker/verse", "R"),
    ("rocker/geospatial", "R"),
    ("dynare/dynare", "MATLAB"),
    ("dataeditors/stata19_5-mp-i-python", "Stata"),
    ("dataeditors/stata19-mp", "Stata"),
]

repos_url = "https://raw.githubusercontent.com/SIVACOR/sivacor-repo-choice/main/allowed_repos.yaml"
allowed = yaml.safe_load(requests.get(repos_url).text)

rows = []
for image_name, software in SAMPLE_IMAGES:
    tag = str(allowed[image_name][0])  # first entry is the most recently added tag
    tag_info = requests.get(
        f"https://hub.docker.com/v2/repositories/{image_name}/tags/{tag}/"
    ).json()
    size_bytes = next(
        (img["size"] for img in tag_info["images"] if img["architecture"] == "amd64"),
        tag_info["full_size"],
    )
    size_gb = size_bytes / 1e9
    unpacked_gb = size_gb * UNPACK_FACTOR
    rows.append(
        (
            software,
            f"{image_name}:{tag}",
            size_gb,
            unpacked_gb,
            DISK_GB - OVERHEAD_GB - unpacked_gb,
        )
    )

# MyST does not render `text/markdown` outputs, so emit HTML directly.
L, R = 'style="text-align:left"', 'style="text-align:right"'
table = (
    "<table>\n<thead><tr>"
    f"<th {L}>Software</th><th {L}>Image</th>"
    f"<th {R}>Download size</th>"
    f"<th {R}>Space used on disk*</th>"
    f"<th {R}>Free space for your package*</th>"
    "</tr></thead>\n<tbody>\n"
)
for software, image_tag, size_gb, unpacked_gb, free_gb in rows:
    table += (
        f"<tr><td {L}>{software}</td><td {L}><code>{image_tag}</code></td>"
        f"<td {R}>{size_gb:.1f} GB</td>"
        f"<td {R}>{unpacked_gb:.1f} GB</td>"
        f"<td {R}>{free_gb:.1f} GB</td></tr>\n"
    )
table += "</tbody>\n</table>"

display(HTML(table))

For more information on the system itself, see Hardware capabilities.

If free space runs low, the run is stopped and you will see an error saying the submission ran out of disk space — see the FAQ for what to do about it.

Prepare a ZIP or tar.gz file¶

Your replication package must be a single ZIP file or tar.gz file.

The next step is to upload your package to SIVACOR.