Ingest

The Slingshot ingest system has three main components, file_types, inputs, and pipelines.

ingest:
  file_types:
    - name: example
      examples: [ my_file.mov, another_version_avid.mov] # (1)!
      regex: .+
      parent_folders_regex: null
      extensions:
        - mov
        - mp4
      priority: 5
      default_task: null
      var_map: null
  inputs:
    - name: Example Input # (2)!
      polling_interval: 180
      use_override_task_queue: false
      input:
        module: test
        pipeline: default
        number_of_packages: 3
        max_items_returned: 4000
        no_items_returned_percentage: 33
  pipelines:
    default:
      finish_timeout: PT2M
      actions:
        - module: test
          description: Example action # (3)!
          status_message: Test
          delay_seconds: 0
          test_file_types:
            - spreadsheet
            - other
        - module: download
          description: Example download 
          operation: download
          overwrite_existing_files: true
          set_local_path: true
          paths: {}
      finishers:
        - module: finish_shotgrid_delivery
          description: null 
          sg_delivery_entity: Delivery
          sg_name_field: title
          sg_summary_field: sg_contents
          sg_finished_status: recd
          sg_failure_status: null

1. Examples are to document sample file names that match the regex and are for our internal reference
2. Names must be unique and will be used in logs and to name the input workflow in temporal.
3. Descriptions are optional, and currently only used for our internal reference.

File types

File types define how what filename/folder patterns are what type of file. For example, common file_types are shots, assets, csv, shot_frames, etc.

See: file types

Inputs

Inputs define the service and location(s) that Slingshot monitors for new files. For example, a specific path in an S3 bucket, or an Aspera on Cloud account.

A show can have multiple inputs.

Each input also specifies a pipeline that files received from that input will process through.

See: input modules

Pipelines

Pipelines define sets of actions that Slingshot can perform on each file.

 pipelines:
    vendor_in: # (1)!
      finish_timeout: PT8M # (2)!
      actions: # (3)!
        - module: download
          ... # module specific configuration here
      finishers: # (4)!
        - module: finish_shotgrid_delivery
          ...

1. The name of the pipeline. This can be anything, but must match a pipeline specified in an input. It will be displayed in the UI.

2. The time we wait after the last item is uploaded to assume the package is finished and run the finishers.

   In iso 8601 format

3. A list of action modules to run, in order from top to bottom

4. A list of finisher modules to run after the finish_timeout

After the last item has been uploaded and the finish_timeout has passed, the finisher modules will then run.

See: action modules, finish modules