# Ingest Queue

<div align="left"><figure><img src="/files/lixKj2eanoRtiSb19sHX" alt=""><figcaption></figcaption></figure></div>

## Details <a href="#h_01he65j87rdh55wz000q3tf2j5" id="h_01he65j87rdh55wz000q3tf2j5"></a>

The **Ingest Queue** is the final step in getting media files into ShotGrid and the production filesystem, whether they were uploaded by a Vendor via the **Vendor Submission** and **Upload Submission** apps, or originated internally and submitted via the **Internal Submission** app.&#x20;

Typically the Ingest Queue runs continuously in the background, looking for Submissions that are ready to ingest and processing them automatically as they come in. You can also direct the Ingest Queue to manually process a single Submission.

When the Ingest Queue receives a Submission, it validates the files within it, copies them into the production file system, and creates records and web-playable versions of each in ShotGrid.

{% hint style="success" %}
The Ingest Queue only works with Submissions that originate from [Vendor Submission](/artist-anywhere/vendor-workflows/vendor-submission.md) or [Internal Submission](/artist-anywhere/i-o-tools/internal-submission.md). To create records in ShotGrid for media without creating a Submission, you can use [Import Media](/artist-anywhere/i-o-tools/import-media.md).
{% endhint %}

## Initial Setup <a href="#h_01he1hf5t1me6zbkkbmtc3xhx9" id="h_01he1hf5t1me6zbkkbmtc3xhx9"></a>

* Install [**ShotGrid Desktop**](/artist-anywhere/i-o-tools/flow-production-tracking-desktop.md) and log into the appropriate ShotGrid site
* **Create a Submission**, either through [**Vendor Submission**](/artist-anywhere/vendor-workflows/vendor-submission.md) and [**Upload Submission**](/artist-anywhere/vendor-workflows/upload-submission.md), or [**Internal Submission**](/artist-anywhere/i-o-tools/internal-submission.md)
* To receive notifications by email, ensure that you have Delivery email notifications turned on in your ShotGrid account settings

{% hint style="success" %}
To turn on email notifications in ShotGrid for Deliveries, go to the **Account Settings** -> **Email Notifications** tab and then check **Email me about Deliveries that I'm involved in (those that I've created or replied to, or where I'm a recipient, except for file attachments)**.&#x20;

![](/files/b3zIgqIn61Zyzpwhveor)

You can learn more in [the Email Notifications section of the ShotGrid User Guide](https://help.autodesk.com/view/SGSUB/ENU/?guid=SG_Administrator_ar_manage_accounts_ar_account_settings_after_migrating_html#change-your-email-notification-settings).
{% endhint %}

## User Interface and Workflow <a href="#h_01he1hqa9h4h2a5m9j62w0995f" id="h_01he1hqa9h4h2a5m9j62w0995f"></a>

### Launch the App <a href="#h_01he1hf5t1zhb768g6802h8v3j" id="h_01he1hf5t1zhb768g6802h8v3j"></a>

<div align="left"><figure><img src="/files/BaMfK2YBPFlQ6IFvseNV" alt="" width="375"><figcaption></figcaption></figure></div>

Open [ShotGrid Desktop](/artist-anywhere/i-o-tools/flow-production-tracking-desktop.md), select your Project, and click on **Ingest Queue**.

### Initial View <a href="#h_01he1hf5t17h90w2pth57n7bny" id="h_01he1hf5t17h90w2pth57n7bny"></a>

Once you’ve launched the app, you’ll see this screen with two buttons at the top, “Start” and “Check now”, and an area for manual drag and drop of Submissions.&#x20;

<div align="left"><figure><img src="/files/k6bUuwZgIyAI6IHzMGMe" alt="" width="375"><figcaption></figcaption></figure></div>

There are several ways to provide Submissions to the Ingest Queue for processing:

* Continuous auto-discovery of Submissions (preferred)
* One-time check for Submissions
* Manual input of Submission

### Continuous Auto-discovery of Submissions <a href="#h_01he1hs8bskbdtm914hw4mm69n" id="h_01he1hs8bskbdtm914hw4mm69n"></a>

The **most common use case** for the Ingest Queue is to have it running continuously in the background. To start the app in this mode, simply **click the Start button**. It will check for new Submissions every 30 seconds. It will begin processing any Submissions it finds, **up to a configurable maximum quantity**, and queue up the rest.&#x20;

{% hint style="info" %}
There is a maximum quantity of simultaneous ingests to prevent overloading ingest machines and/or ShotGrid. By default, the Ingest Queue will process a **maximum of two Submissions at one time,** but this value is configurable. Contact us at <support@artistanywhere.io> to change it.
{% endhint %}

To pause the Ingest Queue from scanning for New Submissions, click the **Pause** button. To stop it, simply close the app.&#x20;

#### One-time check for Submissions <a href="#h_01he1hsjpp64jvmdpj67yt2my0" id="h_01he1hsjpp64jvmdpj67yt2my0"></a>

You can also have the Ingest Queue query ShotGrid for new Submissions only one time without continuously running. To do so, **click the Check Now button**.&#x20;

#### Manual Input of Submission <a href="#h_01he1hspc99ymbzezg299mtamr" id="h_01he1hspc99ymbzezg299mtamr"></a>

If you want to bypass the app’s scanning capability and manually submit a Submission for ingest, you have a couple options:&#x20;

* **Within ShotGrid**, right-click on a Delivery record and choose **Ingest Queue.** This will launch the app with the selected Submission pre-loaded. <br>

  <figure><img src="/files/ZSk3JeklWNjfdlM0ULb1" alt=""><figcaption></figcaption></figure>

* As a less common workflow, you can drag or drop either a folder containing a Submission or the manifest.yml file from a Submission onto the drop area of the app.&#x20;

### Transfer Settings Dialog <a href="#h_01he1ht4f6wxscxrser6cj58ak" id="h_01he1ht4f6wxscxrser6cj58ak"></a>

<figure><img src="/files/nQXxHvzng3jFbqwNIVgn" alt=""><figcaption></figcaption></figure>

If you are using a [transfer method](/artist-anywhere/vendor-workflows/vendor-submission-transfer-methods.md) that requires adding credentials, such as Aspera on Cloud Files or Aspera Faspex, you will receive a popup the first time the app receives a Submission. Follow the instructions for your transfer method from the [Vendor Submission Transfer Methods document](/artist-anywhere/vendor-workflows/vendor-submission-transfer-methods.md).&#x20;

{% hint style="success" %}
To bring up the Transfer Setting Dialog at another time, click the **gear button** to the right of **Start** and **Check Now**.
{% endhint %}

### The Ingest Process <a href="#h_01he1j01pyab6qxz6dtbnjmqxe" id="h_01he1j01pyab6qxz6dtbnjmqxe"></a>

Once the app finds or receives new Submissions, it will immediately begin the ingest process.&#x20;

<figure><img src="/files/84cdtmY4HlZU4rgqRTCe" alt=""><figcaption></figcaption></figure>

{% hint style="success" %}
To return to the drag and drop area from the Ingest Status view, **click the +Delivery button.**
{% endhint %}

Once the Ingest Queue receives a Submission, it does the following:

* Downloads media from a remote server, if necessary
* **Validates media** to ensure that it’s complete and undamaged
* Copies files into the **production file system**, placing them within a pre-configured folder structure and creating any necessary folders

{% hint style="info" %}
As you ingest files via the Artist Anywhere Studio Workflow Tools, the Ingest Queue app copies them into your production filesystem, creating folders as needed to gradually build it out. The tools have a default folder structure, but you can customize it to suit your studio’s needs. To make modifications to your folder structure configuration, contact <support@artistanywhere.io>.&#x20;
{% endhint %}

* **Within ShotGrid**, it:
  * Creates PublishedFiles, **per-file records** for tracking files on disk
  * Creates Versions, **web-playable media** and thumbnails&#x20;
  * Creates a per-Delivery **Playlist** for review
  * Imports any **metadata**, such as the Submission Note, accompanying the Submission
  * Updates **Delivery Status**

{% hint style="info" %}
Note that while the From, To, and Created By fields on the Delivery in ShotGrid will be set when Vendor Submission or Internal Submission is run, the **PublishedFile**, **Version**, and **Playlist** records that are created by the Ingest Queue will have their **Created By** fields set to the **currently logged in user**.
{% endhint %}

If there are already files with the same name in the destination folders, the app will ask you if you want to overwrite them.

<div align="left"><figure><img src="/files/6D3tzpmq0L00eHflpKFM" alt="" width="390"><figcaption></figcaption></figure></div>

### Monitoring Ingest Progress in ShotGrid <a href="#h_01he1hzxk3ha6ws9xe95h3123n" id="h_01he1hzxk3ha6ws9xe95h3123n"></a>

Typically, the Ingest Queue will be running in the background on an ingest server, but you can monitor the status of a Submission from any machine by looking at its Delivery record in  ShotGrid.

{% hint style="success" %}
For more information about Delivery records, see [Autodesk’s documentation](https://help.autodesk.com/view/SGSUB/ENU/?guid=SG_Tutorials_tu_deliveries_html).&#x20;
{% endhint %}

* **Status and Delivery Progress fields:** Throughout the transfer and ingest process, the Artist Anywhere Studio Workflow Tools will update the **Status** and **Delivery Progress** fields on the Submission’s Delivery record. You can look at these fields in a **list page** for a quick status update.

  * When a Submission has been delivered and is ready for ingest, its Status and Delivery Progress will both be **Delivered.**
  * When ingest is complete, its Status and Delivery Progress both change to **Received**.&#x20;
  * If there was an error in the ingest process, the Status will be **Errored** and the Delivery Progress will be **Ingest Failed**.

  <figure><img src="/files/Heme6joeDmXpBFxlPi4T" alt=""><figcaption></figcaption></figure>

* **Reply thread:** For more detailed progress information, look at the **details page** for your Submission’s Delivery record. **Ingest Queue** and all of the Studio Workflow Tools add replies and events to the thread at every stage of the process, with status updates and links to attached files like manifest.yml and log files.

  <figure><img src="/files/JDmUSZ0Z4gexKykoQvnJ" alt=""><figcaption></figcaption></figure>

* **Log files:** The Ingest Queue generates the log files ingest.log and transcoding.log, which contain verbose logging of the app’s processing. You can access these from the reply thread on the **details page** for your Submission’s Delivery record.\ <br>

  <figure><img src="/files/Y7tbuaKDJEti71Yys9wu" alt=""><figcaption></figcaption></figure>

### Ingest Completion <a href="#h_01he63dzzfsdsxpvz9fmj45typ" id="h_01he63dzzfsdsxpvz9fmj45typ"></a>

The app will indicate when ingestion has completed for a Submission. You can click on the arrow on the right of each Submission to see details about the files within it. ShotGrid will also send email notifications to users in Groups mentioned in From, To, and Cc for the Delivery.

{% hint style="success" %}
ShotGrid’s email notifications when a Submission gets ingested contain the Delivery name and Project name in the subject, and Delivery details in the body of the email.
{% endhint %}

<div align="left"><figure><img src="/files/KO7i8LA7cNCl0KqaOZsn" alt="" width="563"><figcaption></figcaption></figure></div>

Additionally, when a Submission has completed ingestion, **the Status and Delivery Progress fields on its Delivery record in ShotGrid will change from Delivered to Received**.

## Troubleshooting <a href="#h_01he63ew6rzz7q7y9pdg41p70e" id="h_01he63ew6rzz7q7y9pdg41p70e"></a>

The suggestions below will help you fix some issues that come up when using the Ingest Queue. If you come across any issues that you can’t diagnose and fix yourself, please contact <support@artistanywhere.io>.&#x20;

### Error Logs <a href="#h_01he63f3t4er143ejskx8pd03s" id="h_01he63f3t4er143ejskx8pd03s"></a>

Sometimes \[RE]DESIGN will ask you to view or send us the ShotGrid Desktop app's error logs. To access them, open **ShotGrid Desktop**. In the pulldown menu in the upper right corner, select "**Open Log Folder**". Please ZIP up the all the files in that folder and send them to us with your ticket or email them to <support@artistanywhere.io>.

### Firewall Configuration <a href="#h_01he63dzzf2n35vpghj6cqnvcn" id="h_01he63dzzf2n35vpghj6cqnvcn"></a>

If you are running the Studio Workflow Suite on your own network, you may need to update your firewall rules if you have having trouble connecting to ShotGrid or uploading files. Please pass the following along to your IT department:

* **ShotGrid**: [General Security Ecosystem Documentation](https://help.autodesk.com/view/SGSUB/ENU/?guid=SG_Administrator_ar_general_security_ar_ecosystem_html)&#x20;
* If you are using Aspera on Cloud, you will need to open up studioname.ibmaspera.com. You can get more information from [IBM’s Aspera on Cloud documentation](https://www.ibm.com/docs/en/aspera-on-cloud?topic=basic-system-requirements-firewall-settings-browser-support).
* If using other services including Aspera Enterprise, NightRaven, Content Hub, Signiant, Media Shuttle, or SFTP, please contact us at <support@artistanywhere.io>.

### Connection Errors <a href="#h_01he63fd3fnrg0w7mmf7p24ndt" id="h_01he63fd3fnrg0w7mmf7p24ndt"></a>

You may experience issues with connecting to your ShotGrid website, AWS, Aspera, or any studio-specific file transfer service. If you do, they should display an error in the Ingest Queue app, on the delivery record in ShotGrid, or in the error log.

* **URL, 10X, 40X, and 50X errors:** An error with a numbered error code such as 104, 404 or 502 means that the Ingest Queue lost contact with the server. This is often accompanied by messages like **"urlopen error"**, **"HTTP error"**, **"Client Error"**, or **"Protocol Error"**. To fix this, on the Submission’s Delivery record in Shotgrid, set the Status and Delivery Progress fields back to **Delivered** and the Ingest Queue will try again.
* **CRUD error:** This is a ShotGrid-related error. It means that the user does not have the necessary permissions to create the needed entities in ShotGrid. Please confirm that the user who is running ShotGrid Desktop has permissions to see all fields and entites.&#x20;

{% hint style="success" %}
If you are using **Aspera**, resetting the Submission’s Status and Delivery Progress to **Delivered** will requeue the Submission for ingestion. However, the Submission will be downloaded again at ingest time. To prevent re-downloading, if you are certain that the local files are good, simply drag the Submission folder back onto the Ingest Queue app to rerun its ingestion.
{% endhint %}

### Local Path Errors <a href="#h_01he63qwt14tq30s6vqwmw2qx0" id="h_01he63qwt14tq30s6vqwmw2qx0"></a>

You may come across errors related to expected paths in the file system to which you’re copying ingested files. For external (Vendor) Submissions, the source folder is generally **/deliveries** (or **/downloads**, **/postoffice**, etc.); for Internal Submissions, it can be any path the user specifies. Below are a few we have seen. The fix for most of them is to address the path issue; if you can fix the path for the Ingest Machine, set the Status and Delivery Progress fields on the Delivery for the Submission back to **Delivered**; if not, a new Submission may be necessary. If you have any questions, please reach out to <support@artistanywhere.io>.

* **Permission Denied:** The user does not have access to the folder containing the files to be ingested
* **Source file does not exist on disk:** The Ingest Machine can’t see the source files. If the source files were on the computer that ran the Submission app, it's possible the Ingest Queue cannot see the folder they are in. Similarly, when using services like Box or DropBox, the path often includes the username, so even if Box is mounted on both machines, the path might be different. If this is the case, you should copy the files to your server before ingesting.
* **The system cannot find the path specified:** Same as above.
* **The network path was not found:** Same as above.
* **Bad file descriptor:** This can affect Artist Anywhere users who have dehydrated the source files. Please ensure all files are hydrated first.
* **No space left on device:** Either the server or the temp drive (i.e., C:\Windows\Temp on Windows) on the Ingest Machine has run out of space.

### Template Errors <a href="#h_01he655v0vjz9ywhckzfasm53b" id="h_01he655v0vjz9ywhckzfasm53b"></a>

Our tools use templates to determine file naming and folder structure. Occasionally template errors arise, such as these:

* **Can't evaluate template \_\_\_\_, keys \[ \_\_\_\_ ] are missing:** The ShotGrid entity linked to the media file being ingested is missing a value required to create a folder. For example, to create folders for a Shot, the Shot must have a Sequence assigned to it in ShotGrid, or to create folders for an Asset, the Asset must have an Asset Type assigned.
* **Not a File Sequence, List index out of range, or frame\_range error**: These errors are typically seen when a file is sent with extra "." in the name. Our tools recognize the "." before the extension, and can only recognize one more "." when a frame range is included. One of these errors would be generated if a file included an extra "." but did not get recognized as a file sequence (e.g., myfile.####.exr).
* **Couldn't find a template for entity type \_\_\_\_\_ and format \_\_\_\_\_:** This means someone sent you a file which is accepted but not expected for the particular Entity. For instance, you may configure receiving a .cube file for a Shot but it was not configured for an Asset. Reach out to the submitter to have them resubmit if this is not desired, or to us at <support@redesign-group.com> if the new template is required.
* **Format \_\_\_\_\_ is not contained inside format group \_\_\_\_\_:** Similar to above, please reach out to use to configure.

### Manifest Errors <a href="#h_01he65f7n4vvgezp25j7hzh897" id="h_01he65f7n4vvgezp25j7hzh897"></a>

Our tools include a file called manifest.yml, which carries all of the technical details about the Submssion. This is a required file and errors can sometimes arise when the user does something unexpected with this file. Reach out to us at <support@artistanywhere.io> if you encounter any of the following and cannot address it yourselves:

* **Codec can't decode byte:** When a Vendor drops in a CSV that contains non-Roman characters in our older tools, this error occurs. You can fix it by either creating a new Submission from the source files or having the Vendor resubmit. As of December 2021, all languages should be accepted; if you still receive this error please reach out to us at <support@artistanywhere.io>.&#x20;
* **Delivery mismatch! The given Delivery was intended for a Delivery with id \_\_\_\_\_\_, current one's is \_\_\_\_\_\_:** This occurs when a user creates a second Delivery with the exact same name as an earlier one, and the manifest.yml file had been uploaded already. Deliveries are not overwritten when uploading, so the manifest.yml will be from the earlier Delivery. Please ensure that your users are using unique names for all Submissions, then reset the Status and Delivery progress fields on the Delivery record to **Ready to Upload**, and use [Upload Submission](/artist-anywhere/vendor-workflows/upload-submission.md) to reupload the the Submission.&#x20;
* **Size is wrong for \_\_\_\_\_\_, got \_\_\_\_\_\_, expected \_\_\_\_\_\_:** This happens when the user tries to ingest a Submission that is still in the process of uploading or the file is swapped before uploading. The Submission apps store file size in manifest.yml; if the size changes at any point after the Submission apps are run, the user will receive an error when ingesting. To fix this, create a new Submission with [Vendor Submission](/artist-anywhere/vendor-workflows/vendor-submission.md) or [Internal Submission](/artist-anywhere/i-o-tools/internal-submission.md).

### Transcoding Errors <a href="#h_01he65gb01k5yk6mhv3yhzedm8" id="h_01he65gb01k5yk6mhv3yhzedm8"></a>

Occasional errors occur during the transcoding process, and they fall into two general categories:

* **Upload issues:** Sometimes ShotGrid will error out when trying to upload a transcoded file. This generally happens on a single item like a thumbnail or filmstrip. If the final Versions seem fine, you can change the **Status** and **Delivery Progress** fields on the Delivery record to **Received** and consider it good. If you wish to run it again, set the Status and Delivery Progress back to **Delivered**.
* **Transcoding issues:** There are files which our transcoder cannot handle, and it will fail when attempting to transcode them. Sometimes it is a bad file, sometimes the image is too large, and sometimes a movie is a codec which our tools cannot understand. Feel free to reach out to us at <support@artistanywhere.io> in this case. One possible workaround is to manually add the source file to the **Uploaded Movie** field in ShotGrid after Version creation, in case ShotGrid’s built-in transcoder has different results. If that is done, set the **Status** and **Delivery Progress** fields on the Delivery record to **Received**.

### Common Aspera Errors <a href="#h_01he65h1yw1je75qhzq8d8yfm5" id="h_01he65h1yw1je75qhzq8d8yfm5"></a>

Some errors are unique to Aspera on Cloud (hosted by IBM) and Aspera Enterprise (hosted by the Studio). Feel free to reach out to us with any of the below at <support@artistanywhere.io>.  Here are a few common ones:

* **No such file or directory:** This happens when the expected path is not found on the Aspera server. Often this is because the **Vendor Remote Folder** or **Aspera Remote Folder** on the **Group** record is not configured correctly in ShotGrid. If that is the case, correcting the folder name on the Group will allow the Delivery record’s Status and Delivery Progress fields to be set back to Delivered, and therefore reingested by the Ingest Queue.&#x20;
* **Insufficient permissions:** This arises when a user does not have permissions to see the folder that stores the files they are uploading or downloading. This generally requires talking to the Studio's Aspera support team to correct.
* **Connection aborted:** This is usually a timeout response from Aspera. It is possible that it is a short-term problem, in which case setting the Delivery’s Status and Delivery Progress fields back to Delivered will address it. If you get the same error a second time, please reach out to both us and your Aspera support team.
* **Cannot allocate memory:** This is an issue with the Aspera ASCP application, which our tools leverage under the hood. This type of error generally means that the computer itself has run out of RAM; close all other applications or use a computer with more RAM.
* **Session token \_\_\_\_ has expired:** This generally means something has changed with the user on Aspera and they cannot access the folders any more. Reach out to your Aspera support team about this.
* **Float division by zero, Socket is not connected, Device not configured:** These errors seem to point at a permissions issue with Aspera command-line writing to disk on MacOS. In the Mac's System Preferences > Security & Privacy settings > Privacy, go to Full Disk Access, then allow Terminal


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://artist-anywhere.gitbook.io/artist-anywhere/i-o-tools/ingest-queue.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.