Start Ingest

Table of Contents

Ingest Process Overview
Ingest of a Movie with More Metadata
Ingest of a Movie with Video Relations
Ingest of a Movie with Image Relations
Combined Movie Example
Ingest Example for All Media Types
Checking JSON Validity

Ingest is a process that can create multiple media entities and relations by uploading a single JSON file into the Mosaic Media Service. This guide walks you through the process of preparing ingest files and metadata to perform successful ingest operations.

This guide assumes the following pre-requisites:

You have access to your Mosaic environment with a deployed Media Service from the Mosaic Media Template.
The Video Encoding Profiles are set up, and at least one video is located in the Azure Blob Storage source location.
The Image Acquisition Profile is set up, and there is at least one image in the Azure Blob Storage source location.

Ingest Process Overview

This chapter gives an overview of the OTT media service ingest. It describes how to create a single movie with only its title property filled.

Start by creating a JSON file with the following contents (or download an example file):

{
  "name": "Minimal Movie Ingest",
  "items": [
    {
      "type": "MOVIE",
      "external_id": "avatar_external_id",
      "data": { "title": "Avatar" }
    }
  ]
}

From the Management System Homepage, navigate to the Ingest workflow. In the Ingest Explorer Station, click the New button on the right.

Figure 1. Ingest Explorer station
Click the folder icon next to the Document field. Choose your JSON file and click Open. Start the ingest process by clicking the Proceed button on the right.

Figure 2. New ingest process
This opens the Ingest Process details station that shows the progress for each entity that is ingested.

Figure 3. Ingest progress
When a single ingested item is green, it can be checked in the Movies Explorer and in the movies details page of the imported "Avatar" movie.

Figure 4. Movies Explorer

This is the most basic version of processing an ingest file. However, the process is still the same when using other entity types like TV shows, seasons, or episodes and all possible sets of metadata. Only the uploaded JSON ingest file is different. The next sections describe some more complex ingest examples.

Ingest of a Movie with More Metadata

In the previous example, the ingested movie was almost empty. This example expands on the previous one by adding all possible movie properties.

Create a JSON file with the following contents (or download an example file):

{
  "name": "Movie with Metadata",
  "items": [
    {
      "type": "MOVIE",
      "external_id": "avatar_external_id",
      "data": {
        "title": "Avatar",
        "original_title": "James Cameron's Avatar",
        "description": "Avatar is a 2009 American epic science fiction film...",
        "synopsis": "In 2154, humans have depleted Earth's natural resources...",
        "released": "2009-12-10",
        "studio": "20th Century Fox",
        "tags": ["3D", "SciFi", "Highlight"],
        "cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
        "production_countries": [
          "United States of America",
          "Estonia",
          "Germany",
          "COL",
          "ESP"
        ],
        "genres": ["Sci-Fi", "Animation"],
        "licenses": [
          {
            "start": "2020-08-01T00:00:00.000+00:00",
            "end": "2020-08-30T23:59:59.999+00:00",
            "countries": ["AW", "AT", "FI"]
          }
        ]
      }
    }
  ]
}

The ingest process is the same as in the previous example. The resulting movie looks like this:
1. Some explanations:
  - The ingest process decides whether to add or update a media entity based on its external_id value. The external ID is a unique identifier, defined by the external system that defines the JSON ingest file. In this case, the first example created the Avatar movie with the external ID "avatar_external_id". This second example updated the existing Avatar movie as the external_id in the system matched the one in the second ingest file. If the external_id values were different, a new movie would have been created.
  - The media service defines all movie properties as optional, except for title.
  - original_title, description, synopsis, and studio are all free-text properties.
  - released is a date property and must follow the following format: YYYY-MM-DD
  - tags, cast, and production_countries are array properties in the ingest document that expect free-text values.
  - genres is an array property that references genres that are already defined as "Movies Genres" in the settings area. Performing an ingest and referencing a non-existent genre results in a failure of the metadata update for one specific media entity.
  - licenses is an array of objects:
    
    start and end values should be defined in the date-time ISO 8601 format.
    
    countries is an array of ISO 3166 Alpha-2 country codes.
    
    Each license object must have at least one property.

Ingest of a Movie with Video Relations

So far, the two examples ingested only movie details that have references within the Media Service (movies to genres, tags, …). However, it is also possible to orchestrate the ingest across service boundaries. A movie entity and other entities can have a video relation. Videos are managed and processed by the Axinom Encoding Service. To be processed, video files must be available in the blob storage. The video encoder process gets the access credentials from the Video Transcoding Acquisition profile.

This example assumes that the acquisition profile is set up correctly and the blob storage container is defined as source. To make sure of this, you can go to the Video Transcoding Acquisition profile details page and check the Secure Storage Location value. The container name is the last segment of a URL.

Providing a Source Video File

When working with videos, each video should be put under its own folder with all its files (e.g. the video stream, audio tracks, subtitle, and caption files). This example provides a sample video with all related files that you can put into your blob storage. Download the zip file, unpack it (e.g. with 7-zip), and upload the resulting folder into your blob storage.

The provisioning of the videos is likely done by your content service provider. In this example, we use the Azure Storage Explorer to manually upload a video into the source container.

Figure 5. Microsoft Azure Storage Explorer

Under Local & Attached, right-click Storage Accounts.
Select Connect to Azure Storage.
Choose a Use a connection string option.
Paste your Azure storage connection string value into the Connection String field.
Click connect and a connection should appear in the Storage Explorer UI.

Figure 6. Blob Storage Container "source"
Upload the extracted video folder into the source blob container as "test_video"`. It should look like this:

Figure 7. Source container root level

Figure 8. Contents of the "test_video" folder

Start an Ingest Process with the Main Video

Now that the video file is available in the blob storage, it can be referenced in a JSON file.

Create a JSON file with the following contents (or download an example file):
```
{
  "name": "Movie with Video",
  "items": [
    {
      "type": "MOVIE",
      "external_id": "avatar_external_id",
      "data": {
        "title": "Avatar",
        "main_video": {
          "source": "test_video",
          "profile": "DEFAULT"
        }
      }
    }
  ]
}
```
- source is a file path to a folder in blob storage that contains the files for a video. In this case, the folder is located in the root of blob storage, so it’s just a folder name.
- profile defines, which of the Video Processing profiles should be used. This value is optional. If it isn’t specified, the default processing profile is used.
After this file is ingested, the existing "Avatar" movie has a video relation to the "test_video".

Figure 9. Ingest status of the metadata update and video import

It’s important to note, that the video status does not reflect the actual video transcoding status. It denotes that the video transcoding has successfully started, a database entry for the video has been created, and the video is assigned to the Avatar movie.

Start an Ingest Process with Trailer Videos

Each movie can have only one main video but multiple trailers. The process of preparing the ingest file with video trailer data is very similar to the main video example. For this particular example, the movie should get two trailers, and the trailers should have a dedicated sub-folder named trailers, located in blob storage container root. You can use the same example video files also for the trailers.

Upload the example video files into the folder trailers/test_video_1 and trailers/test_video_2, respectively:

Figure 10. Trailer video folders
Each folder should have the same example video files in them:

Figure 11. Example video files in the trailer folders
The container root should now include two folders: test_video and trailers.

Figure 12. Root "source" folders

Create a JSON file with the following contents (or download an example file):

{
  "name": "Movie with Trailers",
  "items": [
    {
      "type": "MOVIE",
      "external_id": "avatar_external_id",
      "data": {
        "title": "Avatar",
        "trailers": [
          {
            "source": "trailers/test_video_1",
            "profile": "DEFAULT"
          },
          {
            "source": "trailers/test_video_2",
            "profile": "DEFAULT"
          }
        ]
      }
    }
  ]
}

Note	This time we used a relative path to a trailer folder instead of just the folder name. The same logic would have applied to the main video, if the main video folder was located in a subfolder.

After this file is ingested, the movie has relations to the trailer videos.

Ingest of a Movie with Image Relations

The preparations for an ingest that references images are very similar to video ingest. Each image is just a single file. It does not consist of potentially multiple files like videos do. Therefore, all images can be copied to the root blob storage container directly if you want to.

To ingest a movie with image relations:

You need to have access to the blob storage container that is defined in the Image Acquisition Profile. Use the Microsoft Azure Storage Explorer to connect to the image source blob storage, the same way it was done for videos.
Upload any two images as avatar_1.jpg and avatar_2.jpg.

Figure 13. Images source root folder

Create a JSON file with the following contents (or download an example file):

{
  "name": "Movie with images",
  "items": [
    {
      "type": "MOVIE",
      "external_id": "avatar_external_id",
      "data": {
        "title": "Avatar",
        "images": [
          {
            "path": "avatar_1.jpg",
            "type": "COVER"
          },
          {
            "path": "avatar_2.jpg",
            "type": "TEASER"
          }
        ]
      }
    }
  ]
}

path is a file path to an image file in blob storage. In this case, the images are located in the root of the blob storage container, so it’s just a file name.
type is a pre-defined value for the image type that the movie supports. The Mosaic Media Template supports the COVER and TEASER image types.

After this file is ingested, the Avatar movie shall have relations to the two images.

Combined Movie Example

The chapters above described how to ingest different parts of movie relations and metadata. A combined example would look like the following (download JSON):

{
  "name": "Full Movie",
  "items": [
    {
      "type": "MOVIE",
      "external_id": "avatar_external_id",
      "data": {
        "title": "Avatar",
        "original_title": "James Cameron's Avatar",
        "description": "Avatar is a 2009 American epic science fiction film...",
        "synopsis": "In 2154, humans have depleted Earth's natural resources...",
        "released": "2009-12-10",
        "studio": "20th Century Fox",
        "tags": ["3D", "SciFi", "Highlight"],
        "cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
        "production_countries": [
          "United States of America",
          "Estonia",
          "Germany",
          "COL",
          "ESP"
        ],
        "genres": ["Sci-Fi", "Animation"],
        "licenses": [
          {
            "start": "2020-08-01T00:00:00.000+00:00",
            "end": "2020-08-30T23:59:59.999+00:00",
            "countries": ["AW", "AT", "FI"]
          }
        ],
        "main_video": {
          "source": "test_video",
          "profile": "DEFAULT"
        },
        "trailers": [
          {
            "source": "trailers/test_video_1",
            "profile": "DEFAULT"
          },
          {
            "source": "trailers/test_video_2",
            "profile": "DEFAULT"
          }
        ],
        "images": [
          {
            "path": "avatar_1.jpg",
            "type": "COVER"
          },
          {
            "path": "avatar_2.jpg",
            "type": "TEASER"
          }
        ]
      }
    }
  ]
}

Ingest Example for All Media Types

Apart from Movies, the OTT media service also supports TV Shows, Seasons, and Episodes. Those can also be ingested in the same ingest JSON file. There are some differences between these types. Overall, the process is the same. Here is an example for all types (JSON file, image files, and video files):

{
  "name": "Full Ingest Example",
  "items": [
    {
      "type": "MOVIE",
      "external_id": "avatar",
      "data": {
        "title": "Avatar",
        "original_title": "James Cameron's Avatar",
        "description": "Avatar is a 2009 American epic science fiction film...",
        "synopsis": "In 2154, humans have depleted Earth's natural resources...",
        "released": "2009-12-10",
        "studio": "20th Century Fox",
        "tags": ["3D", "SciFi", "Highlight"],
        "cast": ["Sam Worthington", "Zoe Saldana", "Sigourney Weaver"],
        "production_countries": [
          "United States of America",
          "Estonia",
          "Germany",
          "COL",
          "ESP"
        ],
        "genres": ["Sci-Fi", "Animation"],
        "licenses": [
          {
            "start": "2020-08-01T00:00:00.000+00:00",
            "end": "2020-08-30T23:59:59.999+00:00",
            "countries": ["AW", "AT", "FI"]
          }
        ],
        "main_video": {
          "source": "example_videos/videos/avatar",
          "profile": "DEFAULT"
        },
        "trailers": [
          {
            "source": "example_videos/trailers/avatar_1",
            "profile": "DEFAULT"
          },
          {
            "source": "example_videos/trailers/avatar_2",
            "profile": "DEFAULT"
          }
        ],
        "images": [
          {
            "path": "example_images/covers/avatar_1.jpg",
            "type": "COVER"
          },
          {
            "path": "example_images/teasers/avatar_1.jpg",
            "type": "TEASER"
          }
        ]
      }
    },
    {
      "type": "EPISODE",
      "external_id": "mandalorian_s2_e1",
      "data": {
        "index": 1,
        "parent_external_id": "mandalorian_s2",
        "title": "The Marshal",
        "original_title": "Chapter 9: The Marshal",
        "description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
        "synopsis": "The Mandalorian is drawn to the Outer Rim in search of others of his kind.",
        "released": "2020-10-30",
        "studio": "Lucasfilm",
        "tags": ["star wars", "mandalorian", "bounty hunter"],
        "genres": ["Action", "Adventure", "Sci-Fi"],
        "cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
        "production_countries": ["USA"],
        "licenses": [
          {
            "start": "2020-08-01T00:00:00.000+00:00",
            "end": "2020-08-30T23:59:59.999+00:00",
            "countries": ["AW", "AT", "FI"]
          }
        ],
        "main_video": {
          "source": "example_videos/videos/mandalorian_s2_e1",
          "profile": "DEFAULT"
        },
        "trailers": [
          {
            "source": "example_videos/trailers/mandalorian_s2_e1_t1",
            "profile": "DEFAULT"
          },
          {
            "source": "example_videos/trailers/mandalorian_s2_e1_t2",
            "profile": "DEFAULT"
          }
        ],
        "images": [
          {
            "path": "example_images/covers/mandalorian_s2_e1_t1.jpg",
            "type": "COVER"
          },
          {
            "path": "example_images/teasers/mandalorian_s2_e1_t1.jpg",
            "type": "TEASER"
          }
        ]
      }
    },
    {
      "type": "SEASON",
      "external_id": "mandalorian_s2",
      "data": {
        "index": 2,
        "parent_external_id": "mandalorian",
        "description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
        "synopsis": "A Mandalorian bounty hunter tracks a target for a well-paying, mysterious client.",
        "released": "2020-10-30",
        "studio": "Lucasfilm",
        "tags": ["star wars", "mandalorian", "bounty hunter"],
        "genres": ["Action", "Adventure", "Sci-Fi"],
        "cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
        "production_countries": ["USA"],
        "licenses": [
          {
            "start": "2020-08-01T00:00:00.000+00:00",
            "end": "2020-08-30T23:59:59.999+00:00",
            "countries": ["AW", "AT", "FI"]
          }
        ],
        "trailers": [
          {
            "source": "example_videos/trailers/mandalorian_s2_t1",
            "profile": "DEFAULT"
          },
          {
            "source": "example_videos/trailers/mandalorian_s2_t2",
            "profile": "DEFAULT"
          }
        ],
        "images": [
          {
            "path": "example_images/covers/mandalorian_s2.jpg",
            "type": "COVER"
          },
          {
            "path": "example_images/teasers/mandalorian_s2.jpg",
            "type": "TEASER"
          }
        ]
      }
    },
    {
      "type": "TVSHOW",
      "external_id": "mandalorian",
      "data": {
        "title": "Mandalorian",
        "original_title": "The Mandalorian",
        "description": "After the stories of Jango and Boba Fett, another warrior emerges in the Star Wars universe...",
        "synopsis": "The travels of a lone bounty hunter in the outer reaches of the galaxy, far from the authority of the New Republic.",
        "released": "2019-11-12",
        "studio": "Lucasfilm",
        "tags": ["star wars", "mandalorian", "bounty hunter"],
        "genres": ["Action", "Adventure", "Sci-Fi"],
        "cast": ["Pedro Pascal", "Carl Weathers", "Gina Carano"],
        "production_countries": ["USA"],
        "licenses": [
          {
            "start": "2020-08-01T00:00:00.000+00:00",
            "end": "2020-08-30T23:59:59.999+00:00",
            "countries": ["AW", "AT", "FI"]
          }
        ],
        "trailers": [
          {
            "source": "example_videos/trailers/mandalorian_t1",
            "profile": "DEFAULT"
          },
          {
            "source": "example_videos/trailers/mandalorian_t2",
            "profile": "DEFAULT"
          }
        ],
        "images": [
          {
            "path": "example_images/covers/mandalorian_1.jpg",
            "type": "COVER"
          },
          {
            "path": "example_images/teasers/mandalorian_1.jpg",
            "type": "TEASER"
          }
        ]
      }
    }
  ]
}

All four entity types have similar metadata, but there are some differences:

A season does not have a title but a required numeric index property.
An episode has both a title and a numeric index property which are both required.
All entity types can have trailers, but only Movie and Episode types can have a main_video.
A season has a parent_external_id which should have a value of a TV Show external_id. This is used to link a season to a TV show.
An episode has a parent_external_id which should have a value of a Season external_id. This is used to link an Episode to a Season.

Checking JSON Validity

When a JSON file is uploaded, it is validated against a JSON schema that is defined in the OTT media service. This schema can be used to check if your JSON ingest file is valid without uploading it to the OTT media service. You can use an online tool like https://www.jsonschemavalidator.net/ to get the validation result.

There are also custom validation rules that are not reflected by the JSON schema. There is, for example, a check for duplicate external_ids, since those should be unique per item type. There is no way to check for those custom rules without actually uploading the JSON file.

The schema file provided in this chapter is a copy of the schema that is provided (and potentially customized) in the OTT media service. Please use your custom/latest version of this file to use it for validations.