Migration guide

Introduction

HERE platform projects are resource containers specific to your organization that allow you to group and manage access to platform resources. Projects let you easily manage access control and track resource usage, both via the platform portal and command line interface (CLI).

We recommend that you use projects to manage all your platform resources. This guide walks you through the mostly CLI-based steps to migrate resources for a use case—including catalogs, schemas, pipelines and associated pipeline templates—into a platform project, if those resources were not originally created in a platform project. Once your catalogs for a use case are in a project, you can still share them with all or specific projects in your organization so that they can be reused for other use cases.

Terminology

Home project: This typically means the project in which a resource was created. For this migration guide, it means the project to which you are adding or moving resources. This project now becomes the "home project" for those resources. A resource can only have one home project, but catalogs can be shared to other projects in your organization so that they can be reused in multiple use cases. Storage for a catalog is logged to the home project, and data I/O for using the catalog is logged to the project that's using it.

Adding vs. moving resources: This guide refers to adding catalogs to projects. This means that you are adding the catalog to a project, but not removing any permissions associated with the catalog outside the context of the project. You can choose to remove those permissions later and purely manage the resource in the project, but "adding" rather than completely "moving" the catalog into a project helps mitigate the risk of breaking anything as you migrate your use case to a project. With pipelines, the guide refers to moving them rather than adding them, that is, they will not be accessible outside of project context once moved.

Migration steps

• Step 1: Create a project for your use case
• Step 2: Link HERE catalogs to your project
• Step 3: Add input and output catalogs to your project using the CLI
• Step 4: Add members to your project and manage their access to project resources
• Step 5: Move pipeline(s) and related pipeline template(s) to a project
• Step 6: Recreate batch pipeline(s) in the project (alternative to step 5)
• Step 7: Cancel the batch pipeline version(s) outside the project (alternative to step 5)
• Step 8: Activate the batch pipeline version(s) in the project (alternative to step 5)
• Step 9: Update automated deployments to work in the project context
• Step 10: Remove access to catalogs outside of project context (optional)

Step 1: Create a project for your use case

Create a project using the platform portal Projects Manager or the olp project create CLI command.

Step 2: Link HERE catalogs to your project

Link any HERE catalogs used in your use case to the project you created in step 1 using the platform portal or the olp project resource link availability list and olp project resources link CLI commands.

To link a HERE catalog from the platform portal, go to the Projects Manager, click the project to which you want to add a catalog, and select the Resources tab. Click Add catalog and choose to Link a catalog. A list of available HERE catalogs will be displayed.

IMPORTANT: Link the "HERE Optimized Map for Visualization Plus" catalog (hrn:here:data::olp-here:here-optimized-map-for-visualization-plus-2) to your project so Data Inspector continues to work seamlessly. This will allow you to visually inspect layers of catalogs in your projects (Data Inspector uses the Optimized Map for Visualization to provide the background map).

Step 3: Add input and output catalogs and schemas to your project using the CLI

IMPORTANT: Ensure at least one app has manage access to your input and output catalogs before you add the catalogs to projects so that you can continue to manage access outside of projects. Once you add your catalogs to projects, you won't be able to manage access to them outside the project via the portal but can continue to do so via the CLI, which requires an app to authenticate.

Add your input and output catalogs to the project you created in step 1 using the olp resource project add CLI command.

Permissions outside of the project will not be impacted, that is, any pipelines that use these catalogs will continue to run.

Access to schemas associated to a catalog will be available in the project scope without any additional work required. However, you can also add schemas you created to a project using the olp resource project add CLI command.

What if you make a mistake? Catalogs and schemas created outside of a project can be removed from a project in order to help you correct a mistake in this step of the process. Just use the olp resource project remove CLI command. If the catalog or schema is shared to other projects before it's removed from a project, this action revokes its availability to the other projects. If the catalog or schema is shared and also linked to another project before it's removed from its home project, this action revokes those links.

Step 4: Add members to your project and manage their access to project resources

Add members (users, groups, apps) to your project using the olp project access grant CLI command or the platform portal.

To add a member to a project using the platform portal:

1. In the Projects Manager, click the project to which you want to give a user, group, or an app access.
2. Select the Access and permissions tab to view a list of users, groups, and apps with access to the project.
3. Click Give access.
4. Select from either user, group, or app type.
5. Enter the name in the search field to find the desired user, group, or app by name.
6. Select the desired name and click Give access. The name is now listed.

IMPORTANT: Add run-as ID (aka runtime credentials) for pipeline versions as member(s) of the project.

Full access to all resources in the project will be granted to these project members by default. You can optionally restrict access to resources by a project member by applying existing HERE or custom-created policies using the CLI. The CLI Projects Workflows explain how to set up granular access to the project resources through project policies. The Project Policy Commands allow you to create and manage custom policies.

Restrict access to catalogs by run-as ID (aka runtime credentials) if desired (optional), but make sure they have the minimum required rights of reading from input catalog(s) and writing to output catalog(s).

Step 5: Move pipeline(s) and related pipeline template(s) to a project

This step describes how to move a pipeline created outside of a project into a project. For batch pipelines, you may prefer to simply recreate the pipeline, cancel the versions outside the project, then activate the versions in the project, as described in steps 6-8 below.

Start by checking the pipeline versions associated with the pipeline you want to move to determine if the 5 most recent (the maximum supported) or fewer should be moved. You can do this check using the olp pipeline version list CLI command.

Next, check to see if other pipelines are using pipeline templates used by pipeline versions you need. Use the --report parameter of the olp pipeline move CLI command. If they are, create separate pipeline templates, either for use in this pipeline or the other pipelines.

Now you're ready to move the pipeline, specified pipeline versions, and associated pipeline templates to the project using the olp pipeline move CLI command.

IMPORTANT: The pipeline job will continue to run with the run-as ID (aka runtime credentials) using existing permissions to catalogs outside of the project from before the move. Verify the pipeline job is running as expected. The pipeline should be re-deployed with a new job to switch it to use the run-as ID (aka runtime credentials) and catalogs in project scope, and report the usage in project scope. To re-deploy, you can pause and resume the running pipeline version. You can also deactivate and activate the pipeline version if there is no processing state to be preserved.

Note: The olp pipeline move command revokes permissions attached to the group that was associated to the pipeline outside of project context, without impacting the permissions inside the project. This means you can now only access the pipeline through the project.

Pipeline versions that were not included in the move cannot be reactivated after the move, because their associated templates were not moved.

What if you make a mistake? Use the --revert-to-group parameter of the olp pipeline move CLI command to remove the pipeline, pipeline versions, and associated pipeline templates from the project and add them back to the group that they were in before the migration. Note that the rollback option is not available if any of the following are true:

• More than 5 calendar days have passed since migration.
• The original group has been deleted.
• One or more new pipeline templates have been created and associated to new pipeline version after the move.
• One or more moved pipeline templates have been deleted after the move.
• One or more associated pipeline templates have been shared and/or linked with other projects.
• One or more associated pipeline templates are referenced by a different pipeline's versions (within the same project).

Step 6: Recreate batch pipeline(s) in the project (alternative to step 5)

Recreate batch pipeline(s) in the project (including pipeline template, pipeline, and first pipeline version) using the CLI or platform portal. Ensure that you specify project scope, regardless of the interface. If you use the platform portal and use the Pipelines tab instead of the Projects Manager, ensure that the project is selected in the drop-down menu in the top right-hand side of the screen.

Step 7: Cancel the batch pipeline version(s) outside the project (alternative to step 5)

Cancel the pipeline version(s) outside the project using the olp pipeline version cancel CLI command or platform portal.

Step 8: Activate the batch pipeline version(s) in the project (alternative to step 5)

Ensure the cancellation in step 7 is complete, then activate the pipeline version(s) in the project using the olp activate pipeline version CLI command or platform portal.

Step 9: Update automated deployments to work in the project context

Update automated deployment access to resources to be in the context of your new project by specifying the project in the --scope parameter with CLI commands, in the here.token.scope optional parameter of the credentials file, or by specifying a default project for your app using the olp app scope CLI command. Read more about the order in which these three methods of specifying scope are resolved in the "Scopes" section here.

Step 10: Remove access to catalogs outside of project context (optional)

Once you have validated that your use case is working inside the project, you may choose to remove access to catalogs outside of the project context. This will ensure that access to those catalogs is now managed only within the context of the project. In order to remove permissions on the catalog outside of the project, use the olp catalog permission list and the olp catalog permission revoke CLI commands.