Overview
This is a streamlined guide for creating a new file-based (batch) Destination configuration -- using the Adobe Experience Platform (AEP) Destination SDK -- and testing it in your AEP sandbox. The target audience is an Adobe technology (ISV) partner who desires to publish a file based Destination connector in the AEP Destination catalog for use by all Adobe RT-CDP customers. This guide will not cover all of the configuration options available for making a customized Destination that provides you with the file specs and data patterns you need -- for that see the full Destination SDK documentation. Instead, this guide takes you quickly through all the steps, giving you a solid overview of the technology and the process. This will help you to confidently scope and resource your own Destination creation project.
Prerequisites
To successfully complete the steps in this guide you will need the following:
- Membership in the Adobe Technology Partner Program (formerly called Adobe Exchange program)
- Access to an AEP sandbox environment
- An Adobe developer console project with "Experience Platform API" added; API credentials need to be assigned to Role by admin
- Access to Postman and basic experience with it
- For end-to-end testing, an Amazon S3 bucket will be needed along with the necessary configurations to allow for third party access, and an AWS key and secret to connect AEP.
Postman Prep
In this guide, we will execute several API calls using Postman. Before we do that, we need to import a Postman collection and a Postman environment. We will also need to make a few additions to the environment.
First, download the Create a File-based Destination collection and import it into your workspace.
Second, download the Postman environment file from the Adobe developer console project UI and import it.
Figure 1: Developer Console page to download environment file for Postman
Third, we need to augment the environment with a few additional values.
- Add a new environment variable called "PLATFORM_GATEWAY" with the value "https://platform.adobe.io"
- Add a new environment variable called "SANDBOX_NAME" with the value of your sandbox name. To find the sandbox name, log into your AEP instance and look in the URL for the value after “sname:”. In the following example, the name is “mysandbox”
ex. https://experience.adobe.com/#/@exchangeinternal/sname:mysandbox/platform/home
- Add a new environment variable called "UNIQUE_NAME" to provide a unique label for each time you might go through this tutorial. This is going to be used throughout the calls to set up different aspects of AEP. Initially you can give it a value like - - “tutorial” (if you ran through it a second time you could use “tutorial 2”)
Figure 2: Postman Environment with additional fields ‘PLATFORM_GATEWAY’, ‘SANDBOX_NAME’, and ‘UNIQUE_NAME’
Generate an Access Token
Now that Postman is set up, let’s generate an access token so that we can make all the API calls in the remaining sections.
Open the “Create a File-based Destination” collection in your Postman app. Expand the first subfolder, named “Authentication”, and you will see one request. Run this request.
You’ll know that this was successful if you see a 200 response and if you now have a ‘Current value’ for your ACCESS_TOKEN environment variable.
Figure 3: Postman Environment showing generated Access Token’s Current Value
Sandbox Setup
Now we will be creating the Schema, Dataset and Segment.
The Schema is a blueprint for the data to be stored in Adobe Experience Platform. A schema is made up of a Class and zero or more Field Groups.
A Dataset is a storage and management construct for collecting data. A dataset is always linked with a schema.
Segments is a way to narrow down a large group of profiles by using Segment Definitions, resulting in an Audience.
Start by Making sure to change the UNIQUE_NAME in the environment variables to something unique when running the folder. You might have to scroll down the environment variable list to see it. If you run the postman calls a second time, you can change the UNIQUE_NAME value to prevent naming collisions in the objects that are created.
Figure 4: Postman Environment showing a distinctive ‘UNIQUE_NAME’ value
We can create all the above by running through the postman collection “Sandbox Setup”. Right click the folder and hit “Run folder” (Make sure you have generated an access token, or this step will not work!)
Figure 5: Postman workspace showing how to run the ’Sandbox Setup’ folder
Figure 6: Postman workspace showing options on running the ’Sandbox Setup’ folder
Verify that all 5 calls were given a 200 or 201 success message.
Now that your sandbox has been set up, we will load test data so that you can test your connection once set up.
Click Here to download the Mock Data JSON file.
Next, log into your AEP instance in your browser. You will do this by going to https://experience.adobe.com , then click the 9 dots at the top right to open the applications menu, then select Adobe Experience Platform. Once logged in, scroll down the left navigation bar and click on “Datasets” under the Data Management section.
Figure 7: The Adobe Experience Platform UI at login and showing where to find the “Datasets” UI
Once there, you should see the dataset that was created as part of the sandbox setup API calls (see figure 8 below). It should include the Unique Name you provided. Click the blue area to navigate to the datasets detail view.
Figure 8: This is the Datasets Browse view. Here you should be able to find your dataset, and click the blue link to go into the datasets detail view
Now that you are on the dataset detail page, scroll down on the right hand side to find the ‘drag and drop files’ section. Here you can drag and drop or use the “Choose Files” button to upload the Mock_Data.JSON file
Figure 9: The dataset detail view showing where to find the area to drag and drop your Mock_Data JSON file
Once you have started the upload, you will see the UI update to show the batch, along with the status of the upload. Click on the Batch ID to see a more detailed view of the upload status.
Figure 10: the dataset detail view once the file is uploaded. The boxed area shows the status, and you can click the blue area to get details about the batch
Once the upload is complete, the status should change to “Success”. This may take a few minutes, and you may require a page refresh to see the status updated.
Figure 11: the Batch overview screen showing Success status
That's it. We are now ready to create the Destination connector.
Create a Destination
In this section, we will create a branded card in Adobe's Destination Catalog as well as the workflow for your connector.
Please refer to ‘Destination Setup’ folder in the postman collection
Figure 12: Postman collection showing the ‘Destination Setup’ folder
The End-to-end process involves:
1. Create a destination server – This is configuration for how to connect your desired cloud storage location
2. Create a destination config – This is where you will configure the rest of the connector, including what the user will see in the destination setup UI workflow. This includes:
- The card information found in the UI
- Data fields collected for you to identify the client on your end (bucket name, client id, etc)
- How they will be able to map their data to what you need (establishing what data you want, if they can send added info or require to only send certain data, etc)
- Frequency and timing of the file delivery
3. Create a Destination instance in the UI – going through setting up the connector you have made to see the UI experience and do end-to-end testing.
4. Monitor Destination batch process and confirm the batch is received
1. Create a destination server
In Postman, open the ‘Server Template Configs’ folder and the ‘Create a Destination Server Template’ request.
Figure 13: Postman collection showing ‘Server Template Configs’ folder and ‘Create a Destination Server Template’ request
If you look at the body, you will see that this is where you establish the type of destination (File based S3 in this situation as show in Figure 14) as well as other configurations for the connector workflow that will be presented in the UI, and that you will use to connect to your system. Variables captured in the UI are accessed using the customerData object as seen in the body of this call. Configuration of those fields will be in the ‘Create a Destination Configuration’ call and you will reference them here. No alterations will be needed for this call for our demo setup, so you can run this call to set up the server. If you are looking for more information on how to set up alternative destination types, the page on Configuring Destination Servers will provide example code for other server types and will provide more details on the parameters.
Figure 14: Postman ‘Create a Destination Server Template’ server type is set to S3 bucket and you can see that some details of the connection can be pulled based on the customerData object which will provide data input from the user during their Destination setup in the UI
You’ll see a 201 response if the Destination Server template has been created successfully.
2. Create a destination
Open the ‘Create a Destination Configuration’ request.
Figure 15: Postman collection showing the body of ‘Create a Destination Config’ request. The name and description will show up in the UI when searching for a destination to set up among other things.
The value of the ‘name’ field provided in this call is what appears in the Destination Card of the catalog. The default here will be that Unique name you provided, but you can change this if you wish. The description will also show up on the right-hand side of the page if you click on the card.
Further down in the body of the API, there is a section for customerDataFields. These will be fields in the UI that will be required by the user to fill out during their connector configuration.
Figure 16: ‘Create a Destination Config’ request body had the customerDataFields section to customize the configuration UI for a user.
The Create a Destination Config body has a lot of control for what will be shown in the UI. This is where you edit -
- the card shown in the catalog
- the custom fields shown on the configuration step
- configure your job if you want to a specific frequency, schedule, etc.
- dictate what the data mapping experience will be
Our end-to-end run through following this will help you get a better understanding of this. No alteration should be needed for this, so go ahead and run the call. A 201 response means a Destination instance was created.
We have created a Destination and we should be able to see our Destination card appearing in the Destination Catalog. To verify this, login to https://platform.adobe.com and verify that the correct Org and sandbox are selected in the upper right.
3. Create a Destination instance in the UI
Navigate to the Destinations section on the left-hand side indicated by number 1 in Figure 17.
Once there, make sure you are on the Catalog tab, and search for the Destination name; in this example we have used the name ‘Tutorial’. Click the ‘Set up’ button to begin setting up your Destination instance.
Figure 17: Once logged in you will find your card by going to Destinations, then using the Catalog tab. Then search or scroll to find it.
As a prerequisite, we had mentioned you will need your Sandbox to allow for a third party to add files to your bucket. In this first step, you will want to provide your access key and secret that you have set up for your S3 Bucket as shown in Figure 18. Then click ‘Connect to destination’.
Figure 18: AEP Destination Configuration of Connection type which will require S3 credentials to set up
You will be presented with additional data fields. This is where your custom data fields will be added using the ‘Create a Destination Config’ API. You will want to fill out the necessary information to match your S3 Bucket.
Figure 19: Fill this form to connect to your specific S3 bucket
Under the ‘Governance’ step, choose the appropriate Marketing Actions that are acceptable for this destination and click the “Next” button. Again, what you select is not important for this exercise.
Figure 20: AEP Destination Configuration Governance Marketing Actions page
Under the ‘Select Segments’ step, Choose the Segments to be activated to this Destination and select next, in this exercise you should see the segment using your unique Name value that was automatically created earlier in the Sandbox Setup. Your segment might have a slightly different name.
Figure 20: AEP Destination Configuration Segment Selection page
Next up, you will need to configure your schedule for your job to run. First, click on the ‘Create schedule’ option.
Figure 21: the scheduling screen is where you can click the Create schedule option to configure the time and frequency of your batch job
Once you click the “Create schedule” option, you can figure the batch frequency and timing as you wish. In our example shown below, I have chosen to export full files (1) as a one-time job (3) and requested to send it after segment evaluation (2). Please note that if you do a scheduled job, the time zone of the time you are selecting is UTC time as it will indicate. Once done, click ‘Create’.
Figure 22: the job scheduler allows you to configure frequency, scope, and timing of your batch.
Now that you have created the schedule, click to begin the Mapping step.
In the Mapping step, we are going to define the fields that our Destination accepts. In the Create Destination Config API, you can configure this to allow for full flexibility, to require certain fields, or to make this rigid in nature where you just ask them to map to specific options only. For this example, the API call had 3 required fields.
Figure 23: user will map fields in their database (source fields on the left) with the fields you expect to receive (target field on the right)
Source Fields are the fields from AEP and the Target Fields are the fields defined when we created our Destination. You can make your target fields non-editable as in this example, while the customer would choose which source fields to select.
We have selected the firstName, lastName, and Email fields to be exported. You will need to select the source field to map to the target field. If you type the values below, the values will start to prepopulate. Alternatively, you can click the pointer icon to open up a browse option. Below are the 3 values to choose for this Demo
- Source: xdm:personalEmail.address -> Target:Email
- Source: xdm:person.name.firstName -> Target: FirstName
- Source: xdm: person.name.lastName -> Target: LastName
Once the mapping is done, press the Next button. You will then see a confirmation page with an option to select Finish. This completes the loop of creating a Destination through API, Configuring the Destination Instance in the UI, activating a Segment to be exported.
Once activated, you will be presented with a summary screen that has a row showing each audience and providing some information on file name and timing of the batch.
Figure 24: AEP Destinations Configuration complete page with activated segments.
4. Monitor Batch Process and Confirm File is Received
For Step 4, we will monitor your batch job to confirm it runs successfully. Navigate to Destinations in the left hand sidebar. Once there, ensure you are on the Browse option. If so, you will see a screen that has your destination, and the status of the last job. Here, you will notice it shows No Run. You will want to check in on this status once your job was expected to run. This will be updated to success when complete.
Figure 25: showing the Destination Browse screen which allows you to see when your job was last run. This shows no run initially, but will be updated to Success once the job runs.
Once you have confirmed the batch was successful, the next step is to confirm the file is delivered in your S3 bucket!
Summary and Next Steps
In this guide you have created and tested a new Destination connector within your AEP sandbox. Once published this connector will be available to AEP customers that have access to the public Destinations Catalog. They can then use the connector to push audiences and profile data to your platform.
However, before you offer this Destination connector to AEP customers you will need to go through this exercise again and create a connector that better matches the requirements of your file ingestion process. You can modify the Postman requests provided and make the needed adjustments to fit the naming, authentication, data mapping (etc.) to configure a successful connector for your customers.
See the full Destination SDK documentation.
After you have configured, tested and documented your destination, you can use the destination publishing endpoint to submit it to Adobe for review and publishing. Read more in the article on submitting your destination for review.