Working with Most recent file in an storage connector

  • 24 August 2022
  • 0 replies
  • 27 views

Userlevel 5
Badge +5
  • Community Manager
  • 160 replies

When working with storage connectors, there are two different import modes, Fixed file or Most recent file.  This article will discuss Fixed file but go into a more in-depth look at how to use Most recent file.

 

Table of Contents

 

Fixed File 

 

This import mode will choose one individual file name.

Just input the file name and location (ex: folder1/folder2/file.csv) and click on Import.

This is a good set up if your file’s name will not change or you only wanted to update a single file with the storage connector.  If the configuration is saved and imported again, it will take the latest version of that file, as long as the name hasn’t changed.

 

Most recent file

 

This import mode allows you to define a path and Pigment will upload the most recent file depending on a defined naming convention that must contain a ISO-8601 compatible date format within it.

 

Naming convention in storage connector  

When naming your files in the storage connector, your file name must follow a consistent naming convention with the date included.  It doesn’t matter where the date is in the name as long as it uses the ISO-8601 format and it remains consistent. 

For example, you could have 2022-09-30-rev.csv, where the date is in the front and -rev is at the end.  You could also have rev-2022-09-30.csv.  The important thing is to remain consistent with all files after that.  Once you define the naming convention, you’ll have to write the correct path and naming convention in Pigment. 

 

Here are some examples of file names for a rev.csv with the ISO-8601 compatible date formats located at the start.

 

ISO-8601 compatible date formats example of file names {{date}}-rev.csv naming convention
yyyy-MM-dd 2022-09-30-rev.csv
yyyy-MM-ddThh:mm:ss 2022-09-30T00:00:00-rev.csv
yyyy-MM-ddThh:mm:ssZ 2022-09-30T00:00:00Z-rev.csv

 

When writing the file path in Pigment

 

Similar to the Fixed file, you must first define the path with the location(ex: folder1/folder2/ ) ,then you must define file naming convention with a date included in the file name.  When entering the case sensitive path in Pigment, use {{date}} where the date is located. 

It should look something like this  folder1/folder2/file-{{date}}.csv

Here are some examples of file names and the results depending on what path you entered.

If the bucket contains the following files:

  • /myfolderA/2022-09-15-salaries.csv
  • /myfolderA/2022-10-15-salaries.csv
  • /myfolderA/2022-10-16-revenue.csv
  • /myfolderB/2022-11-15-salaries.csv

Here are the results I would get depending on my file path :

  • path "myfolderA/{{date}}-salaries.csv" will import /myfolderA/2022-10-15-salaries.csv because this is the most recent file in myfolderA.
  • path "myfolderB/{{date}}-salaries.csv" will import /myfolderB/2022-11-15-salaries.csv because this is the most recent file in myfolderB.
  • path "/{{date}}-salaries.csv" will import nothing because its missing the folder location.

 

In order to get the Start Import button to be clickable, you must have {{date}}  and .csv and in the pattern.

 

Example

Here is an example from a an S3 folder.  Take a look at the naming convention and see if you can input the correct

 

 

Click Show Content below to see the correct path needed to pull the most recent file.

folder/Country-{{date}}.csv is the correct file path and naming convention.  

 

It is important to match case and to include {{date}} where the date is located. 

 

Click Show Content below to see which file would be uploaded 

Country-2022-07-26TO81417.csv

 


This topic has been closed for comments