Planet GEE Pipeline CLI¶
While moving between assets from Planet Inc and Google Earth Engine it was imperative to create a pipeline that allows for easy transitions between the two service end points and this tool is designed to act as a step by step process chain from Planet Assets to batch upload and modification within the Google Earth Engine environment. The ambition is apart from helping user with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as need arises. tab.
Table of contents¶
- Installation
- Getting started
- Usage examples
- Planet Tools
- Earth Engine Tools
- EE User
- Create
- Upload a directory with images and associate properties with each image:
- Upload a directory with images with specific NoData value to a selected destination:
- Asset List
- Asset Size
- Earth Engine Asset Report
- Task Query
- Task Report
- Delete a collection with content:
- Assets Move
- Assets Copy
- Assets Access
- Set Collection Property
- Cancel all tasks
- Credits
Installation¶
We assume Earth Engine Python API is installed and EE authorised as desribed here. We also assume Planet Python API is installed you can install by simply running.
pip install planet
This toolbox also uses some functionality from GDAL For installing GDAL in Ubuntu
sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install gdal-bin
To install Planet-GEE-Pipeline-CLI:
git clone https://github.com/samapriya/Planet-GEE-Pipeline-CLI.git
cd Planet-GEE-Pipeline-CLI && pip install -r requirements.txt
for linux use sudo pip install -r requirements.txt
ppipe -h
python setup.py develop or python setup.py install
In a linux distribution
sudo python setup.py develop or sudo python setup.py install
Getting started¶
As usual, to print help:
usage: ppipe.py [-h]
{
,planetkey,aoijson,activatepl,space,downloadpl,metadata,ee_user,create,upload,lst,collsize,delete,tasks,taskquery,report,cancel,mover,copy,access,collprop,cleanout}
...
Planet Pipeline with Google Earth Engine Batch Addons
positional arguments:
{ ,planetkey,aoijson,activatepl,space,downloadpl,metadata,ee_user,create,upload,lst,collsize,delete,tasks,taskquery,report,cancel,mover,copy,access,collprop,cleanout}
---------------------------------------
-----Choose from Planet Tools Below-----
---------------------------------------
planetkey Enter your planet API Key
aoijson Tool to convert KML, Shapefile,WKT,GeoJSON or Landsat
WRS PathRow file to AreaOfInterest.JSON file with
structured query for use with Planet API 1.0
activatepl Tool to query and/or activate Planet Assets
space Tool to query total download size of activated assets
& local space left for download
downloadpl Tool to download Planet Assets
metadata Tool to tabulate and convert all metadata files from
Planet or Digital Globe Assets
-------------------------------------------
----Choose from Earth Engine Tools Below----
-------------------------------------------
ee_user Get Earth Engine API Key & Paste it back to Command
line/shell to change user
create Allows the user to create an asset collection or
folder in Google Earth Engine
upload Batch Asset Uploader.
lst List assets in a folder/collection or write as text
file
collsize Collects collection size in Human Readable form &
Number of assets
delete Deletes collection and all items inside. Supports
Unix-like wildcards.
tasks Queries currently running, enqued,failed
taskquery Queries currently running, enqued,failed ingestions
and uploaded assets
report Create a report of all tasks and exports to a CSV file
cancel Cancel all running tasks
mover Moves all assets from one collection to another
copy Copies all assets from one collection to another:
Including copying from other users if you have read
permission to their assets
access Sets Permissions for Images, Collection or all assets
in EE Folder Example: python ee_permissions.py --mode
"folder" --asset "users/john/doe" --user
"jimmy@doe.com:R"
collprop Sets Overall Properties for Image Collection
cleanout Clear folders with datasets from earlier downloaded
optional arguments:
-h, --help show this help message and exit
To obtain help for a specific functionality, simply call it with help
switch, e.g.: ppipe upload -h. If you didn't install ppipe, then you
can run it just by going to ppipe directory and running python
ppipe.py [arguments go here]
Batch uploader¶
The script creates an Image Collection from GeoTIFFs in your local directory. By default, the collection name is the same as the local directory name; with optional parameter you can provide a different name. Another optional parameter is a path to a CSV file with metadata for images, which is covered in the next section: Parsing metadata.
usage: ppipe.py upload [-h] --source SOURCE --dest DEST [-m METADATA]
[-mf MANIFEST] [--large] [--nodata NODATA] [-u USER]
[-s SERVICE_ACCOUNT] [-k PRIVATE_KEY] [-b BUCKET]
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--source SOURCE Path to the directory with images for upload.
--dest DEST Destination. Full path for upload to Google Earth
Engine, e.g. users/pinkiepie/myponycollection
-u USER, --user USER Google account name (gmail address).
Optional named arguments:
-m METADATA, --metadata METADATA
Path to CSV with metadata.
-mf MANIFEST, --manifest MANIFEST
Manifest type to be used,for planetscope use
"planetscope"
--large (Advanced) Use multipart upload. Might help if upload
of large files is failing on some systems. Might cause
other issues.
--nodata NODATA The value to burn into the raster as NoData (missing
data)
-s SERVICE_ACCOUNT, --service-account SERVICE_ACCOUNT
Google Earth Engine service account.
-k PRIVATE_KEY, --private-key PRIVATE_KEY
Google Earth Engine private key file.
-b BUCKET, --bucket BUCKET
Google Cloud Storage bucket name.
Parsing metadata¶
By metadata we understand here the properties associated with each image. Thanks to these, GEE user can easily filter collection based on specified criteria. The file with metadata should be organised as follows:
| filename (without extension) | property1 header | property2 header |
|---|---|---|
| file1 | value1 | value2 |
| file2 | value3 | value4 |
Note that header can contain only letters, digits and underscores.
Example:
| id_no | class | category | binomial | system:time_start |
|---|---|---|---|---|
| my_file_1 | GASTROPODA | EN | Aaadonta constricta | 1478943081000 |
| my_file_2 | GASTROPODA | CR | Aaadonta irregularis | 1478943081000 |
The corresponding files are my_file_1.tif and my_file_2.tif. With each of the files five properties are associated: id_no, class, category, binomial and system:time_start. The latter is time in Unix epoch format, in milliseconds, as documented in GEE glosary. The program will match the file names from the upload directory with ones provided in the CSV and pass the metadata in JSON format:
{ id_no: my_file_1, class: GASTROPODA, category: EN, binomial: Aaadonta constricta, system:time_start: 1478943081000}
The program will report any illegal fields, it will also complain if not all of the images passed for upload have metadata associated. User can opt to ignore it, in which case some assets will have no properties.
Having metadata helps in organising your asstets, but is not mandatory - you can skip it.
Usage examples¶
Usage examples have been segmented into two parts focusing on both planet tools as well as earth engine tools, earth engine tools include additional developments in CLI which allows you to recursively interact with their python API
Planet Tools¶
The Planet Toolsets consists of tools required to access control and download planet labs assets (PlanetScope and RapidEye OrthoTiles) as well as parse metadata in a tabular form which maybe required by other applications.
Planet Key¶
This tool basically asks you to input your Planet API Key using a password prompt this is then used for all subsequent tools
usage: ppipe.py planetkey [-h]
optional arguments:
-h, --help show this help message and exit
If using on a private machine the Key is saved as a csv file for all future runs of the tool.
AOI JSON¶
The aoijson tab within the toolset allows you to create filters and structure your existing input file to that which can be used with Planet's API. The tool requires inputs with start and end date, along with cloud cover. You can choose from multiple input files types such as KML, Zipped Shapefile, GeoJSON, WKT or even Landsat Tiles based on PathRow numbers. The geo option asks you to select existing files which will be converted into formatted JSON file called aoi.json. If using WRS as an option just type in the 6 digit PathRow combination and it will create a json file for you.
usage: ppipe.py aoijson [-h] [--start START] [--end END] [--cloud CLOUD]
[--inputfile INPUTFILE] [--geo GEO] [--loc LOC]
optional arguments:
-h, --help show this help message and exit
--start START Start date in YYYY-MM-DD?
--end END End date in YYYY-MM-DD?
--cloud CLOUD Maximum Cloud Cover(0-1) representing 0-100
--inputfile INPUTFILE
Choose a kml/shapefile/geojson or WKT file for
AOI(KML/SHP/GJSON/WKT) or WRS (6 digit RowPath
Example: 023042)
--geo GEO map.geojson/aoi.kml/aoi.shp/aoi.wkt file
--loc LOC Location where aoi.json file is to be stored
Activate or Check Asset¶
The activatepl tab allows the users to either check or activate planet assets, in this case only PSOrthoTile and REOrthoTile are supported because I was only interested in these two asset types for my work but can be easily extended to other asset types. This tool makes use of an existing json file sturctured for use within Planet API or the aoi.json file created earlier
usage: ppipe.py activatepl [-h] [--aoi AOI] [--action ACTION] [--asst ASST]
optional arguments:
-h, --help show this help message and exit
--aoi AOI Choose aoi.json file created earlier
--action ACTION choose between check/activate
--asst ASST Choose between planet asset types (PSOrthoTile
analytic/REOrthoTile analytic/PSOrthoTile
analytic_xml/REOrthoTile analytic_xml
Check Total size of assets¶
It is important to sometimes estimate the overall size of download before you can actually download activated assets. This tool allows you to estimate local storage available at any location and overall size of download in MB or GB. This tool makes use of an existing url get request to look at content size and estimate overall download size of download for the activated assets.
usage: ppipe.py space [-h] [--aoi AOI] [--local LOCAL] [--asset ASSET]
optional arguments:
-h, --help show this help message and exit
--aoi AOI Choose aoi.json file created earlier
--local LOCAL local path where you are downloading assets
--asset ASSET Choose between planet asset types (PSOrthoTile
analytic/PSOrthoTile analytic_dn/PSOrthoTile
visual/PSScene4Band analytic/PSScene4Band
analytic_dn/PSScene3Band analytic/PSScene3Band
analytic_dn/PSScene3Band visual/REOrthoTile
analytic/REOrthoTile visual
Download Asset¶
Having metadata helps in organising your asstets, but is not mandatory - you can skip it. The downloadpl tab allows the users to download assets. The platform can download Asset or Asset_XML which is the metadata file to desired folders.One again I was only interested in these two asset types(PSOrthoTile and REOrthoTile) for my work but can be easily extended to other asset types.
usage: ppipe.py downloadpl [-h] [--aoi AOI] [--action ACTION] [--asst ASST]
[--pathway PATHWAY]
optional arguments:
-h, --help show this help message and exit
--aoi AOI Choose aoi.json file created earlier
--action ACTION choose download
--asst ASST Choose between planet asset types (PSOrthoTile
analytic/REOrthoTile analytic/PSOrthoTile
analytic_xml/REOrthoTile analytic_xml
--pathway PATHWAY Folder Pathways where PlanetAssets are saved exampled
./PlanetScope ./RapidEye
Metadata Parser¶
The metadata tab is a more powerful tool and consists of metadata parsing for All PlanetScope and RapiEye Assets along with Digital Globe MultiSpectral and DigitalGlobe PanChromatic datasets. This was developed as a standalone to process xml metadata files from multiple sources and is important step is the user plans to upload these assets to Google Earth Engine.
usage: ppipe.py metadata [-h] [--asset ASSET] [--mf MF] [--mfile MFILE]
[--errorlog ERRORLOG]
optional arguments:
-h, --help show this help message and exit
--asset ASSET Choose PS OrthoTile(PSO)|PS OrthoTile DN(PSO_DN)|PS
OrthoTile Visual(PSO_V)|PS4Band Analytic(PS4B)|PS4Band
DN(PS4B_DN)|PS3Band Analytic(PS3B)|PS3Band
DN(PS3B_DN)|PS3Band Visual(PS3B_V)|RE OrthoTile
(REO)|RE OrthoTile Visual(REO_V)|DigitalGlobe
MultiSpectral(DGMS)|DigitalGlobe Panchromatic(DGP)?
--mf MF Metadata folder?
--mfile MFILE Metadata filename to be exported along with Path.csv
--errorlog ERRORLOG Errorlog to be exported along with Path.csv
Earth Engine Tools¶
The ambition is apart from helping user with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as need arises. This is also a seperate package for earth engine users to use and can be downloaded here
EE User¶
This tool is designed to allow different users to change earth engine authentication credentials. The tool invokes the authentication call and copies the authentication key verification website to the clipboard which can then be pasted onto a browser and the generated key can be pasted back
Create¶
This tool allows you to create a collection or folder in your earth engine root directory. The tool uses the system cli to achieve this and this has been included so as to reduce the need to switch between multiple tools and CLI.
usage: ppipe.py create [-h] --typ TYP --path PATH
optional arguments:
-h, --help show this help message and exit
--typ TYP Specify type: collection or folder
--path PATH This is the path for the earth engine asset to be created full
path is needsed eg: users/johndoe/collection
Upload a directory with images to your myfolder/mycollection and associate properties with each image:¶
ppipe upload -u johndoe@gmail.com --source path_to_directory_with_tif -m path_to_metadata.csv -mf maifest_type(ex:planetscope) --dest users/johndoe/myfolder/myponycollection
Upload a directory with images with specific NoData value to a selected destination¶
ppipe upload -u johndoe@gmail.com --source path_to_directory_with_tif --dest users/johndoe/myfolder/myponycollection --nodata 222
Asset List¶
This tool is designed to either print or output asset lists within folders or collections using earthengine ls tool functions.
usage: geeadd.py lst [-h] --location LOCATION --typ TYP [--items ITEMS]
[--output OUTPUT]
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--location LOCATION This it the location of your folder/collection
--typ TYP Whether you want the list to be printed or output as
text[print/report]
Optional named arguments:
--items ITEMS Number of items to list
--output OUTPUT Folder location for report to be exported
Asset Size¶
This tool allows you to query the size of any Earth Engine asset[Images, Image Collections, Tables and Folders] and prints out the number of assets and total asset size in non-byte encoding meaning KB, MB, GB, TB depending on size.
usage: geeadd assetsize [-h] --asset ASSET
optional arguments:
-h, --help show this help message and exit
--asset ASSET Earth Engine Asset for which to get size properties
Earth Engine Asset Report¶
This tool recursively goes through all your assets(Includes Images, ImageCollection,Table,) and generates a report containing the following fields [Type,Asset Type, Path,Number of Assets,size(MB),unit,owner,readers,writers].
usage: geeadd.py ee_report [-h] --outfile OUTFILE
optional arguments:
-h, --help show this help message and exit
--outfile OUTFILE This it the location of your report csv file
geeadd --outfile "C:\johndoe\report.csv"
Task Query¶
This script counts all currently running and ready tasks along with failed tasks.
usage: geeadd.py tasks [-h]
optional arguments:
-h, --help show this help message and exit
geeadd.py tasks
Task Report¶
Sometimes it is important to generate a report based on all tasks that is running or has finished. Generated report includes taskId, data time, task status and type
usage: geeadd taskreport [-h] [--r R]
optional arguments:
-h, --help show this help message and exit
--r R Folder Path where the reports will be saved
Delete a collection with content:¶
The delete is recursive, meaning it will delete also all children assets: images, collections and folders. Use with caution!
geeadd delete users/johndoe/test
Console output:
2016-07-17 16:14:09,212 :: oauth2client.client :: INFO :: Attempting refresh to obtain initial access_token
2016-07-17 16:14:09,213 :: oauth2client.client :: INFO :: Refreshing access_token
2016-07-17 16:14:10,842 :: root :: INFO :: Attempting to delete collection test
2016-07-17 16:14:16,898 :: root :: INFO :: Collection users/johndoe/test removed
Delete all directories / collections based on a Unix-like pattern¶
geeadd delete users/johndoe/*weird[0-9]?name*
Assets Move¶
This script allows us to recursively move assets from one collection to the other.
usage: geeadd.py mover [-h] [--assetpath ASSETPATH] [--finalpath FINALPATH]
optional arguments:
-h, --help show this help message and exit
--assetpath ASSETPATH
Existing path of assets
--finalpath FINALPATH
New path for assets
geeadd.py mover --assetpath "users/johndoe/myfolder/myponycollection" --destination "users/johndoe/myfolder/myotherponycollection"
Assets Copy¶
This script allows us to recursively copy assets from one collection to the other. If you have read acess to assets from another user this will also allow you to copy assets from their collections.
usage: geeadd.py copy [-h] [--initial INITIAL] [--final FINAL]
optional arguments:
-h, --help show this help message and exit
--initial INITIAL Existing path of assets
--final FINAL New path for assets
geeadd.py mover --initial "users/johndoe/myfolder/myponycollection" --final "users/johndoe/myfolder/myotherponycollection"
Assets Access¶
This tool allows you to set asset acess for either folder , collection or image recursively meaning you can add collection access properties for multiple assets at the same time.
usage: geeadd access [-h] --mode MODE --asset ASSET --user USER
optional arguments:
-h, --help show this help message and exit
--mode MODE This lets you select if you want to change permission or
folder/collection/image
--asset ASSET This is the path to the earth engine asset whose permission
you are changing folder/collection/image
--user USER This is the email address to whom you want to give read or
write permission Usage: "john@doe.com:R" or "john@doe.com:W"
R/W refers to read or write permission
geeadd.py access --mode folder --asset "folder/collection/image" --user "john@doe.com:R"
Set Collection Property¶
This script is derived from the ee tool to set collection properties and will set overall properties for collection.
usage: geeadd.py collprop [-h] [--coll COLL] [--p P]
optional arguments:
-h, --help show this help message and exit
--coll COLL Path of Image Collection
--p P "system:description=Description"/"system:provider_url=url"/"sys
tem:tags=tags"/"system:title=title
Cancel all tasks¶
This is a simpler tool, can be called directly from the earthengine cli as well
earthengine cli command
earthengine task cancel all
usage: geeadd.py cancel [-h]
optional arguments:
-h, --help show this help message and exit
Credits¶
JetStream A portion of the work is suported by JetStream Grant TG-GEO160014.
Also supported by Planet Labs Ambassador Program
Original upload function adapted from Lukasz's asset manager tool
Changelog¶
v0.3.0¶
- Allows for quiet authentication for use in Google Colab or non interactive environments
- Improved planet key entry and authentication protocols
v0.2.91¶
- Fixed issue with Surface Reflectance metadata and manifest lib
- Improved ingestion support for (PSScene4Band analytic_Sr)[PS4B_SR]
v0.2.9¶
- Fixed issues with generating id list
- Improved overall security of command calls
v0.2.2¶
- Major improvements to ingestion using manifest ingest in Google Earth Engine
- Contains manifest for all commonly used Planet Data item and asset combinations
- Added additional tool to Earth Engine Enhancement including quota check before upload to GEE
v0.2.1¶
- Fixed initialization loop issue
v0.2.0¶
- Metadata parser and Uploader Can now handle PlanetScope 4 Band Surface Reflectance Datasets
- General Improvements
v0.1.9¶
- Changes made to reflect updated GEE Addon tools
- general improvements
v0.1.8¶
- Minor fixes to parser and general improvements
- Planet Key is now stored in a configuration folder which is safer "C:\users.config\planet"
- Earth Engine now requires you to assign a field type for metadata meaning an alphanumeric column like satID cannot also have numeric values unless specified explicitly . Manifest option has been added to handle this (just use -mf "planetscope")
- Added capability to query download size and local disk capacity before downloading planet assets.
- Added the list function to generate list of collections or folders including reports
- Added the collection size tool which allows you to estimate total size or quota used from your allocated quota.
- ogr2ft feature is removed since Earth Engine now allows vector and table uploading.