Merge pull request #8 from geospatial-jeff/dev

Lambda Layers

Author: geospatial-jeff

Dockerfile

@@ -0,0 +1,32 @@
+FROM lambci/lambda:build-python3.6
+
+
+# Installing system libraries
+RUN \
+    yum install -y wget; \
+    yum install -y geos-devel; \
+    yum clean all; \
+    yum autoremove;
+
+
+# Paths
+ENV \
+    PREFIX=/usr/local \
+    LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64
+
+# Switch to build directory
+WORKDIR /build
+
+# Installing cognition-datasources + requirements
+COPY requirements-dev.txt ./
+
+RUN \
+    pip install -r requirements-dev.txt; \
+    pip install git+https://github.com/geospatial-jeff/cognition-datasources.git
+
+
+
+# Copy shell scripts
+COPY bin/* /usr/local/bin/
+
+WORKDIR /home/cognition-datasources

MANIFEST.in

@@ -1,61 +1,47 @@
-# cognition-datasources
-
 ## About
-This library defines a STAC-compliant standardized interface for searching geospatial assets, primarily remotely sensed imagery.  The [Spatio-Temporal-Asset-Catalog (STAC)](https://github.com/radiantearth/stac-spec) specification provides common metadata and API schemas to search and access geospatial data.  The standardized interface used by the library is based on the STAC spec and allows searching across three dimensions:
-
-- **Spatial:** Find all assets which intersect a bounding box.
-- **Temporal:** Find all assets acquired within a temporal window.
-- **Properties:** Find all assets with certain  metadata.
 
-Not all commonly used datasources are currently STAC-compliant.  In such cases, the library maintains a standardized search interface by wrapping the API with a STAC-compatible API which parses the initial search parameters into a format compatible with the underlying API.  A request to the API is sent and the response is parsed into a STAC Item and returned to the user.  The table below of supported datasources states which are STAC-compliant.
+This library defines a pluggable, STAC-compliant, service for searching geospatial assets, primarily remotely sensed imagery, and serves two primary purposes:
 
-![title](docs/images/api-diagram.png)
+1. Define a pluggable driver interface (similar to GraphQL resolvers) for wrapping the STAC spec around legacy datasources.
+2. Provide a framework for loading / executing drivers both locally and in the cloud.
 
-#### Datasource Drivers
-The interface defined by the library is extended by datasource drivers which are defined in external github repositories and loaded into the library through a command line interface.  Similar to how drivers control hardware, the logic implemented in the datasource driver influences how cognition-datasources accesses the underlying datasource.  Each driver is expected to follow a specific pattern and pass a standard set of test cases (enforced with CircleCI).  Check out the [contribution guidelines](/docs/contributing.md) for a guide on how to develop your own datasource driver!
+Each driver translates the STAC-compliant request into a format compatible with the underlying API while translating the API response to a valid STAC Item.  Drivers are packaged and deployed to AWS Lambda and a single API Gateway endpoint is created which allows searching the loaded datasources.  The goal is to create an extensible service which allows users to integrate their datasets with the STAC ecosystem without having to change how their data is stored and queried.
 
+![title](docs/images/service-diagram.png?style=centerme)
 
-## Setup
+## Installation
 ```
-# Install library
-pip install git+https://github.com/geospatial-jeff/cognition-datasources
-
-# Load datasources
-cognition-datasources load -d Landsat8 -d Sentinel2
+git clone https://github.com/geospatial-jeff/cognition-datasources
+cd cognition-datasources
+python setup.py develop
 ```
 
-## Usage
-
-#### Python
-```python
-from datasources import Manifest
-
-# Create manifest
-manifest = Manifest()
+## Deployment
+```
+# Load datasources
+cognition-datasources -d Landsat8 -d Sentinel2 -d SRTM -d NAIP
 
-# Search arguments
-spatial = {"type": "Polygon", "coordinates": [[...]]}
-temporal = ("2018-10-30", "2018-12-31")
-properties = {'eo:cloud_cover': {'lt': 10}}
+# Build docker container
+docker build . -t cognition-datasources:latest
 
-# Create searches for Landsat8 and Sentinel2
-manifest['Landsat8'].search(spatial, temporal=temporal, properties=properties)
-manifest['Sentinel2'].search(spatial, temporal=temporal, properties=properties)
+# Package service
+docker run --rm -v $PWD:/home/cognition-datasources -it cognition-datasources:latest package-service.sh
 
-# Execute searches
-response = manifest.execute()
+# Deploy to AWS
+sls deploy -v
 ```
+Read the [deployment docs](./docs/deployment.md) for more information on deployment.
 
-#### CLI
-```
-cognition-datasources search xmin ymin xmax ymax --start-date "2018-10-30" --end-date "2018-12-31" -d Landsat8 -d Sentinel2 --output response.json
-```
+## Usage
+The deployment generates an AWS API Gateway endpoint which supports STAC-compliant searches of the loaded datasources through the `/stac/search` endpoint (POST).  Read the [API docs](./docs/README.md) for usage details.
+
+A live example lives [here](https://github.com/geospatial-jeff/cognition-datasources-api).
 
 ## Testing
-Each driver must pass a [standard set of test cases](datasources/tests.py) and uses CircleCI to ensure only working drivers are loaded into the library.  View the status of each driver [here](/docs/datasource-status.md).
+Each driver must pass a [standard set of test cases](./datasources/tests.py) and uses CircleCI to ensure only working drivers are loaded into the library.  View the status of each driver [here](./docs/datasource-status.md).
 
-## Documentation
-Read the [quickstart](./docs/quickstart.ipynb) and [documentation](./docs).
+## Contributing
+Check out the [contributing docs](./docs/contributing.md) for step-by-step guide for building your own driver.
 
 ## Supported Datasource Drivers
 | Name | Source | STAC-Compliant | Notes |
@@ -69,4 +55,4 @@ Read the [quickstart](./docs/quickstart.ipynb) and [documentation](./docs).
 | [Sentinel2](https://github.com/geospatial-jeff/cognition-datasources-sentinel2) | [AWS Earth: Sentinel2](https://registry.opendata.aws/sentinel-2/) | True | Sends requests to [sat-api](https://github.com/sat-utils/sat-api). |
 | [SRTM](https://github.com/geospatial-jeff/cognition-datasources-srtm) | [AWS: Terrain Tiles](https://registry.opendata.aws/terrain-tiles/) | False | Does not send any requests. |
 | [USGS 3DEP](https://github.com/geospatial-jeff/cognition-datasources-usgs3dep) | [AWS: USGS 3DEP](https://registry.opendata.aws/usgs-lidar/) | False | Sends request to AWS S3 Bucket. |
-| [Microsoft Building Footprints](https://github.com/geospatial-jeff/cognition-datasources-mbf) | [Microsoft](https://github.com/Microsoft/USBuildingFootprints) / [ESRI](https://www.arcgis.com/home/item.html?id=f40326b0dea54330ae39584012807126) | False | Sends requests to ESRI Feature Layer |
+| [Microsoft Building Footprints](https://github.com/geospatial-jeff/cognition-datasources-mbf) | [Microsoft](https://github.com/Microsoft/USBuildingFootprints) / [ESRI](https://www.arcgis.com/home/item.html?id=f40326b0dea54330ae39584012807126) | False | Sends requests to ESRI Feature Layer |

README.md

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+# Directory used for deployment
+export DEPLOY_DIR=layer
+
+mkdir $DEPLOY_DIR
+
+PYPATH=/var/lang/lib/python3.6/site-packages
+
+
+echo Creating deployment package for cognition-datasources
+
+# Moving python libraries
+mkdir $DEPLOY_DIR/python
+EXCLUDE="shapely* stac_validator* s3transfer* boto3* botocore* pip* docutils* *.pyc setuptools* wheel* coverage* testfixtures* mock* *.egg-info *.dist-info __pycache__ easy_install.py"
+
+EXCLUDES=()
+for E in ${EXCLUDE}
+do
+    EXCLUDES+=("--exclude ${E} ")
+done
+
+rsync -ax $PYPATH/ $DEPLOY_DIR/python/ ${EXCLUDES[@]}
+
+cd $DEPLOY_DIR
+zip -ruq ../lambda-layer.zip ./

bin/package-layer.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# Directory used for deployment
+export DEPLOY_DIR=service
+
+mkdir $DEPLOY_DIR
+
+# Moving handler
+cp handler.py $DEPLOY_DIR
+
+cd $DEPLOY_DIR
+zip -ruq ../lambda-service-package.zip ./

bin/package-service.sh

@@ -2,3 +2,4 @@
 from .manifest import Manifest
 
 
+layer_arn = 'arn:aws:lambda:us-east-1:725820063953:layer:cognition-datasources:6'