housing-data-ingest

Pipeline for ingesting data about housing near RIT.

Contributing

How to

1. Configure your environment (instructions on the wiki).
2. Choose an unassigned issue, and comment that you're working on it.
3. Open a PR containing a new fetch, parse, or normalize script! (details on these stages)

Results are periodically committed to vaccine-feed-ingest-results. Once your PR is merged, you will be able to see the output of your work there!

Run the tool

See the wiki for instructions on how to run housing-data-ingest.

Production Details

For more information on (pipeline stages) and how to contribute, see the wiki!

The below details on interacting with our production environment are intended for staff developers.

Overall setup

In production, all stages for all runners are run, and outputs are stored to the vaccine-feeds bucket on GCS.

If you are developing a feature that interacts with the remote storage, you need to test GCS then install the gcloud SDK from setup instructions and use the vaccine-feeds-dev bucket (you will need to be granted access).

Results are also periodically committed to vaccine-feed-ingest-results.

Instructions

1. Authenticate to gcloud with an account that has access to vaccine-feeds-dev bucket.

gcloud auth application-default login

1. Run ingestion with an GCS --output-dir

poetry run housing-data-ingest all-stages --output-dir=gs://vaccine-feeds-dev/locations/

Load Source Locations

VIAL Setup

1. Request an account on the VIAL staging server https://vial-staging.calltheshots.us

2. Create an API Key for yourself at https://vial-staging.calltheshots.us/admin/api/apikey/

3. Store the API key in project .env file with the var VIAL_APIKEY

Load Usage

• Load SF.GOV source feed to VIAL

  poetry run housing-data-ingest load-to-vial ca/sf_gov