You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is an alphabetical list of contributors, if you feel you have contributed to the project and we have neglected adding you to the list please accept our apologizes and [let us know](mailto:[email protected]) to correct it.
When contributing to this repository, please first discuss the change you wish to make via [issues](https://github.com/mgckind/easyaccess/issues),
5
+
[email](mailto:[email protected]), or any other method with the owners of this repository before making a change.
6
+
7
+
Please note we have a [code of conduct](CODE_OF_CONDUCT.md) for this project, please follow it in all your interactions with the project.
8
+
9
+
## Reporting Bugs
10
+
11
+
When creating a bug or an issue please include as many details as possible. Create your issues [here](https://github.com/mgckind/easyaccess/issues) and follow this steps:
12
+
13
+
- Check [previous closed issues](https://github.com/mgckind/easyaccess/issues?q=is%3Aissue+is%3Aclosed) , [current open issues](https://github.com/mgckind/easyaccess/issues?q=is%3Aopen+is%3Aissue) or our FAQ for a list of common questions or previously solved problems
14
+
- Use a clear and descriptive title for the issue to identify the problem.
15
+
- Provide all information about OS
16
+
- Provide information about the installation process (source, pip, conda, Docker, etc)
17
+
- software version including Python
18
+
- Detailed information on how to reproduce the problem
19
+
- Provide screenshots when useful
20
+
21
+
## Other contributions
22
+
23
+
All kind of contributions are welcome, from fixing bugs, resolving issues or suggesting features. Look at the [current issues](https://github.com/mgckind/easyaccess/issues) to see how you can contribute and provide help.
24
+
25
+
## Pull Request Process
26
+
27
+
1. Ensure any install or build dependencies are removed before the end of the layer when doing a
28
+
build.
29
+
2. Update the README.md with details of changes to the interface, this includes new environment
30
+
variables, exposed ports, useful file locations and container parameters.
31
+
3. Increase the version numbers in any examples files and the README.md to the new version that this
32
+
Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/).
33
+
4. You may merge the Pull Request in once you have the sign-off of two other developers, or if you
34
+
do not have permission to do that, you may request the second reviewer to merge it for you.
35
+
36
+
## Authors
37
+
38
+
Please see our [list of contributors](AUTHORS.md), if you feel you have contributed to the project and we have neglected adding you to the list please accept our apologizes and [let us know](mailto:[email protected]) to correct it.
Here we collect a list of FAQ related to installation, usage, etc. Please refre to the [issues page](https://github.com/mgckind/easyaccess/issues) for more information.
4
+
5
+
- The installation with `pip` went smoothly but it was not obvious where "easyaccess" was installed.
6
+
- When installed using pip, one can use the following: `pip show -f easyaccess` to check the installation folders
7
+
8
+
- When trying to connect I keep getting this error `ORA-21561: OID generation failed`. Any idea how to solve it?
9
+
- Most of the time this problem can be solved by adding the name of your computer in the `/etc/hosts` file, next to the line that says 127.0.0.1 localhost. Just add the name of your computer (type `hostname`) to that line, so it looks like `127.0.0.1 localhost my-computer`
10
+
- Where is the configuration file and authentication file?
11
+
- Usually the configuration file by default is at `$HOME/.easyaccess/config.ini` and the authentication file is at `$HOME/.desservices.ini` or can be set at the env variable `$DES_SERVICES`
12
+
- I am a DES Collaborator, where can I reset my credentials?
13
+
- For collaborators only: Please use [this](https://deslogin.wufoo.com/forms/help-me-with-my-desdm-account/) form
14
+
- The client hangs after getting an Oracle error and I need to close to the window.
15
+
- This is a long-standing [issue](https://github.com/mgckind/easyaccess/issues/130) but we haven't been able to fix it or to reproduce it in all the systems. Please report it there and add details about OS and versions. We'd recommend a clean conda installation which sometimes fixes the issue.
16
+
- How can I install the Oracle client by myself.
17
+
- Please follow the instructions posted [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html)
18
+
- Are there other requirements besides python modules and Oracle clients?
19
+
- Usually not, for new OS or Virtual Machines you'd need to install `libaio` and `libbz2`
20
+
- How can I contribute to the project?
21
+
- Please take a look st our [Code of Conduct](CODE_OF_CONDUCT.md) and or [contribution guide](CONTRIBUTING.md)
Enhanced command line SQL interpreter client for astronomical databases.
3
+
Enhanced command line SQL interpreter client for astronomical surveys.
4
+
5
5
6
-
Python Command Line Interpreter to access Oracle DES DB
7
-
using cx_Oracle
6
+
## Description
7
+
`easyaccess` is an enhanced command line interpreter and Python package created to facilitate access to astronomical catalogs stored in SQL Databases. It provides a custom interface with custom commands and was specifically designed to access data from the Dark Energy Survey Oracle database, including autocompletion of tables, columns, users and commands, simple ways to upload and download tables using csv, fits and HDF5 formats, iterators, search and description of tables among others. It can easily be extended to another surveys or SQL databases. The package was completely written in Python and support customized addition of commands and functionalities.
8
8
9
9
For a short tutorial check [here](http://matias-ck.com/easyaccess)
10
10
11
-
**Current version = 1.4.5**
11
+
**Current version = 1.4.6**
12
12
13
+
#### DES DR1 users
13
14
For DES public data release, you can start `easyaccess` with:
14
15
15
16
easyaccess -s desdr
@@ -18,24 +19,89 @@ To create an account click [here](https://des.ncsa.illinois.edu/easyweb/signup/)
18
19
19
20
## Requirements
20
21
21
-
- Oracle Client > 11g.2 (External library, no python)
22
-
Check [here](https://opensource.ncsa.illinois.edu/confluence/display/DESDM/Instructions+for+installing+Oracle+client+and+easyaccess+without+EUPS) for instructions on how to install these libraries
Note that cx_Oracle needs libaio on some Linux systems (e.g., #98)
26
-
27
-
Note that cx_Oracle needs libbz2 on some Linux systems
22
+
-[Oracle Client](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html) > 11g.2 (External library, no python)
23
+
Check [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html) for instructions on how to install these libraries
-[gnureadline](https://github.com/ludwigschwardt/python-gnureadline) (optional, for better console behavior in OS X)
35
-
- importlib (This is only needed if running python 2.6)
36
35
36
+
## Installation
37
+
38
+
Installing `easyaccess` can be a little bit tricky given the external libraries required, in particular the Oracle libraries which are free to use. `easyaccess` is based heavily on the Oracle python client `cx_Oracle`, you can follow the installation instructions from [here](https://cx-oracle.readthedocs.io/en/latest/installation.html#quick-start-cx-oracle-installation). For `cx_Oracle` to work, you will need the Oracle Instant Client packages which can be obtained from [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html).
39
+
40
+
Make sure you have these libraries installed before proceeding to the installation of easyaccess, you can try by opening a Python interpreter and type:
41
+
42
+
import cx_Oracle
43
+
44
+
If you have issues, please check the [Troubleshooting page](https://cx-oracle.readthedocs.io/en/latest/installation.html#troubleshooting) or our [FAQ page](FAQ.md).
45
+
46
+
#### Source Installation
47
+
48
+
You can clone this repository and install `easyaccess` with:
49
+
50
+
python setup.py install
51
+
52
+
#### Pip installation
53
+
`easyaccess` can also be installed using `pip` but it'd require the installation of the oracle instant client first as described above
For collaborators, We have a Docker image with easyaccess pre-installed which you can obtained from:
68
+
69
+
docker pull mgckind/easyaccess
70
+
71
+
## FAQ
72
+
We have a running list of [FAQ](FAQ.md) which we will constantly update, please check [here](FAQ.md).
73
+
74
+
#### Contributing
75
+
Please take a look st our [Code of Conduct](CODE_OF_CONDUCT.md) and or [contribution guide](CONTRIBUTING.md).
76
+
77
+
78
+
## Citation
79
+
If you use `easyaccess` in your work we would encourage to use this reference [https://arxiv.org/abs/1810.02721](https://arxiv.org/abs/1810.02721) or copy/paste this BibTeX:
80
+
```
81
+
@ARTICLE{2018arXiv181002721C,
82
+
author = {{Carrasco Kind}, M. and {Drlica-Wagner}, A. and {Koziol}, A.~M.~G. and
83
+
{Petravick}, D.},
84
+
title = "{easyaccess: Enhanced SQL command line interpreter for astronomical surveys}",
85
+
journal = {arXiv e-prints},
86
+
keywords = {Astrophysics - Instrumentation and Methods for Astrophysics},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
96
+
}
97
+
```
98
+
99
+
100
+
## Usage
101
+
102
+
For a short tutorial and documentation see [here](http://matias-ck.com/easyaccess), note that not all the features are available for the public use, i.e., DR1 users.
103
+
104
+
#### Some *great* features
39
105
- Nice output format (using pandas)
40
106
- Very flexible configuration
41
107
- Smart tab autocompletion for commands, table names, column names, and file paths
@@ -48,23 +114,14 @@ To create an account click [here](https://des.ncsa.illinois.edu/easyweb/signup/)
48
114
- Show the execution plan of a query if needed
49
115
- Python functions can be run in a inline query
50
116
51
-
## Conda installation
52
-
Now easyaccess can be installed using [conda](http://conda.pydata.org/docs/install/quick.html) out of the box!
53
-
54
-
conda install easyaccess==1.4.5 -c mgckind
55
-
56
-
## Pip installation
57
-
easyaccess can also be installed using `pip` but it'd require the installation of the oracle instant client first
58
117
59
-
pip install easyaccess==1.4.5
60
-
61
-
## Interactive interpreter
118
+
#### Interactive interpreter
62
119
63
120
Assuming that ```easyaccess``` is in your path, you can enter the interactive interpreter by calling ```easyaccess``` without any command line arguments:
64
121
65
122
easyaccess
66
123
67
-
### Running SQL commands
124
+
####Running SQL commands
68
125
Once inside the interpreter run SQL queries by adding a ";" at the end::
69
126
70
127
DESDB ~> select ... from ... where ... ;
@@ -75,15 +132,15 @@ To save the results into a table add ">" after the end of the query (after ";")
75
132
76
133
The file types supported so far are: .csv, .tab, .fits, and .h5. Any other extension is ignored.
77
134
78
-
### Load tables
135
+
####Load tables
79
136
To load a table it needs to be in a csv format with columns names in the first row
80
137
the name of the table is taken from filename or with optional argument --tablename
81
138
82
139
DESDB ~> load_table <filename> --tablename <mytable> --chunksize <number of rows to read/upload> --memsize <memory in MB to read at a time>
83
140
84
141
The --chunsize and --memsize are optional arguments to facilitate uploading big files.
85
142
86
-
### Load SQL queries
143
+
####Load SQL queries
87
144
To load SQL queries just run:
88
145
89
146
DESDB ~> loadsql <filename.sql>
@@ -93,7 +150,7 @@ or
93
150
94
151
The query format is the same as the interpreter, SQL statement must end with ";" and to write output files the query must be followed by " > <outputfile>"
95
152
96
-
### Configuration
153
+
####Configuration
97
154
98
155
The configuration file is located at ```$HOME/.easyaccess/config.ini``` but everything can be configured from inside easyaccess type:
99
156
@@ -111,8 +168,14 @@ and to see any particular option (e.g., timeout):
111
168
112
169
DESDB ~> config timeout show
113
170
114
-
## Command line usage
171
+
####Command line usage
115
172
116
173
Much of the functionality provided through the interpreter is also available directly from the command line. To see a list of command-line options, use the ```--help``` option
117
174
118
175
easyaccess --help
176
+
177
+
## Architecture
178
+
179
+
We have included a simplified UML diagram describing the architecture and dependencies of `easyaccess` which shows only the different methods for a given class and the name of the file hosting a given class. The main class, `easy_or()`, inherits all methods from all different subclasses, making this model flexible and extendable to other surveys or databases. These methods are then converted to command line commands and functions that can be called inside `easyaccess`. Given that there are some DES specific functions, we have moved DES methods into a separate class `DesActions()`.
0 commit comments