With some exceptions (e.g. see the section on redirection) The libcurl authorization mechanisms can be accessed in two ways
http://username:password@host/...This username and password will be used if the server asks for authentication. Note that only simple password authentication is supported in this format. Specifically note that redirection based authorization will not work with this because the username and password will only be used on the initial request, not the redirection
The file must be called one of the following names: ".daprc" or ".dodsrc" If both .daprc and .dodsrc exist, then the .daprc file will take precedence.
The rc file is searched for first in the current directory
and then in the home directory (as defined by the HOME environment
variable).
The rc file format is a series of lines of the general form:
The currently defined set of authorization-related keys are as follows.
The second column is the affected curl_easy_setopt option(s), if any.
HTTP.SSL.KEY is essentially the same as HTTP.SSL.CERTIFICATE
and should always have the same value.
HTTP.SSL.KEYPASSWORD
specifies the password for accessing the HTTP.SSL.CERTIFICAT/HTTP.SSL.key file.
HTTP.SSL.CAPATH
specifies the path to a directory containing
trusted certificates for validating server certificates.
HTTP.SSL.VALIDATE
is a boolean (1/0) value that if true (1)
specifies that the client should verify the server's presented certificate.
HTTP.PROXY_SERVER
specifies the url for accessing the proxy:
e.g. http://[username:password@]host[:port]
HTTP.NETRC
specifies the absolute path of the .netrc file.
See redirection authorization
for information about using .netrc.
The process is usually as follows.
It turns out that libcurl uses the password in the .daprc file — or from the url —
only for the initial connection. This causes problems because
the redirected connection is the one that actually requires the password.
This is where .netrc comes in. Libcurl will use .netrc for
the redirected connection. It is possible to cause libcurl to use
the .daprc password always, but this introduces a security hole
because it may send the initial user+pwd to the redirection site.
In summary, if you are using redirection, then you must create a .netrc
file to hold the password for the site to which the redirection is sent.
The format of this .netrc file will contain content that
typically look like this.
One final note. In using this, it is almost certain that you will
need to specify a real cookie jar file (HTTP.COOKIEJAR) so that the
redirect site can pass back authorization information.
Examples.
For example, the URL
Similarly,
In order to access ESG datasets, however, it is necessary to
register as a user with ESG and to setup your environment
so that proper authentication is established between an zz
client program and the ESG data server. Specifically, it
is necessary to use what is called "client-side keys" to
enable this authentication. Normally, when a client accesses
a server in a secure fashion (using "https"), the server
provides an authentication certificate to the client.
With client-side keys, the client must also provide a
certificate to the server so that the server can know with
whom it is communicating.
The zz library uses the curl library and it is that
underlying library that must be properly configured.
As noted, also uses re-direction based authentication.
So, when it receives an initial connection from a client, it
redirects to a separate authentication server. When that
server has authenticated the client, it redirects back to
the original url to complete the request.
[<host:port>]<key>=<value>
where the bracket-enclosed host:port is optional and will be discussed
subsequently.
Key Affected curl_easy_setopt Options Notes
HTTP.COOKIEJAR CURLOPT_COOKIEJAR
HTTP.COOKIEFILE CURLOPT_COOKIEJAR Alias for CURLOPT_COOKIEJAR
HTTP.PROXY_SERVER CURLOPT_PROXY, CURLOPT_PROXYPORT, CURLOPT_PROXYUSERPWD
HTTP.SSL.CERTIFICATE CURLOPT_SSLCERT
HTTP.SSL.KEY CURLOPT_SSLKEY
HTTP.SSL.KEYPASSWORD CURLOPT_KEYPASSWORD
HTTP.SSL.CAINFO CURLOPT_SSLCAINFO
HTTP.SSL.CAPATH CURLOPT_SSLCAPATH
HTTP.SSL.VERIFYPEER CURLOPT_SSL_VERIFYPEER
HTTP.SSL.VALIDATE CURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST
HTTP.CREDENTIALS.USERPASSWORD CURLOPT_USERPASSWORD
HTTP.NETRC N.A. Specify path of the .netrc file
Password Authentication
The key
HTTP.CREDENTIALS.USERPASSWORD
can be used to set the simple password authentication.
This is an alternative to setting it in the url.
The value must be of the form "username:password".
See redirection authorization
for important additional information.
Cookie Jar
The HTTP.COOKIEJAR key
specifies the name of file from which
to read cookies (CURLOPT_COOKIEJAR) and also
the file into which to store cookies (CURLOPT_COOKIEFILE).
The same value is used for both CURLOPT values.
It defaults to in-memory storage.
See redirection authorization
for important additional information.
Certificate Authentication
HTTP.SSL.CERTIFICATE
specifies a file path for a file containing a PEM cerficate.
This is typically used for client-side authentication.
Redirection-Based Authentication
Some sites provide authentication by using a third party site
to do the authentication. Examples include ESG and URS.
machine mmmmmm login xxxxxx password yyyyyy
where the machine, mmmmmm, is the hostname of the machine to
which the client is redirected for authorization, and the
login and password are those needed to authenticate on that machine.
URL Constrained RC File Entries
Each line of the rc file can begin with
a host+port enclosed in square brackets.
The form is "host:port".
If the port is not specified
then the form is just "host".
The reason that more of the url is not used is that
libcurl's authorization grain is not any finer than host level.
[remotetest.unidata.ucar.edu]HTTP.VERBOSE=1
or
[fake.ucar.edu:9090]HTTP.VERBOSE=0
If the url request from, say, the zz_open method
has a host+port matching one of the prefixes in the rc file, then
the corresponding entry will be used, otherwise ignored.
http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc
will have HTTP.VERBOSE set to 1.
http://fake.ucar.edu:9090/dts/test.01
will have HTTP.VERBOSE set to 0.
Client-Side Certificates
Some systems, notably ESG (Earth System Grid), requires
the use of client-side certificates, as well as being
re-direction based.
This requires setting the following entries:
Note that the first two are to support re-direction based authentication.
Appendix A. All RC-File Keys
For completeness, this is the list of all rc-file keys.
If this documentation is out of date with respect to the actual code,
the code is definitive.
Key curl_easy_setopt Option
HTTP.DEFLATE CUROPT_DEFLATE
with value "deflate,gzip"
HTTP.VERBOSE CUROPT_VERBOSE
HTTP.TIMEOUT CUROPT_TIMEOUT
HTTP.USERAGENT CUROPT_USERAGENT
HTTP.COOKIEJAR CUROPT_COOKIEJAR
HTTP.COOKIE_JAR CUROPT_COOKIEJAR
HTTP.PROXY_SERVER CURLOPT_PROXY,
CURLOPT_PROXYPORT,
CURLOPT_PROXYUSERPWD
HTTP.SSL.CERTIFICATE CUROPT_SSLCERT
HTTP.SSL.KEY CUROPT_SSLKEY
HTTP.SSL.KEYPASSWORD CUROPT_KEYPASSWORD
HTTP.SSL.CAINFO CUROPT_SSLCAINFO
HTTP.SSL.CAPATH CUROPT_SSLCAPATH
HTTP.SSL.VERIFYPEER CUROPT_SSL_VERIFYPEER
HTTP.CREDENTIALS.USERPASSWORD CUROPT_USERPASSWORD
HTTP.NETRC CURLOPT_NETRC,CURLOPT_NETRC_FILE
Appendix B. URS Access in Detail
It is possible to use the NASA Earthdata Login System (URS)
with zz by using using the process specified in the
redirection based authorization section.
In order to access URS controlled datasets, however, it is necessary to
register as a user with NASA at the
https://uat.urs.earthdata.nasa.gov/
website.
Appendix C. ESG Access in Detail
It is possible to access Earth Systems Grid (ESG) datasets
from ESG servers through the ZZ API using the techniques
described in the section on Client-Side Certificates.
Terminology
The key elements for client-side keys requires the constructions of
two "stores" on the client side.
The server actually has a similar set of stores, but the client
need not be concerned with those.
Initial Steps
The first step is to obtain authorization from ESG.
Note that this information may evolve over time, and
may be out of date.
This discussion is in terms of BADC and NCSA. You will need
to substitute as necessary.
https://ceda.ac.uk/openid/Firstname.Lastname
Building the KeyStore
You will have to modify the keyfile in the previous step
and then create a keystore and install the key and a certificate.
The commands are these:
openssl pkcs8 -topk8 -nocrypt -in x509up_u13615 -inform PEM -out key.der -outform DER
openssl x509 -in x509up_u13615 -inform PEM -out cert.der -outform DER
java -classpath
Note, the file names "key.der" and "cert.der" can be whatever you choose.
It is probably best to leave the .der extension, though.
Building the TrustStore
Building the truststore is a bit tricky because as provided, the
certificates in ".globus" need some massaging. See the script below
for the details. The primary command is this, which is executed for every
certificate, c, in globus. It sticks the certificate into the file
named "truststore"
keytool -trustcacerts -storepass "password" -v -keystore "truststore" -importcert -file "${c}"
Running the C Client
Refer to the section on Client-Side Certificates.
The keys specified there must be set in the rc file to support
ESG access.
Of course, the file paths above are suggestions only;
you can modify as needed.
The HTTP.SSL.CERTIFICATE and HTTP.SSL.KEY
entries should have same value, which is the file path for the
certificate produced by MyProxyLogon. The HTTP.SSL.CAPATH entry
should be the path to the "certificates" directory produced by
MyProxyLogon.
Script for creating Stores
The following script shows in detail how to actually construct the key
and trust stores. It is specific to the format of the globus file
as it was when ESG support was first added. It may have changed
since then, in which case, you will need to seek some help
in fixing this script. It would help if you communicated
what you changed to the author so this document can be updated.
#!/bin/sh -x
KEYSTORE="esgkeystore"
TRUSTSTORE="esgtruststore"
GLOBUS="globus"
TRUSTROOT="certificates"
CERT="x509up_u13615"
TRUSTROOTPATH="$GLOBUS/$TRUSTROOT"
CERTFILE="$GLOBUS/$CERT"
PWD="password"
D="-Dglobus=$GLOBUS"
CCP="bcprov-jdk16-145.jar"
CP="./build:${CCP}"
JAR="myproxy.jar"
# Initialize needed directories
rm -fr build
mkdir build
rm -fr $GLOBUS
mkdir $GLOBUS
rm -f $KEYSTORE
rm -f $TRUSTSTORE
# Compile MyProxyCmd and ImportKey
javac -d ./build -classpath "$CCP" *.java
javac -d ./build ImportKey.java
# Execute MyProxyCmd
java -cp "$CP myproxy.MyProxyCmd
# Build the keystore
openssl pkcs8 -topk8 -nocrypt -in $CERTFILE -inform PEM -out key.der -outform DER
openssl x509 -in $CERTFILE -inform PEM -out cert.der -outform DER
java -Dkeypassword=$PWD -Dkeystore=./${KEYSTORE} -cp ./build ImportKey key.der cert.der
# Clean up the certificates in the globus directory
for c in ${TRUSTROOTPATH}/*.0 ; do
alias=`basename $c .0`
sed -e '0,/---/d' <$c >/tmp/${alias}
echo "-----BEGIN CERTIFICATE-----" >$c
cat /tmp/${alias} >>$c
done
# Build the truststore
for c in ${TRUSTROOTPATH}/*.0 ; do
alias=`basename $c .0`
echo "adding: $TRUSTROOTPATH/${c}"
echo "alias: $alias"
yes | keytool -trustcacerts -storepass "$PWD" -v -keystore ./$TRUSTSTORE -alias $alias -importcert -file "${c}"
done
exit