org.kitesdk.data

Class Datasets

java.lang.Object

org.kitesdk.data.Datasets

public class Datasets
extends java.lang.Object

Methods for working with Dataset instances.

URIs

All methods require a URI that identifies a dataset, view, or repository. The URI must begin with the scheme dataset:, view:, or repo:. The remainder of the URI is implementation specific, depending on the dataset scheme.

For example, the URI dataset:hive:movies/ratings references a Hive dataset named ratings in the movies namespace.

The URI view:hdfs:/user/me/movies/ratings?year=2015&month=3 references a view of an HDFS dataset named ratings in the movies namespace in the /user/me path. The view is filtered to include records from only March, 2015.

See Dataset and View URIs.

Entities

Entities are analagous to records in database terminology. The term is used in the API to emphasize that an entity can include not only primitive objects, but also complex objects such as hash maps.

Since:

0.8.0

Constructor Summary

Constructors

Constructor and Description
Datasets()

Method Summary

Methods

Modifier and Type	Method and Description
static <V extends > V	create(java.lang.String uri, DatasetDescriptor descriptor) Create a Dataset for the given dataset or view URI string.
static <E,V extends > V	create(java.lang.String uri, DatasetDescriptor descriptor, java.lang.Class<E> type) Create a Dataset for the given dataset or view URI string.
static <V extends > V	create(java.net.URI uri, DatasetDescriptor descriptor) Create a Dataset for the given dataset or view URI string.
static <E,V extends > V	create(java.net.URI uri, DatasetDescriptor descriptor, java.lang.Class<E> type) Create a Dataset for the given dataset or view URI string.
static boolean	delete(java.lang.String uri) Delete a Dataset identified by the given dataset URI string.
static boolean	delete(java.net.URI uri) Delete a Dataset identified by the given dataset URI.
static boolean	exists(java.lang.String uri) Check whether a Dataset identified by the given URI string exists.
static boolean	exists(java.net.URI uri) Check whether a Dataset identified by the given URI exists.
static java.util.Collection<java.net.URI>	list(java.lang.String uri) List the Dataset URIs in the repository identified by the URI string.
static java.util.Collection<java.net.URI>	list(java.net.URI uri) List the Dataset URIs in the repository identified by the URI.
static <V extends > V	load(java.lang.String uriString) Load a Dataset or View for the given URI.
static <E,V extends > V	load(java.lang.String uriString, java.lang.Class<E> type) Load a Dataset or View for the given URI.
static <V extends > V	load(java.net.URI uri) Load a Dataset or View for the given URI.
static <E,V extends > V	load(java.net.URI uri, java.lang.Class<E> type) Load a Dataset or View for the given URI.
static <D extends > D	update(java.lang.String uri, DatasetDescriptor descriptor) Update a Dataset for the given dataset or view URI string.
static <E,D extends > D	update(java.lang.String uri, DatasetDescriptor descriptor, java.lang.Class<E> type) Update a Dataset for the given dataset or view URI string.
static <D extends > D	update(java.net.URI uri, DatasetDescriptor descriptor) Update a Dataset for the given dataset or view URI.
static <E,D extends > D	update(java.net.URI uri, DatasetDescriptor descriptor, java.lang.Class<E> type) Update a Dataset for the given dataset or view URI.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Datasets

public Datasets()

Method Detail

load

public static <E,V extends > V load(java.net.URI uri,
                         java.lang.Class<E> type)

Load a Dataset or View for the given URI.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

If you use a dataset URI, load returns the unfiltered dataset. If you use a view URI, load returns a View configured to read a subset of the dataset.

Type Parameters:

E - the type used for readers and writers created by this Dataset

V - the type of View expected

Parameters:

uri - a Dataset or View URI

type - a Java class that represents an entity in the dataset

Returns:

a View for the given URI

Throws:

DatasetNotFoundException - if there is no dataset for the given URI

java.lang.NullPointerException - if any arguments are null

load

public static <V extends > V load(java.net.URI uri)

Load a Dataset or View for the given URI.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

If you use a dataset URI, load returns the unfiltered dataset. If you use a view URI, load returns a View configured to read a subset of the dataset.

Type Parameters:

V - the type of View expected

Parameters:

uri - a Dataset or View URI

Returns:

a View for the given URI

Throws:

DatasetNotFoundException - if there is no dataset for the given URI

java.lang.NullPointerException - if any arguments are null

load

public static <E,V extends > V load(java.lang.String uriString,
                         java.lang.Class<E> type)

Load a Dataset or View for the given URI.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

If you use a dataset URI, load returns the unfiltered dataset. If you use a view URI, load returns a View configured to read a subset of the dataset.

Type Parameters:

E - the type used for readers and writers created by this Dataset

V - the type of View expected

Parameters:

uriString - a Dataset or View URI

type - a Java class that represents an entity in the dataset

Returns:

a View for the given URI

Throws:

DatasetNotFoundException - if there is no dataset for the given URI

java.lang.NullPointerException - if any arguments are null

load

public static <V extends > V load(java.lang.String uriString)

Load a Dataset or View for the given URI.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

If you use a dataset URI, load returns the unfiltered dataset. If you use a view URI, load returns a View configured to read a subset of the dataset.

Type Parameters:

V - the type of View expected

Parameters:

uriString - a Dataset or View URI

Returns:

a View for the given URI

Throws:

DatasetNotFoundException - if there is no dataset for the given URI

java.lang.NullPointerException - if any arguments are null

create

public static <E,V extends > V create(java.net.URI uri,
                           DatasetDescriptor descriptor,
                           java.lang.Class<E> type)

Create a Dataset for the given dataset or view URI string. create returns an empty dataset. You can use DatasetWriter to populate your dataset.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme. If the URI is a view URI, this method creates the underlying dataset and returns a view of it.

Type Parameters:

E - the type used for readers and writers created by this Dataset

V - the type of Dataset or View expected

Parameters:

uri - a Dataset or View URI

type - a Java class that represents an entity in the dataset

Returns:

a newly created Dataset responsible for the given URI

Throws:

java.lang.NullPointerException - if URI or descriptor is null

DatasetExistsException - if a Dataset for the given URI already exists

IncompatibleSchemaException - if the schema is not compatible with existing datasets with shared storage (for example, in the same HBase table)

create

public static <V extends > V create(java.net.URI uri,
                         DatasetDescriptor descriptor)

Create a Dataset for the given dataset or view URI string. create returns an empty dataset. You can use DatasetWriter to populate your dataset.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme. If the URI is a view URI, this method creates the underlying dataset and returns a view of it.

Type Parameters:

V - the type of Dataset or View expected

Parameters:

uri - a Dataset or View URI

Returns:

a newly created Dataset responsible for the given URI

Throws:

java.lang.NullPointerException - if URI or descriptor is null

DatasetExistsException - if a Dataset for the given URI already exists

IncompatibleSchemaException - if the schema is not compatible with existing datasets with shared storage (for example, in the same HBase table)

create

public static <E,V extends > V create(java.lang.String uri,
                           DatasetDescriptor descriptor,
                           java.lang.Class<E> type)

Create a Dataset for the given dataset or view URI string. create returns an empty dataset. You can use DatasetWriter to populate your dataset.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme. If the URI is a view URI, this method creates the underlying dataset and returns a view of it.

Type Parameters:

E - the type used for readers and writers created by this Dataset

V - the type of Dataset or View expected

Parameters:

uri - a Dataset or View URI string

type - a Java class that represents an entity in the dataset

Returns:

a newly created Dataset responsible for the given URI

Throws:

java.lang.NullPointerException - if URI or descriptor is null

DatasetExistsException - if a Dataset for the given URI already exists

IncompatibleSchemaException - if the schema is not compatible with existing datasets with shared storage (for example, in the same HBase table)

create

public static <V extends > V create(java.lang.String uri,
                         DatasetDescriptor descriptor)

Create a Dataset for the given dataset or view URI string. create returns an empty dataset. You can use DatasetWriter to populate your dataset.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme. If the URI is a view URI, this method creates the underlying dataset and returns a view of it.

Type Parameters:

V - the type of Dataset or View expected

Parameters:

uri - a Dataset or View URI string

Returns:

a newly created Dataset responsible for the given URI

Throws:

java.lang.NullPointerException - if URI or descriptor is null

DatasetExistsException - if a Dataset for the given URI already exists

IncompatibleSchemaException - if the schema is not compatible with existing datasets with shared storage (for example, in the same HBase table)

update

public static <E,D extends > D update(java.net.URI uri,
                           DatasetDescriptor descriptor,
                           java.lang.Class<E> type)

Update a Dataset for the given dataset or view URI.

You can add columns, remove columns, or change the data type of columns in your dataset, provided you don't attempt a change that is incompatible with written data. Avro defines rules for compatible schema evolution. See Schema Evolution.

This method updates the dataset descriptor, so you can also add or change properties.

The recommended way to update a dataset descriptor is to build it based on an existing descriptor. Use DatasetDescriptor.Builder(DatasetDescriptor descriptor) to build a DatasetDescriptor based on an existing instance.

You cannot change a dataset format or partition strategy.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Type Parameters:

E - the type used for readers and writers created by this Dataset

D - the type of Dataset expected

Parameters:

uri - a Dataset or View URI

type - a Java class that represents an entity in the dataset

Returns:

a Dataset for the given URI; does not return a view, even when you pass a view URI

Throws:

java.lang.NullPointerException - if URI is null

DatasetNotFoundException - if there is no dataset for the given URI

java.lang.UnsupportedOperationException - if descriptor updates are not supported by the implementation

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

IncompatibleSchemaException - if the schema is not compatible with previous schemas, or with existing datasets with shared storage (for example, in the same HBase table)

update

public static <D extends > D update(java.net.URI uri,
                         DatasetDescriptor descriptor)

Update a Dataset for the given dataset or view URI.

You can add columns, remove columns, or change the data type of columns in your dataset, provided you don't attempt a change that is incompatible with written data. Avro defines rules for compatible schema evolution. See Schema Evolution.

This method updates the dataset descriptor, so you can also add or change properties.

The recommended way to update a dataset descriptor is to build it based on an existing descriptor. Use DatasetDescriptor.Builder(DatasetDescriptor descriptor) to build a DatasetDescriptor based on an existing instance.

You cannot change a dataset format or partition strategy.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Type Parameters:

D - the type of Dataset expected

Parameters:

uri - a Dataset or View URI

Returns:

a Dataset for the given URI; does not return a view, even when you pass a view URI

Throws:

java.lang.NullPointerException - if URI is null

DatasetNotFoundException - if there is no dataset for the given URI

java.lang.UnsupportedOperationException - if descriptor updates are not supported by the implementation

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

IncompatibleSchemaException - if the schema is not compatible with previous schemas, or with existing datasets with shared storage (for example, in the same HBase table)

update

public static <E,D extends > D update(java.lang.String uri,
                           DatasetDescriptor descriptor,
                           java.lang.Class<E> type)

Update a Dataset for the given dataset or view URI string.

You can add columns, remove columns, or change the data type of columns in your dataset, provided you don't attempt a change that is incompatible with written data. Avro defines rules for compatible schema evolution. See Schema Evolution.

This method updates the dataset descriptor, so you can also add or change properties.

The recommended way to update a dataset descriptor is to build it based on an existing descriptor. Use DatasetDescriptor.Builder(DatasetDescriptor descriptor) to build a DatasetDescriptor based on an existing instance.

You cannot change a dataset format or partition strategy.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Type Parameters:

E - the type used for readers and writers created by this Dataset

D - the type of Dataset expected

Parameters:

uri - a Dataset or View URI string

type - a Java class that represents an entity in the dataset

Returns:

a Dataset for the given URI; does not return a view, even when you pass a view URI

Throws:

java.lang.NullPointerException - if URI is null

DatasetNotFoundException - if there is no dataset for the given URI

java.lang.UnsupportedOperationException - if descriptor updates are not supported by the implementation

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

IncompatibleSchemaException - if the schema is not compatible with previous schemas, or with existing datasets with shared storage (for example, in the same HBase table)

update

public static <D extends > D update(java.lang.String uri,
                         DatasetDescriptor descriptor)

Update a Dataset for the given dataset or view URI string.

You can add columns, remove columns, or change the data type of columns in your dataset, provided you don't attempt a change that is incompatible with written data. Avro defines rules for compatible schema evolution. See Schema Evolution.

This method updates the dataset descriptor, so you can also add or change properties.

The recommended way to update a dataset descriptor is to build it based on an existing descriptor. Use DatasetDescriptor.Builder(DatasetDescriptor descriptor) to build a DatasetDescriptor based on an existing instance.

You cannot change a dataset format or partition strategy.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Type Parameters:

D - the type of Dataset expected

Parameters:

uri - a Dataset or View URI string

Returns:

a Dataset for the given URI; does not return a view, even when you pass a view URI

Throws:

java.lang.NullPointerException - if URI is null

DatasetNotFoundException - if there is no dataset for the given URI

java.lang.UnsupportedOperationException - if descriptor updates are not supported by the implementation

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

IncompatibleSchemaException - if the schema is not compatible with previous schemas, or with existing datasets with shared storage (for example, in the same HBase table)

delete

public static boolean delete(java.net.URI uri)

Delete a Dataset identified by the given dataset URI.

When you call this method using a dataset URI, both data and metadata are deleted. After you call this method, the dataset no longer exists, unless an exception is thrown.

When you call this method using a view URI, data in that view is deleted. The dataset's metadata is not changed. This can throw an UnsupportedOperationException if the delete requires additional work. For example, if some, but not all, of the data in an underlying data file must be removed, then the implementation is allowed to reject the deletion rather than copy the remaining records to a new file. An implementation must document under what conditions it accepts deletes, and under what conditions it rejects them.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Parameters:

uri - a Dataset URI

Returns:

true if any data or metadata is removed, false otherwise

Throws:

java.lang.NullPointerException - if URI is null

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

delete

public static boolean delete(java.lang.String uri)

Delete a Dataset identified by the given dataset URI string.

When you call this method using a dataset URI, both data and metadata are deleted. After you call this method, the dataset no longer exists, unless an exception is thrown.

When you call this method using a view URI, data in that view is deleted. The dataset's metadata is not changed. This can throw an UnsupportedOperationException if the delete requires additional work. For example, if some, but not all, of the data in an underlying data file must be removed, then the implementation is allowed to reject the deletion rather than copy the remaining records to a new file. An implementation must document under what conditions it accepts deletes, and under what conditions it rejects them.

URIs must begin with dataset: or view:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Parameters:

uri - a Dataset URI string

Returns:

true if any data or metadata is removed, false otherwise

Throws:

java.lang.NullPointerException - if URI is null

ConcurrentSchemaModificationException - if the Dataset schema is updated concurrently

exists

public static boolean exists(java.net.URI uri)

Check whether a Dataset identified by the given URI exists.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Parameters:

uri - a Dataset URI

Returns:

true if the dataset exists, false otherwise

Throws:

java.lang.NullPointerException - if URI is null

exists

public static boolean exists(java.lang.String uri)

Check whether a Dataset identified by the given URI string exists.

URIs must begin with dataset:. The remainder of the URI is implementation specific, depending on the dataset scheme.

Parameters:

uri - a Dataset URI string

Returns:

true if the dataset exists, false otherwise

list

public static java.util.Collection<java.net.URI> list(java.net.URI uri)

List the Dataset URIs in the repository identified by the URI.

URI formats are defined by Dataset implementations. The repository URIs you pass to this method must begin with repo:. For example, to list the Dataset URIs for the Hive repository, provide the URI repo:hive.

Parameters:

uri - a DatasetRepository URI

Returns:

the URIs present in the DatasetRepository

list

public static java.util.Collection<java.net.URI> list(java.lang.String uri)

List the Dataset URIs in the repository identified by the URI string.

URI formats are defined by Dataset implementations. The repository URIs you pass to this method must begin with repo:. For example, to list the Dataset URIs for the Hive repository, provide the URI repo:hive.

Parameters:

uri - a DatasetRepository URI string

Returns:

the URIs present in the DatasetRepository