Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
A name for your application, to display on the cluster web UI.
Location where Spark is installed on cluster nodes.
Collection of JARs to send to the cluster. These can be paths on the local file system or HDFS, HTTP, HTTPS, or FTP URLs.
Environment variables to set on worker nodes.
Create an Accumulable shared variable, to which tasks can add values with +=.
Create an Accumulable shared variable, to which tasks can add values with +=.
Only the driver can access the accumuable's value.
accumulator type
type that can be added to the accumulator
Create an accumulator from a "mutable collection" type.
Create an accumulator from a "mutable collection" type.
Growable and TraversableOnce are the standard APIs that guarantee += and ++=, implemented by standard mutable collections. So you can use this with mutable Map, Set, etc.
Create an Accumulator variable of a given type, which tasks can "add" values
to using the += method.
Create an Accumulator variable of a given type, which tasks can "add" values
to using the += method. Only the driver can access the accumulator's value.
Add a file to be downloaded with this Spark job on every node.
Add a file to be downloaded with this Spark job on every node.
The path passed can be either a local file, a file in HDFS (or other Hadoop-supported
filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
use SparkFiles.get(path) to find its download location.
Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
The path passed can be either a local file, a file in HDFS (or other Hadoop-supported
filesystems), or an HTTP, HTTPS or FTP URI.
A name for your application, to display on the cluster web UI.
Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions.
Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.
Clear the job's list of files added by addFile so that they do not get downloaded to
any new nodes.
Clear the job's list of JARs added by addJar so that they do not get downloaded to
any new nodes.
Default min number of partitions for Hadoop RDDs when not given by user
Default level of parallelism to use when not given by user (e.
Default level of parallelism to use when not given by user (e.g. parallelize and makeRDD).
Environment variables to set on worker nodes.
Return a map from the slave to the max memory available for caching and the remaining memory available for caching.
Return information about blocks stored in all of the slaves
Return information about what RDDs are cached, if they are in mem or on disk, how much space they take, etc.
A default Hadoop Configuration for the Hadoop code (e.
A default Hadoop Configuration for the Hadoop code (e.g. file systems) that we reuse.
Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.
Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly. Instead, callers can just write, for example,
val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path)
Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.
Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly. Instead, callers can just write, for example,
val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path, minSplits)
Get an RDD for a Hadoop file with an arbitrary InputFormat
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf giving its InputFormat and any other necessary info (e.
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).
Collection of JARs to send to the cluster.
Collection of JARs to send to the cluster. These can be paths on the local file system or HDFS, HTTP, HTTPS, or FTP URLs.
Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object.
Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object. Create a new partition for each collection item.
Distribute a local Scala collection to form an RDD.
Cluster URL to connect to (e.
Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
Get an RDD for a Hadoop file with an arbitrary new API InputFormat.
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.
Distribute a local Scala collection to form an RDD.
Run a job that can return approximate results.
Run a job on all partitions in an RDD and pass the results to a handler function.
Run a job on all partitions in an RDD and pass the results to a handler function.
Run a job on all partitions in an RDD and return the results in an array.
Run a job on all partitions in an RDD and return the results in an array.
Run a job on a given set of partitions of an RDD, but take a function of type
Iterator[T] => U instead of (TaskContext, Iterator[T]) => U.
Run a function on a given set of partitions in an RDD and return the results as an array.
Run a function on a given set of partitions in an RDD and return the results as an array. The allowLocal flag specifies whether the scheduler can run the computation on the driver rather than shipping it out to the cluster, for short actions like first().
Run a function on a given set of partitions in an RDD and pass the results to the given handler function.
Run a function on a given set of partitions in an RDD and pass the results to the given handler function. This is the main entry point for all actions in Spark. The allowLocal flag specifies whether the scheduler can run the computation on the driver rather than shipping it out to the cluster, for short actions like first().
Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter.
Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter. For example, to access a SequenceFile where the keys are Text and the values are IntWritable, you could simply write
sparkContext.sequenceFile[String, Int](path, ...)
WritableConverters are provided in a somewhat strange way (by an implicit function) to support both subclasses of Writable and types for which we define a converter (e.g. Int to IntWritable). The most natural thing would've been to have implicit objects for the converters, but then we couldn't have an object for every subclass of Writable (you can't have a parameterized singleton object). We use functions instead to create a new converter for the appropriate type. In addition, we pass the converter a ClassManifest of its type to allow it to figure out the Writable class to use in the subclass case.
Get an RDD for a Hadoop SequenceFile with given key and value types.
Get an RDD for a Hadoop SequenceFile with given key and value types.
Set the directory under which RDDs are going to be checkpointed.
Set the directory under which RDDs are going to be checkpointed. The directory must be a HDFS path if running on a cluster. If the directory does not exist, it will be created. If the directory exists and useExisting is set to true, then the exisiting directory will be used. Otherwise an exception will be thrown to prevent accidental overriding of checkpoint files in the existing directory.
Location where Spark is installed on cluster nodes.
Shut down the SparkContext.
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
Build the union of a list of RDDs passed as variable-length arguments.
Build the union of a list of RDDs.
Main entry point for Spark functionality. A SparkContext represents the connection to a Spark cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster.