Spyral.Corpus(id)

The Corpus class in Spyral. To get started you first need to load a Corpus, using either a pre-existing Corpus id, or some input data. For this you can use the loadCorpus method, which is an alias of Spyral.Corpus.load.

Here's a simple example:

loadCorpus("Hello World!").summary();

This loads a corpus and returns an asynchronous Promise, but all of the methods of Corpus are appended to the Promise, so Spyral.Corpus#summary will be called once the Corpus promise is fulfilled. It's equivalent to the following:

loadCorpus("Hello World!").then(corpus -> corpus.summary());

Have a look at the Spyral.Corpus~CorpusConfig configuration for more examples.

new Corpus(id)

Create a new Corpus using the specified Corpus ID

Parameters:
Name Type Description
id string

The Corpus ID

Methods

analysis(config) → {Promise.<Object>}

Performs one of several dimension reduction statistical analysis techniques.

For more details see the scatterplot tutorial.

Parameters:
Name Type Description
config Object
Properties
Name Type Description
type string

The type of analysis technique to use: 'ca', 'pca', 'tsne', 'docsim'

start number

The zero-based start of the list

limit number

A limit to the number of items to return at a time

dimensions number

The number of dimensions to render, either 2 or 3.

bins number

The number of bins to separate a document into.

clusters number

The number of clusters within which to group words.

perplexity number

The TSNE perplexity value.

iterations number

The TSNE iterations value.

comparisonType string

The value to use for comparing terms. Options are: 'raw', 'relative', and 'tfidf'.

target string

The term to set as the target. This will filter results to terms that are near the target.

term string

Used in combination with "target" as a white list of terms to keep.

query string

A term query (see search tutorial)

stopList string

A list of stopwords to include (see stopwords tutorial)

Returns:
Promise.<Object>

collocates(config) → {Promise.<Array>}

Returns an array of collocates (either document or corpus collocates, depending on the specified mode).

The mode is set to "documents" when any of the following is true

  • the mode parameter is set to "documents"
  • a docIndex parameter being set
  • a docId parameter being set

The following is an example a Corpus Collocate (corpus mode):

{
	"term": "love",
	"rawFreq": 568,
	"contextTerm": "mr",
	"contextTermRawFreq": 24
}

The following is an example of Document Collocate (documents mode):

{
		"docIndex": 4,
		"keyword": "love",
		"keywordContextRawFrequency": 124,
		"term": "fanny",
		"termContextRawFrequency": 8,
		"termContextRelativeFrequency": 0.021680217,
		"termDocumentRawFrequency": 816,
		"termDocumentRelativeFrequency": 0.0050853477,
		"termContextDocumentRelativeFrequencyDifference": 0.01659487
}

The following config parameters are valid in both modes:

  • start: the zero-based start index of the list (for paging)
  • limit: the maximum number of terms to provide per request
  • query: a term query (see search tutorial)
  • stopList: a list of stopwords to include (see stopwords tutorial)
  • collocatesWhitelist: collocates will be limited to this list
  • context: the size of the context (the number of words on each side of the keyword)
  • dir: sort direction, **ASC**ending or **DESC**ending

The following are specific to corpus mode:

  • sort: the order of the terms, one of the following: RAWFREQ, TERM, CONTEXTTERM, CONTEXTTERMRAWFREQ

The following are specific to documents mode:

  • sort: the order of the terms, one of the following: TERM, REL, REL, RAW, DOCREL, DOCRAW, CONTEXTDOCRELDIFF
  • docIndex: the zero-based index of the documents to include (use commas to separate multiple values)
  • docId: the document IDs to include (use commas to separate multiple values)

An example:

// show top 5 collocate terms
loadCorpus("austen").collocates({stopList: 'auto', limit: 5}).then(terms => terms.map(term => term.term))
Parameters:
Name Type Description
config Object

an Object specifying parameters (see list above)

Properties
Name Type Description
start number

the zero-based start index of the list (for paging)

limit number

the maximum number of terms to provide per request

query string

a term query (see search tutorial)

stopList string

a list of stopwords to include (see stopwords tutorial)

collocatesWhitelist string

collocates will be limited to this list

context number

the size of the context (the number of words on each side of the keyword)

dir string

sort direction, **ASC**ending or **DESC**ending

Returns:
Promise.<Array> -

a Promise for a Array of Terms

contexts(config) → {Promise.<Array>}

Returns an array of Objects that contain keywords in contexts (KWICs).

An individual KWIC Object looks something like this:

{
		"docIndex": 0,
		"query": "love",
		"term": "love",
		"position": 0,
		"left": "FREINDSHIP AND OTHER EARLY WORKS",
		"middle": "Love",
		"right": " And Friendship And Other Early"
}

The following are valid in the config parameter:

  • start: the zero-based start index of the list (for paging)
  • limit: the maximum number of terms to provide per request
  • query: a term query (see search tutorial)
  • sort: the order of the contexts: TERM, DOCINDEX, POSITION, LEFT, RIGHT
  • dir: sort direction, **ASC**ending or **DESC**ending
  • perDocLimit: the limit parameter is for the total number of terms returned, this parameter allows you to specify a limit value per document
  • stripTags: for the left, middle and right values, one of the following: ALL, BLOCKSONLY (tries to maintain blocks for line formatting), NONE (default)
  • context: the size of the context (the number of words on each side of the keyword)
  • docIndex: the zero-based index of the documents to include (use commas to separate multiple values)
  • docId: the document IDs to include (use commas to separate multiple values)
  • overlapStrategy: determines how to handle cases where there's overlap between KWICs, such as "to be or not to be" when the keyword is "be"; here are the options:
    • none: nevermind the overlap, keep all words
      • {left: "to", middle: "be", right: "or not to be"}
      • {left: "to be or not to", middle: "be", right: ""}
    • first: priority goes to the first occurrence (some may be dropped)
      • {left: "to", middle: "be", right: "or not to be"}
    • merge: balance the words between overlapping occurrences
      • {left: "to", middle: "be", right: "or"}
      • {left: "not to", middle: "be", right: ""}

An example:

// load the first 20 words in the corpus
loadCorpus("austen").contexts({query: "love", limit: 10})
Parameters:
Name Type Description
config Object

an Object specifying parameters (see above)

Properties
Name Type Description
start number

the zero-based start index of the list (for paging)

limit number

the maximum number of terms to provide per request

query string

a term query (see search tutorial)

sort string

the order of the contexts: TERM, DOCINDEX, POSITION, LEFT, RIGHT

dir string

sort direction, **ASC**ending or **DESC**ending

perDocLimit number

the limit parameter is for the total number of terms returned, this parameter allows you to specify a limit value per document

stripTags string

for the left, middle and right values, one of the following: ALL, BLOCKSONLY (tries to maintain blocks for line formatting), NONE (default)

context number

the size of the context (the number of words on each side of the keyword)

docIndex number

the zero-based index of the documents to include (use commas to separate multiple values)

docId string

the document IDs to include (use commas to separate multiple values)

overlapStrategy string

determines how to handle cases where there's overlap between KWICs, such as "to be or not to be" when the keyword is "be"

Returns:
Promise.<Array> -

a Promise for an Array of KWIC Objects

correlations(config) → {Promise.<Array>}

Returns an array of correlations (either document or corpus correlations, depending on the specified mode).

The mode is set to "documents" when any of the following is true

  • the mode parameter is set to "documents"
  • a docIndex parameter being set
  • a docId parameter being set

The following is an example a Corpus correlation (corpus mode):

{
	"source": {
		"term": "mrs",
		"inDocumentsCount": 8,
		"rawFreq": 2531,
		"relativePeakedness": 0.46444246,
		"relativeSkewness": -0.44197384
	},
	"target": {
		"term": "love",
		"inDocumentsCount": 8,
		"rawFreq": 568,
		"relativePeakedness": 5.763066,
		"relativeSkewness": 2.2536576
	},
	"correlation": -0.44287738,
	"significance": 0.08580014
}

The following is an example of Document correlation (documents mode), without positions requested:

{
	"source": {
		"term": "confide",
		"rawFreq": 3,
		"relativeFreq": 89.3948,
		"zscore": -0.10560975,
		"zscoreRatio": -0.7541012,
		"tfidf": 1.1168874E-5,
		"totalTermsCount": 33559,
		"docIndex": 0,
		"docId": "8a61d5d851a69c03c6ba9cc446713574"
	},
	"target": {
		"term": "love",
		"rawFreq": 54,
		"relativeFreq": 1609.1063,
		"zscore": 53.830048,
		"zscoreRatio": -707.44696,
		"tfidf": 0.0,
		"totalTermsCount": 33559,
		"docIndex": 0,
		"docId": "8a61d5d851a69c03c6ba9cc446713574"
	},
	"correlation": 0.93527687,
	"significance": 7.0970666E-5
}

The following config parameters are valid in both modes:

  • start: the zero-based start index of the list (for paging)
  • limit: the maximum number of terms to provide per request
  • termsOnly: a very compact data view of the correlations
  • sort: the order of the terms, one of the following: CORRELATION, CORRELATIONABS
  • dir: sort direction, **ASC**ending or **DESC**ending

The following is specific to corpus mode:

  • minInDocumentsCountRatio: the minimum coverage (as a percentage between 0 and 100) of the term, amongst all the documents

The following are specific to documents mode:

  • docIndex: the zero-based index of the documents to include (use commas to separate multiple values)
  • docId: the document IDs to include (use commas to separate multiple values)

An example:

// load the first 10 phrases in the corpus
loadCorpus("austen").correlations({query: "love", limit: 10})
Parameters:
Name Type Description
config Object

an Object specifying parameters (see above)

Properties
Name Type Description
start number

the zero-based start index of the list (for paging)

limit number

the maximum number of terms to provide per request

minInDocumentsCountRatio number

the minimum coverage (as a percentage between 0 and 100) of the term, amongst all the documents

termsOnly boolean

a very compact data view of the correlations

sort string

the order of the terms, one of the following: CORRELATION, CORRELATIONABS

dir string

sort direction, **ASC**ending or **DESC**ending

Returns:
Promise.<Array> -

a Promise for an Array of phrase Objects

documents(config) → {Promise.<Array>}

Returns an array of documents metadata for the corpus.

The following are valid in the config parameter:

  • start: the zero-based start of the list
  • limit: a limit to the number of items to return at a time
  • docIndex: a zero-based list of documents (first document is zero, etc.); multiple documents can be separated by a comma
  • docId: a set of document IDs; multiple documents can be separated by a comma
  • query: one or more term queries for the title, author or full-text
  • sort: one of the following sort orders: INDEX, TITLE, AUTHOR, TOKENSCOUNTLEXICAL, TYPESCOUNTLEXICAL, TYPETOKENRATIOLEXICAL, PUBDATE
  • dir: sort direction, **ASC**ending or **DESC**ending
Parameters:
Name Type Description
config Object

an Object specifying parameters (see list above)

Properties
Name Type Description
start number

the zero-based start of the list

limit number

a limit to the number of items to return at a time

docIndex number

a zero-based list of documents (first document is zero, etc.); multiple documents can be separated by a comma

docId string

a set of document IDs; multiple documents can be separated by a comma

query string

one or more term queries for the title, author or full-text

sort string

one of the following sort orders: INDEX, TITLE, AUTHOR, TOKENSCOUNTLEXICAL, TYPESCOUNTLEXICAL, TYPETOKENRATIOLEXICAL, PUBDATE

dir string

sort direction, **ASC**ending or **DESC**ending

Returns:
Promise.<Array> -

a Promise for an Array of documents metadata

entities(config) → {Promise.<Array>}

Returns an array of entities.

The config object as parameter can contain the following:

  • docIndex: document index to restrict to (can be comma-separated list)
  • annotator: the annotator to use: 'stanford' or 'nssi' or 'spacy'
Parameters:
Name Type Description
config Object
Properties
Name Type Description
docIndex number | string

document index to restrict to (can be comma-separated list)

annotator string

the annotator to use: 'stanford' or 'nssi' or 'spacy'

Returns:
Promise.<Array>

async filterByCategory(categories, categoryNameopt) → {Promise.<Object>}

Given a Categories instance or ID, returns an object mapping category names to corpus terms. The results can be limited to specific category names by providing one or more of them.

Parameters:
Name Type Attributes Description
categories String | Spyral.Categories

A categories ID or a Spyral.Categories instance.

categoryName String | Array.<String> <optional>

One or more names of categories within the instance.

Returns:
Promise.<Object>

id() → {Promise.<string>}

Returns the ID of the corpus.

Returns:
Promise.<string> -

a Promise for the string ID of the corpus

lemmas(config) → {Promise.<Array>}

Get lemmas. This is the equivalent of calling: this.tokens({ withPosLemmas: true, noOthers: true })

Parameters:
Name Type Description
config Object

an Object specifying parameters (see above)

Returns:
Promise.<Array> -

a Promise for an Array of lemma Objects

metadata(config) → {Promise.<object>}

Returns the metadata object (of the corpus or document, depending on which mode is used).

The following is an example of the object return for the metadata of the Jane Austen corpus:

{
	"id": "b50407fd1cbbecec4315a8fc411bad3c",
	"alias": "austen",
	"title": "",
	"subTitle": "",
	"documentsCount": 8,
	"createdTime": 1582429585984,
	"createdDate": "2020-02-22T22:46:25.984-0500",
	"lexicalTokensCount": 781763,
	"lexicalTypesCount": 15368,
	"noPasswordAccess": "NORMAL",
	"languageCodes": [
		"en"
	]
}

The following is an example of what is returned as metadata for the first document:

[
	{
	"id": "ddac6b12c3f4261013c63d04e8d21b45",
	"extra.X-Parsed-By": "org.apache.tika.parser.DefaultParser",
	"tokensCount-lexical": "33559",
	"lastTokenStartOffset-lexical": "259750",
	"parent_modified": "1548457455000",
	"typesCount-lexical": "4235",
	"typesCountMean-lexical": "7.924203",
	"lastTokenPositionIndex-lexical": "33558",
	"index": "0",
	"language": "en",
	"sentencesCount": "1302",
	"source": "stream",
	"typesCountStdDev-lexical": "46.626404",
	"title": "1790 Love And Freindship",
	"parent_queryParameters": "VOYANT_BUILD=M16&textarea-1015-inputEl=Type+in+one+or+more+URLs+on+separate+lines+or+paste+in+a+full+text.&VOYANT_REMOTE_ID=199.229.249.196&accessIP=199.229.249.196&VOYANT_VERSION=2.4&palette=default&suppressTools=false",
	"extra.Content-Type": "text/plain; charset=windows-1252",
	"parentType": "expansion",
	"extra.Content-Encoding": "windows-1252",
	"parent_source": "file",
	"parent_id": "ae47e3a72cd3cad51e196e8a41e21aec",
	"modified": "1432861756000",
	"location": "1790 Love And Freindship.txt",
	"parent_title": "Austen",
	"parent_location": "Austen.zip"
	}
]

In Corpus mode there's no reason to specify arguments. In documents mode you can request specific documents in the config object:

  • start: the zero-based start of the list
  • limit: a limit to the number of items to return at a time
  • docIndex: a zero-based list of documents (first document is zero, etc.); multiple documents can be separated by a comma
  • docId: a set of document IDs; multiple documents can be separated by a comma
  • query: one or more term queries for the title, author or full-text
  • sort: one of the following sort orders: INDEX, TITLE, AUTHOR, TOKENSCOUNTLEXICAL, TYPESCOUNTLEXICAL, TYPETOKENRATIOLEXICAL, PUBDATE
  • dir: sort direction, **ASC**ending or **DESC**ending

An example:

// this would show the number 8 (the size of the corpus)
loadCorpus("austen").metadata().then(metadata => metadata.documentsCount)
Parameters:
Name Type Description
config Object

an Object specifying parameters (see list above)

Returns:
Promise.<object> -

a Promise for an Object containing metadata

phrases(config) → {Promise.<Array>}

Returns an array of phrases or n-grams (either document or corpus phrases, depending on the specified mode).

The mode is set to "documents" when any of the following is true

  • the mode parameter is set to "documents"
  • a docIndex parameter being set
  • a docId parameter being set

The following is an example a Corpus phrase (corpus mode), without distributions requested:

{
	"term": "love with",
	"rawFreq": 103,
	"length": 2
}

The following is an example of Document phrase (documents mode), without positions requested:

{
	"term": "love with",
	"rawFreq": 31,
	"length": 2,
	"docIndex": 5
}

The following config parameters are valid in both modes:

  • start: the zero-based start index of the list (for paging)
  • limit: the maximum number of terms to provide per request
  • minLength: the minimum length of the phrase
  • maxLength: the maximum length of the phrase
  • minRawFreq: the minimum raw frequency of the phrase
    • sort: the order of the terms, one of the following: RAWFREQ, TERM, LENGTH
  • dir: sort direction, **ASC**ending or **DESC**ending
  • overlapFilter: it happens that phrases contain other phrases and we need a strategy for handling overlap:
    • NONE: nevermind the overlap, keep all phrases
    • LENGTHFIRST: priority goes to the longest phrases
    • RAWFREQFIRST: priority goes to the highest frequency phrases
    • POSITIONFIRST: priority goes to the first phrases

The following are specific to documents mode:

  • docIndex: the zero-based index of the documents to include (use commas to separate multiple values)
  • docId: the document IDs to include (use commas to separate multiple values)

An example:

// load the first 20 phrases in the corpus
loadCorpus("austen").phrases({query: "love", limit: 10})
Parameters:
Name Type Description
config Object

an Object specifying parameters (see above)

Properties
Name Type Description
start number

the zero-based start index of the list (for paging)

limit number

the maximum number of terms to provide per request

minLength number

the minimum length of the phrase

maxLength number

the maximum length of the phrase

minRawFreq number

the minimum raw frequency of the phrase

sort string

the order of the terms, one of the following: RAWFREQ, TERM, LENGTH

dir string

sort direction, **ASC**ending or **DESC**ending

overlapFilter string

it happens that phrases contain other phrases and we need a strategy for handling overlap

Returns:
Promise.<Array> -

a Promise for an Array of phrase Objects

summary() → {Promise.<string>}

Returns a brief summary of the corpus that includes essential metadata (documents count, terms count, etc.)

An example:

loadCorpus("austen").summary();
Returns:
Promise.<string> -

a Promise for a string containing a brief summary of the corpus metadata

terms(config) → {Promise.<Array>}

Returns an array of terms (either CorpusTerms or DocumentTerms, depending on the specified mode). These terms are actually types, so information about each type is collected (as opposed to the tokens method which is for every occurrence in document order).

The mode is set to "documents" when any of the following is true

  • the mode parameter is set to "documents"
  • a docIndex parameter being set
  • a docId parameter being set

The following is an example a Corpus Term (corpus mode):

{
	"term": "the",
	"inDocumentsCount": 8,
	"rawFreq": 28292,
	"relativeFreq": 0.036189996,
	"comparisonRelativeFreqDifference": 0
}

The following is an example of Document Term (documents mode):

{
	"term": "the",
	"rawFreq": 1333,
	"relativeFreq": 39721.086,
	"zscore": 28.419,
	"zscoreRatio": -373.4891,
	"tfidf": 0.0,
	"totalTermsCount": 33559,
	"docIndex": 0,
	"docId": "8a61d5d851a69c03c6ba9cc446713574"
}

The following config parameters are valid in both modes:

  • start: the zero-based start index of the list (for paging)
  • limit: the maximum number of terms to provide per request
  • minRawFreq: the minimum raw frequency of terms
  • query: a term query (see search tutorial)
  • stopList: a list of stopwords to include (see stopwords tutorial)
  • withDistributions: a true value shows distribution across the corpus (corpus mode) or across the document (documents mode)
  • whiteList: a keyword list – terms will be limited to this list
  • tokenType: the token type to use, by default lexical (other possible values might be title and author)
  • dir: sort direction, **ASC**ending or **DESC**ending

The following are specific to corpus mode:

  • bins: by default there are the same number of bins as there are documents (for distribution values), this can be modified
  • corpusComparison: you can provide the ID of a corpus for comparison of frequency values
  • inDocumentsCountOnly: if you don't need term frequencies but only frequency per document set this to true
  • sort: the order of the terms, one of the following: INDOCUMENTSCOUNT, RAWFREQ, TERM, RELATIVEPEAKEDNESS, RELATIVESKEWNESS, COMPARISONRELATIVEFREQDIFFERENCE

The following are specific to documents mode:

  • bins: by default the document is divided into 10 equal bins(for distribution values), this can be modified
  • sort: the order of the terms, one of the following: RAWFREQ, RELATIVEFREQ, TERM, TFIDF, ZSCORE
  • perDocLimit: the limit parameter is for the total number of terms returned, this parameter allows you to specify a limit value per document
  • docIndex: the zero-based index of the documents to include (use commas to separate multiple values)
  • docId: the document IDs to include (use commas to separate multiple values)

An example:

// show top 5 terms
loadCorpus("austen").terms({stopList: 'auto', limit: 5}).then(terms => terms.map(term => term.term))

// show top term for each document
loadCorpus("austen").terms({stopList: 'auto', perDocLimit: 1, mode: 'documents'}).then(terms => terms.map(term => term.term))
Parameters:
Name Type Description
config Object

an Object specifying parameters (see list above)

Properties
Name Type Description
start number

the zero-based start index of the list (for paging)

limit number

the maximum number of terms to provide per request

minRawFreq number

the minimum raw frequency of terms

query string

a term query (see search tutorial)

stopList string

a list of stopwords to include (see stopwords tutorial)

withDistributions boolean

a true value shows distribution across the corpus (corpus mode) or across the document (documents mode)

whiteList string

a keyword list – terms will be limited to this list

tokenType string

the token type to use, by default lexical (other possible values might be title and author)

dir string

sort direction, **ASC**ending or **DESC**ending

Returns:
Promise.<Array> -

a Promise for a Array of Terms

text(config) → {Promise.<string>}

Returns the text of the entire corpus.

Texts are concatenated together with two new lines and three dashes (\n\n---\n\n)

The following are valid in the config parameter:

  • noMarkup: strips away the markup
  • compactSpace: strips away superfluous spaces and multiple new lines
  • limit: a limit to the number of characters (per text)
  • format: text for plain text, any other value for the simplified Voyant markup

An example:

// fetch 1000 characters from each text in the corpus into a single string
loadCorpus("austen").text({limit:1000})
Parameters:
Name Type Description
config Object

an Object specifying parameters (see list above)

Properties
Name Type Description
noMarkup boolean

strips away the markup

compactSpace boolean

strips away superfluous spaces and multiple new lines

limit number

a limit to the number of characters (per text)

format string

text for plain text, any other value for the simplified Voyant markup

Returns:
Promise.<string> -

a Promise for a string of the corpus

texts(config) → {Promise.<Array>}

Returns an array of texts from the entire corpus.

The following are valid in the config parameter:

  • noMarkup: strips away the markup
  • compactSpace: strips away superfluous spaces and multiple new lines
  • limit: a limit to the number of characters (per text)
  • format: text for plain text, any other value for the simplified Voyant markup

An example:

// fetch 1000 characters from each text in the corpus into an Array
loadCorpus("austen").texts({limit:1000})
Parameters:
Name Type Description
config Object

an Object specifying parameters (see list above)

Properties
Name Type Description
noMarkup boolean

strips away the markup

compactSpace boolean

strips away superfluous spaces and multiple new lines

limit number

a limit to the number of characters (per text)

format string

text for plain text, any other value for the simplified Voyant markup

Returns:
Promise.<Array> -

a Promise for an Array of texts from the corpus

titles(config) → {Promise.<Array>}

Returns an array of document titles for the corpus.

The following are valid in the config parameter:

  • start: the zero-based start of the list
  • limit: a limit to the number of items to return at a time
  • docIndex: a zero-based list of documents (first document is zero, etc.); multiple documents can be separated by a comma
  • docId: a set of document IDs; multiple documents can be separated by a comma
  • query: one or more term queries for the title, author or full-text
  • sort: one of the following sort orders: INDEX, TITLE, AUTHOR, TOKENSCOUNTLEXICAL, TYPESCOUNTLEXICAL, TYPETOKENRATIOLEXICAL, PUBDATE
  • dir: sort direction, **ASC**ending or **DESC**ending

An example:

loadCorpus("austen").titles().then(titles => "The last work is: "+titles[titles.length-1])
Parameters:
Name Type Description
config Object

an Object specifying parameters (see list above)

Properties
Name Type Description
start number

the zero-based start of the list

limit number

a limit to the number of items to return at a time

docIndex number

a zero-based list of documents (first document is zero, etc.); multiple documents can be separated by a comma

docId string

a set of document IDs; multiple documents can be separated by a comma

query string

one or more term queries for the title, author or full-text

sort string

one of the following sort orders: INDEX, TITLE, AUTHOR, TOKENSCOUNTLEXICAL, TYPESCOUNTLEXICAL, TYPETOKENRATIOLEXICAL, PUBDATE

dir string

sort direction, **ASC**ending or **DESC**ending

Returns:
Promise.<Array> -

a Promise for an Array of document titles

toString()

An alias for summary.

tokens(config) → {Promise.<Array>}

Returns an array of document tokens.

The promise returns an array of document token objects. A document token object can look something like this:

	{
		"docId": "8a61d5d851a69c03c6ba9cc446713574",
		"docIndex": 0,
		"term": "LOVE",
		"tokenType": "lexical",
		"rawFreq": 54,
		"position": 0,
		"startOffset": 3,
		"endOffset": 7
	}

The following are valid in the config parameter:

  • start: the zero-based start index of the list (for paging)
  • limit: the maximum number of terms to provide per request
  • stopList: a list of stopwords to include (see stopwords tutorial)
  • whiteList: a keyword list – terms will be limited to this list
  • perDocLimit: the limit parameter is for the total number of terms returned, this parameter allows you to specify a limit value per document
  • noOthers: only include lexical forms, no other tokens
  • stripTags: one of the following: ALL, BLOCKSONLY, NONE (BLOCKSONLY tries to maintain blocks for line formatting)
  • withPosLemmas: include part-of-speech and lemma information when available (reliability of this may vary by instance)
  • docIndex: the zero-based index of the documents to include (use commas to separate multiple values)
  • docId: the document IDs to include (use commas to separate multiple values)

An example:

// load the first 20 tokens (don't include tags, spaces, etc.)
loadCorpus("austen").tokens({limit: 20, noOthers: true})
Parameters:
Name Type Description
config Object

an Object specifying parameters (see above)

Properties
Name Type Description
start number

the zero-based start index of the list (for paging)

limit number

the maximum number of terms to provide per request

stopList string

a list of stopwords to include (see stopwords tutorial)

whiteList string

a keyword list – terms will be limited to this list

perDocLimit number

the limit parameter is for the total number of terms returned, this parameter allows you to specify a limit value per document

noOthers boolean

only include lexical forms, no other tokens

stripTags string

one of the following: ALL, BLOCKSONLY, NONE (BLOCKSONLY tries to maintain blocks for line formatting)

withPosLemmas boolean

include part-of-speech and lemma information when available (reliability of this may vary by instance)

docIndex number

the zero-based index of the documents to include (use commas to separate multiple values)

docId string

the document IDs to include (use commas to separate multiple values)

Returns:
Promise.<Array> -

a Promise for an Array of document tokens

tool(tool, config) → {Promise.<string>}

Returns an HTML snippet that will produce the specified Voyant tools to appear.

In its simplest form we can simply call the named tool:

loadCorpus("austen").tool("Cirrus");

Each tool supports some options (that are summarized below), and those can be specified as options:

loadCorpus("austen").tool("Trends", {query: "love"});

There are also parameters (width, height, style, float) that apply to the actual tool window:

loadCorpus("austen").tool("Trends", {query: "love", style: "width: 500px; height: 500px"});

It's also possible to have several tools appear at once, though they won't be connected by events (clicking in a window won't modify the other windows):

loadCorpus("austen").tool("Cirrus", "Trends");

One easy way to get connected tools is to use the CustomSet tool and experiment with the layout:

loadCorpus("austen").tool("CustomSet", {tableLayout: "Cirrus,Trends", style: "width:800px; height: 500px"});

See the list of corpus tools for available tools and options.

Parameters:
Name Type Description
tool string

The tool to display

config Object

The config object for the tool

Returns:
Promise.<string>

async topics(config) → {Promise.<Object>}

Performs topic modelling using the latent Dirichlet allocation. Returns an object that has two primary properties:

  • topics: an array of topics (words organized into bunches of a specified size)
  • topicDocuments: an array of documents and their topic weights

Each topic in the topics array is an object with the following properties:

  • words: an array of the actual words that form the topic. Each word has the same properties as the topic, as well as a "word" property that contains the text content.
  • tokens
  • documentEntropy
  • wordLength
  • coherence
  • uniformDist
  • corpusDist
  • effNumWords
  • tokenDocDiff
  • rank1Docs
  • allocationRatio
  • allocationCount
  • exclusivity

Each document in the topicDocuments array is an object with the following properties:

  • docId: the document ID
  • weights: an array of the numbers corresponding to the the weight of each topic in this document

The config object as parameter can contain the following:

  • topics: the number of topics to get (default is 10)
  • termsPerTopic: the number of terms for each topic (default is 10)
  • iterations: the number of iterations to do, more iterations = more accurate (default is 100)
  • perDocLimit: the token limit per document, starting at the beginning of the document
  • seed: specify a particular seed to use for random number generation
  • stopList: a list of stopwords to include
Parameters:
Name Type Description
config Object

(see above)

Properties
Name Type Description
topics number

the number of topics to get (default is 10)

termsPerTopic number

the number of terms for each topic (default is 10)

iterations number

the number of iterations to do, more iterations = more accurate (default is 100)

perDocLimit number

specify a token limit per document, starting at the beginning of the document

seed number

specify a particular seed to use for random number generation

stopList string

a list of stopwords to include (see stopwords tutorial)

Returns:
Promise.<Object>

words(config) → {Promise.<Array>}

Returns an array of words from the corpus.

The array of words are in document order, much like tokens.

The following are valid in the config parameter:

  • start: the zero-based start index of the list (for paging)
  • limit: the maximum number of terms to provide per request
  • stopList: a list of stopwords to include (see stopwords tutorial)
  • whiteList: a keyword list – terms will be limited to this list
  • perDocLimit: the limit parameter is for the total number of terms returned, this parameter allows you to specify a limit value per document
  • docIndex: the zero-based index of the documents to include (use commas to separate multiple values)
  • docId: the document IDs to include (use commas to separate multiple values)

An example:

// load the first 20 words in the corpus
loadCorpus("austen").tokens({limit: 20})
Parameters:
Name Type Description
config Object

an Object specifying parameters (see above)

Properties
Name Type Description
start number

the zero-based start index of the list (for paging)

limit number

the maximum number of terms to provide per request

stopList string

a list of stopwords to include (see stopwords tutorial)

whiteList string

a keyword list – terms will be limited to this list

perDocLimit number

the limit parameter is for the total number of terms returned, this parameter allows you to specify a limit value per document

docIndex number

the zero-based index of the documents to include (use commas to separate multiple values)

docId string

the document IDs to include (use commas to separate multiple values)

Returns:
Promise.<Array> -

a Promise for an Array of words

static load(config, api) → {Promise.<Corpus>}

Load a Corpus using the provided config and api

Parameters:
Name Type Description
config Spyral.Corpus~CorpusConfig

the Corpus config

api Object

any additional API values

Returns:
Promise.<Corpus>

Type Definitions

CorpusConfig

The Corpus config

Properties:
Name Type Description
corpus String

The ID of a previously created corpus.

A corpus ID can be used to try to retrieve a corpus that has been previously created. Typically the corpus ID is used as a first string argument, with an optional second argument for other parameters (especially those to recreate the corpus if needed).

loadCorpus("goldbug");

loadCorpus("goldbug", {
	// if corpus ID "goldbug" isn't found, use the input
	input: "https://gist.githubusercontent.com/sgsinclair/84c9da05e9e142af30779cc91440e8c1/raw/goldbug.txt",
	inputRemoveUntil: 'THE GOLD-BUG',
	inputRemoveFrom: 'FOUR BEASTS IN ONE'
});
input String | Array.<String>

Input sources for the corpus.

The input sources can be either normal text or URLs (starting with http).

Typically input sources are specified as a string or an array in the first argument, with an optional second argument for other parameters.

loadCorpus("Hello Voyant!"); // one document with this string

loadCorpus(["Hello Voyant!", "How are you?"]); // two documents with these strings

loadCorpus("http://hermeneuti.ca/"); // one document from URL

loadCorpus(["http://hermeneuti.ca/", "https://en.wikipedia.org/wiki/Voyant_Tools"]); // two documents from URLs

loadCorpus("Hello Voyant!", "http://hermeneuti.ca/"]); // two documents, one from string and one from URL

loadCorpus("https://gist.githubusercontent.com/sgsinclair/84c9da05e9e142af30779cc91440e8c1/raw/goldbug.txt", {
	inputRemoveUntil: 'THE GOLD-BUG',
	inputRemoveFrom: 'FOUR BEASTS IN ONE'
});

// use a corpus ID but also specify an input source if the corpus can't be found
loadCorpus("goldbug", {
	input: "https://gist.githubusercontent.com/sgsinclair/84c9da05e9e142af30779cc91440e8c1/raw/goldbug.txt",
	inputRemoveUntil: 'THE GOLD-BUG',
	inputRemoveFrom: 'FOUR BEASTS IN ONE'
});
inputFormat String

The input format of the corpus (the default is to auto-detect).

The auto-detect format is usually reliable and inputFormat should only be used if the default behaviour isn't desired. Most of the relevant values are used for XML documents:

  • DTOC: Dynamic Table of Contexts XML format
  • HTML: Hypertext Markup Language
  • RSS: Really Simple Syndication XML format
  • TEI: Text Encoding Initiative XML format
  • TEICORPUS: Text Encoding Initiative Corpus XML format
  • TEXT: plain text
  • XML: treat the document as XML (sometimes overridding auto-detect of XML vocabularies like RSS and TEI)

Other formats include PDF, MSWORD, XLSX, RTF, ODT, and ZIP (but again, these rarely need to be specified).

tableDocuments String

Determine what is a document in a table (the entire table, by row, by column); only used for table-based documents.

Possible values are:

  • undefined or blank (default): the entire table is one document
  • rows: each row of the table is a separate document
  • columns: each column of the table is a separate document

See also Creating a Corpus with Tables.

tableContent String

Determine how to extract body content from the table; only used for table-based documents.

Columns are referred to by numbers, the first is column 1 (not 0). You can specify separate columns by using a comma or you can combined the contents of columns/cells by using a plus sign.

Some examples:

  • 1: use column 1
  • 1,2: use columns 1 and 2 separately
  • 1+2,3: combine columns 1 and two and use column 3 separately

See also Creating a Corpus with Tables.

tableAuthor String

Determine how to extract the author from each document; only used for table-based documents.

Columns are referred to by numbers, the first is column 1 (not 0). You can specify separate columns by using a comma or you can combined the contents of columns/cells by using a plus sign.

Some examples:

  • 1: use column 1
  • 1,2: use columns 1 and 2 separately
  • 1+2,3: combine columns 1 and two and use column 3 separately

See also Creating a Corpus with Tables.

tableTitle String

Determine how to extract the title from each document; only used for table-based documents.

Columns are referred to by numbers, the first is column 1 (not 0). You can specify separate columns by using a comma or you can combined the contents of columns/cells by using a plus sign.

Some examples:

  • 1: use column 1
  • 1,2: use columns 1 and 2 separately
  • 1+2,3: combine columns 1 and two and use column 3 separately

See also Creating a Corpus with Tables.

tableGroupBy String

Specify a column (or columns) by which to group documents; only used for table-based documents, in rows mode.

Columns are referred to by numbers, the first is column 1 (not 0). You can specify separate columns by using a comma or you can combined the contents of columns/cells by using a plus sign.

Some examples:

  • 1: use column 1
  • 1,2: use columns 1 and 2 separately
  • 1+2,3: combine columns 1 and two and use column 3 separately

See also Creating a Corpus with Tables.

tableNoHeadersRow String

Determine if the table has a first row of headers; only used for table-based documents.

Provide a value of "true" if there is no header row, otherwise leave it blank or undefined (default).

See also Creating a Corpus with Tables.

tokenization String

The tokenization strategy to use

This should usually be undefined, unless specific behaviour is required. These are the valid values:

  • undefined or blank: use the default tokenization (which uses Unicode rules for word segmentation)
  • wordBoundaries: use any Unicode character word boundaries for tokenization
  • whitespace: tokenize by whitespace only (punctuation and other characters will be kept with words)

See also Creating a Corpus Tokenization.

xmlContentXpath String

The XPath expression that defines the location of document content (the body); only used for XML-based documents.

loadCorpus("<doc><head>Hello world!</head><body>This is Voyant!</body></doc>", {
	xmlContentXpath: "//body"
}); // document would be: "This is Voyant!"

See also Creating a Corpus with XML.

xmlTitleXpath String

The XPath expression that defines the location of each document's title; only used for XML-based documents.

loadCorpus("<doc><title>Hello world!</title><body>This is Voyant!</body></doc>", {
	xmlTitleXpath: "//title"
}); // title would be: "Hello world!"

See also Creating a Corpus with XML.

xmlAuthorXpath String

The XPath expression that defines the location of each document's author; only used for XML-based documents.

loadCorpus("<doc><author>Stéfan Sinclair</author><body>This is Voyant!</body></doc>", {
	xmlAuthorXpath: "//author"
}); // author would be: "Stéfan Sinclair"

See also Creating a Corpus with XML.

xmlPubPlaceXpath String

The XPath expression that defines the location of each document's publication place; only used for XML-based documents.

loadCorpus("<doc><pubPlace>Montreal</pubPlace><body>This is Voyant!</body></doc>", {
	xmlPubPlaceXpath: "//pubPlace"
}); // publication place would be: "Montreal"

See also Creating a Corpus with XML.

xmlPublisherXpath String

The XPath expression that defines the location of each document's publisher; only used for XML-based documents.

loadCorpus("<doc><publisher>The Owl</publisher><body>This is Voyant!</body></doc>", {
	xmlPublisherXpath: "//publisher"
}); // publisher would be: "The Owl"

See also Creating a Corpus with XML.

xmlKeywordXpath String

The XPath expression that defines the location of each document's keywords; only used for XML-based documents.

loadCorpus("<doc><keyword>text analysis</keyword><body>This is Voyant!</body></doc>", {
	xmlKeywordXpath: "//keyword"
}); // publisher would be: "text analysis"

See also Creating a Corpus with XML.

xmlCollectionXpath String

The XPath expression that defines the location of each document's collection name; only used for XML-based documents.

loadCorpus("<doc><collection>documentation</collection><body>This is Voyant!</body></doc>", {
	xmlCollectionXpath: "//collection"
}); // publisher would be: "documentation"

See also Creating a Corpus with XML.

xmlDocumentsXpath String

The XPath expression that defines the location of each document; only used for XML-based documents.

See also Creating a Corpus with XML.

xmlGroupByXpath String

The XPath expression by which to group multiple documents; only used for XML-based documents.

loadCorpus("<doc><sp s='Juliet'>Hello!</sp><sp s='Romeo'>Hi!</sp><sp s='Juliet'>Bye!</sp></doc>", {
	xmlDocumentsXpath: '//sp',
	xmlGroupByXpath: "//@s"
}); // two docs: "Hello! Bye!" (Juliet) and "Hi!" (Romeo)

See also Creating a Corpus with XML.

xmlExtraMetadataXpath String

A value that defines the location of other metadata; only used for XML-based documents.

loadCorpus("<doc><tool>Voyant</tool><phase>1</phase><body>This is Voyant!</body></doc>", {
	xmlExtraMetadataXpath: "tool=//tool\nphase=//phase"
}); // tool would be "Voyant" and phase would be "1"

Note that xmlExtraMetadataXpath is a bit different from the other XPath expressions in that it's possible to define multiple values (each on its own line) in the form of name=xpath.

See also Creating a Corpus with XML.

xmlExtractorTemplate String

Pass the XML document through the XSL template located at the specified URL before extraction (this is ignored in XML-based documents).

This is an advanced parameter that allows you to define a URL of an XSL template that can be called before text extraction (in other words, the other XML-based parameters apply after this template has been processed).

inputRemoveUntil String

Omit text up until the start of the matching regular expression (this is ignored in XML-based documents).

loadCorpus("Hello world! This is Voyant!", {
	inputRemoveUntil: "This"
}); // document would be: "This is Voyant!"

See also Creating a Corpus with Text.

inputRemoveUntilAfter String

Omit text up until the end of the matching regular expression (this is ignored in XML-based documents).

loadCorpus("Hello world! This is Voyant!", {
	inputRemoveUntilAfter: "world!"
}); // document would be: "This is Voyant!"

See also Creating a Corpus with Text.

inputRemoveFrom String

Omit text from the start of the matching regular expression (this is ignored in XML-based documents).

loadCorpus("Hello world! This is Voyant!", {
	inputRemoveFrom: "This"
}); // document would be: "Hello World!"

See also Creating a Corpus with Text.

inputRemoveFromAfter String

Omit text from the end of the matching regular expression (this is ignored in XML-based documents).

loadCorpus("Hello world! This is Voyant!", {
	inputRemoveFromAfter: "This"
}); // document would be: "Hello World! This"

See also Creating a Corpus with Text.

subTitle String

A sub-title for the corpus.

This is currently not used, except in the Dynamic Table of Contexts skin. Still, it may be worth specifying a subtitle for later use.

title String

A title for the corpus.

This is currently not used, except in the Dynamic Table of Contexts skin. Still, it may be worth specifying a title for later use.

curatorTsv String

a simple TSV of paths and labels for the DToC interface (this isn't typically used outside of the specialized DToC context).

The DToC skin allows curation of XML tags and attributes in order to constrain the entries shown in the interface or to provide friendlier labels. This assumes plain text unicode input with one definition per line where the simple XPath expression is separated by a tab from a label.

 p    	 paragraph
 ref[@target*="religion"]    	 religion

For more information see the DToC documentation on Curating Tags