README for dta-tokwrap - programs, scripts, and perl modules for DTA XML corpus tokenization
This package contains various utilities for tokenization of DTA "base-format" XML documents. see "INSTALLATION" for requirements and installation instructions, see "USAGE" for a brief introduction to the high-level command-line interface, and see "TOOLS" for an overview of the individual tools included in this distribution.
tested version(s): 1.95.8, 2.0.1
tested version(s): 2.7.3, 2.7.8
tested version(s): 1.1.24, 1.1.26
See DTA-TokWrap/README.txt for a full list of required perl modules.
tested version(s): gcc / linux: v4.3.3, 4.4.6
tested version(s): 2.5.33, 2.5.35
Only needed if you plan on making changes to the lexer sources.
tested version(s): 2.61, 2.67
Required for building from SVN sources.
tested version(s): 1.9.6, 1.11.1
Required for building from SVN sources.
To build this package from SVN sources, you must first run the shell command:
bash$ sh ./autoreconf.shfrom the distribution root directory BEFORE running ./configure. Building from SVN sources requires additional development tools to present on the build system. Then, follow the instructions in "Building from Source".
To build and install the entire package, issue the following commands to the shell:
bash$ cd dta-tokwrap-0.01 # (or wherever you unpacked this distribution)
bash$ sh ./configure # configure the package
bash$ make # build the package
bash$ make install # install the package on your systemMore details on the top-level installation process can be found in the file INSTALL in the distribution root directory.
More details on building and installing the DTA::TokWrap perl module included in this distribution can be found in the perlmodinstall(1) manpage.
The perl program dta-tokwrap.perl installed from the DTA-TokWrap/ distribution subdirectory provides a flexible high-level command-line interface to the tokenization of DTA XML documents.
The dta-tokwrap.perl script takes as its input DTA "base-format" XML files, which are simply (TEI-conformant) UTF-8 encoded XML files with one <c> element per character:
the document MUST be encoded in UTF-8,
all text nodes to be tokenized should be descendants of a <text> element, and may optionally be immediate daughters of a <c> element (XPath //text//text()|//text//c/text()). <c> elements may not be nested.
Prior to dta-tokwrap v0.38, <c> elements were required.
Assume we wish to tokenize a single DTA "base-format" XML file doc1.xml. Issue the following command to the shell:
bash$ dta-tokwrap.perl doc1.xml... This will create the following output files:
"Master" tokenizer output file encoding sentence boundaries, token boundaries, and tokenizer-provided token analyses. Source for various stand-off annotation formats. This format can also be passed directly to and from the DTA::CAB(3pm) analysis suite using the DTA::CAB::Format::XmlNative(3pm) formatter class.
Assume we wish to tokenize a corpus of three DTA "base-format" XML files doc1.xml, doc2.xml, and doc3.xml. This is as easy as:
bash$ dta-tokwrap.perl doc1.xml doc2.xml doc3.xmlFor each input document specified on the command line, master output files and stand-off annotation files will be created.
See "the dta-tokwrap.perl manpage" for more details.
Assume we wish to tokenize a large corpus of XML input files doc*.xml, and would like to have some feedback on the progress of the tokenization process. Try:
bash$ dta-tokwrap.perl -verbose=1 doc*.xmlor:
bash$ dta-tokwrap.perl -verbose=2 doc*.xmlor even:
bash$ dta-tokwrap.perl -traceAll doc*.xmlAssume we have a TEI-like document doc.tei.xml which we want to encode as TCF to the file doc.tei.tcf, using only whitespace tokenizer "hints", but not actually tokenizing the document yet. This can be accomplished by:
$ dta-tokwrap.perl -t=tei2tcf -weak-hints doc1.tei.xmlIf the output should instead be written to STDOUT, just call:
$ dta-tokwrap.perl -t=tei2tcf -weak-hints -dO=tcffile=- doc1.tei.xmlAssume that the resulting TCF document has undergone further processing (e.g. via WebLicht) to produce an annotated TCF document doc.out.tcf.
selected TCF layers (in particular the tokens and sentences layers) can be spliced back into the TEI document as doc.out.xml by calling:
$ dta-tokwrap.perl -t=tcf2tei doc.out.tcf -dO=tcffile=doc.out.tcf -dO=tcfcwsfile=doc.out.xmlThis section provides a brief overview of the individual tools included in the dta-tokwrap distribution.
The perl scripts and programs included with this distribution are installed by default in /usr/local/bin and/or wherever your perl installs scripts by default (e.g. in `perl -MConfig -e 'print $Config{installsitescript}'`).
Top-level wrapper script for document tokenization using the DTA::TokWrap perl API.
Script to insert <c> elements and/or xml:id attributes for such elements into an XML document which does not yet contain them. Guaranteed not to clobber any existing //c IDs.
//c/@xml:id attributes are generated by a simple document-global counter ("c1", "c2", ..., "c65536").
See "the dtatw-add-c.perl manpage" for more details.
Script to convert //c/@xml:id attributes to page-local encoding. Never really used.
See "the dtatw-cids2local.perl manpage" for more details.
Script to splice <s> and <w> elements encoded from a standoff (.t.xml or .u.xml) XML file into the original "base-format" (.chr.xml) file, producing a .cws.xml file. A tad too generous with partial word segments, due to strict adjacency and boundary criteria.
In earlier versions of dta-tokwrap, this functionality was split between the scripts dtatw-add-w.perl and dtatw-add-s.perl, which required only an id-compatible base-format (.chr.xml) file as the splice target. As of dta-tokwrap v0.35, the splice target base-format file must be original source file itself, since the current implementation uses byte offsets to perform the splice.
See "the dtatw-add-ws.perl manpage" for more details.
Script to splice generic standoff attributes and/or content into a base file; useful e.g. for merging flat DTA::CAB standoff analyses into TEI-structured *.cws.xml files.
See "the dtatw-splice.perl manpage" for more details.
Script to insert DDC-relevant attributes extracted from a base file into a *.t.xml file, producing a pre-DDC XML format file (by convention *.ddc.t.xml, a subset of the *.t.xml format).
See "the dtatw-get-ddc-attrs.perl manpage" for more details.
Simple script to extract a single header element from an XML file (e.g. for later inclusion in a DDC XML format file).
See "the dtatw-get-header.perl manpage" for more details.
See "the dtatw-get-header.perl manpage" for more details.
Script to conver insert <p>...</p> wrappers for //s/@pn key attributes in "flat" *.t.xml files.
Script to convert *.ddc.t.xml files and optional headers to DDC-XML format.
See "the dtatw-xml2ddc.perl manpage" for more details.
Simple script to check consistency of tokenizer output (*.t) offset + length fields with input (*.txt) file.
Script to add <c> elements to an XML document which does not already contain them. Not really useful as of dta-tokwrap v0.38.
Script to remove <c> elements from an XML document. Regex hack, fast but not exceedingly robust, use with caution. See also "dtatw-rm-c.xsl"
Fast regex hack to remove <w> elements from an XML document
Fast regex hack to remove <s> elements from an XML document.
Script to remove <lb> (line-break) elements from an XML document, replacing them with newlines. Regex hack, fast but not robust, use with caution. See also "dtatw-rm-lb.xsl"
Encodes newlines under //text//text() in an XML document as <lb> (line-break) elements using high-level file heuristics only. Regex hack, fast but not robust, use with caution. See also "dtatw-ensure-lb.perl", "dtatw-add-lb.xsl", "dtatw-rm-lb.perl".
Script to ensure that all //text//text() newlines in an XML document are explicitly encoded with <lb> (line-break) elements, using optional file-, element-, and line-level heuristics. Robust but slow, since it actually parses XML input documents. See also "dtatw-lb-encode.perl", "dtatw-add-lb.xsl", "dtatw-rm-lb.perl".
Script to apply a type-"dictionary" in one-word-per-line (.tt) format to a token corpus in one-word-per-line (.tt) format. Especially useful together with standard UNIX utilities such as cut, grep, sort, and uniq.
Script to convert DTA::CAB::Format::TT (one-word-per-line with variable analysis fields identified by conventional prefixes) files to expanded .t.xml format used by dta-tokwrap. The expanded format should be identical to that used by the DTA::CAB::Format::Xml class. See also dtatw-txml2tt.xsl.
Script to extract a portion of a file, specified by byte offset and length. Useful for debugging index files created by other tools.
The distribution directory make/ contains a "template" for using GNU make to organizing the conversion of large corpora with the dta-tokwrap utilities. This is useful because:
make's intuitive, easy-to-read syntax provides a wonderful vehicle for user-defined configuration files, obviating the need to remember the names of all 64 (at last count) dta-tokwrap.perl|/dta-tokwrap.perl options,
make is very good at tracking complex dependencies of the sort that exist between the various temporary files generated by the dta-tokwrap utilities,
make jobs can be made "robust" simply by adding a -k (--keep-going) to the command-line, and
last but certainly not least, make has built-in support for parallelization of complex tasks by means of the -j N (--jobs=N) option, allowing us to take advantage of multiprocessor systems.
By default, the contents of the distribution make/ subdirectory are installed to /usr/local/share/dta-tokwrap/make/. See the comments at the top of make/User.mak for instructions.
Top-level tokenization-wrapper module, used by dta-tokwrap.perl.
Object-oriented wrapper for documents to be processed.
Abstract base class for elementary document-processing operations.
See the DTA::TokWrap::Intro(3pm) manpage for more details on included modules, APIs, calling conventions, etc.
The XSL stylesheets included with this distribution are installed by default in /usr/local/share/dta-tokwrap/stylesheets.
Replaces newlines with <lb/> elements in input document.
Assigns missing //c/@xml:id attributes using the XSL generate-id() function.
Removes <c> elements from the input document. Slow but robust.
Replaces <lb/> elements with newlines.
Converts "master" tokenized XML output format (*.t.xml) to TAB-separated one-word-per-line format (*.mr.t aka *.t aka *.tt aka "tt" aka "CSV" aka DTA::CAB::Format::TT aka "TnT" aka "TreeTagger" aka "vertical" aka "moot-native" aka ...). See the mootfiles(5) manpage for basic format details, and see the top of the XSL script for some influential transformation parameters.
Several C programs are included with the distribution. These are used by the dta-tokwrap.perl script to perform various intermediate document processing operations, and should not need to be called by the user directly.
Caveat Scriptor: The following programs are meant for internal use by the DTA::TokWrap modules only, and their names, calling conventions, and very presence is subject to change without notice.
Splits input document doc.xml into a "character index" doc.cx (CSV), a "structural index" doc.sx (XML), and a "text index" doc.tx (UTF-8 text).
Removes namespaces from any XML document by renaming "xmlns" attributes to "xmlns_" and "xmlns:*" attributes to "xmlns_*". Useful because XSL's namespace handling is annoyingly slow and ugly.
Dummy flex tokenizer. Useful for testing.
Converts "master" tokenized XML output format (*.t.xml) to sentence-level stand-off XML format (*.s.xml).
Converts "master" tokenized XML output format (*.t.xml) to token-level stand-off XML format (*.w.xml).
Converts "master" tokenized XML output format (*.t.xml) to token-analysis-level stand-off XML format (*.a.xml).
perl(1).
Bryan Jurish <moocow@cpan.org>
Copyright (C) 2009-2018 by Bryan Jurish
This package is free software. Redistribution and modification of C portions of this package are subject to the terms of the version 3 or greater of the GNU Lesser General Public License; see the files COPYING and COPYING.LESSER which came with the distribution for details.
Redistribution and/or modification of the Perl portions of this package are subject to the same terms as Perl itself, either Perl version 5.24.1 or, at your option, any later version of Perl 5 you may have available.