foo 1 U.S. 1
' plain_text = clean_text(text, ['html', 'inline_whitespace', my_func]) found_citations = get_citations(plain_text) See the `Annotating Citations <#annotating-citations>`_ section for how to insert links into the original text using citations extracted from the cleaned text. :code:`clean_text` currently accepts these values as cleaners: 1. :code:`inline_whitespace`: replace all runs of tab and space characters with a single space character 2. :code:`all_whitespace`: replace all runs of any whitespace character with a single space character 3. :code:`underscores`: remove two or more underscores, a common error in text extracted from PDFs 4. :code:`html`: remove non-visible HTML content using the lxml library 5. Custom function: any function taking a string and returning a string. Annotating Citations -------------------- For simple plain text, you can insert links to citations using the :code:`annotate` function: :: from eyecite import get_citations, annotate plain_text = 'bob lissner v. test 1 U.S. 12, 347-348 (4th Cir. 1982)' citations = get_citations(plain_text) linked_text = annotate(plain_text, [[c.span(), "", ""] for c in citations]) returns: 'bob lissner v. test 1 U.S. 12, 347-348 (4th Cir. 1982)' Each citation returned by get_citations keeps track of where it was found in the source text. As a result, :code:`annotate` must be called with the *same* cleaned text used by :code:`get_citations` to extract citations. If you do not, the offsets returned by the citation's :code:`span` method will not align with the text, and your annotations will be in the wrong place. If you want to clean text and then insert annotations into the original text, you can pass the original text in as :code:`source_text`: :: from eyecite import get_citations, annotate, clean_text source_text = 'bob lissner v. test 1 U.S. 12, 347-348 (4th Cir. 1982)
' plain_text = clean_text(source_text, ['html', 'inline_whitespace']) citations = get_citations(plain_text) linked_text = annotate(plain_text, [[c.span(), "", ""] for c in citations], source_text=source_text) returns: 'bob lissner v. test 1 U.S. 12, 347-348 (4th Cir. 1982)
' The above example extracts citations from :code:`plain_text` and applies them to :code:`source_text`, using a diffing algorithm to insert annotations in the correct locations in the original text. Wrapping HTML Tags ^^^^^^^^^^^^^^^^^^ Note that the above example includes mismatched HTML tags: "1 U.S. 12". To specify handling for unbalanced tags, use the :code:`unbalanced_tags` parameter: * :code:`unbalanced_tags="skip"`: annotations that would result in unbalanced tags will not be inserted. * :code:`unbalanced_tags="wrap"`: unbalanced tags will be wrapped, resulting in :code:`1 U.S. 12` Important: :code:`unbalanced_tags="wrap"` uses a simple regular expression and will only work for HTML where angle brackets are properly escaped, such as the HTML emitted by :code:`lxml.html.tostring`. It is intended for regularly formatted documents such as case text published by courts. It may have unpredictable results for deliberately-constructed challenging inputs such as citations containing partial HTML comments or :code:`` tags.
Customizing Annotation
^^^^^^^^^^^^^^^^^^^^^^
If inserting text before and after isn't sufficient, supply a callable under the :code:`annotator` parameter
that takes :code:`(before, span_text, after)` and returns the annotated text:
::
def annotator(before, span_text, after):
return before + span_text.lower() + after
linked_text = annotate(plain_text, [[c.span(), "", ""] for c in citations], annotator=annotator)
returns:
'bob lissner v. test 1 u.s. 12, 347-348 (4th Cir. 1982)'
Resolving Citations
-------------------
Once you have extracted citations from a document, you may wish to resolve them to their common references.
To do so, just pass the results of :code:`get_citations()` into :code:`resolve_citations()`. This function will
do its best to resolve each "full," "short form," "supra," and "id" citation to a common :code:`Resource` object,
returning a dictionary that maps resources to lists of associated citations:
::
from eyecite import get_citations, resolve_citations
text = 'first citation: 1 U.S. 12. second citation: 2 F.3d 2. third citation: Id.'
found_citations = get_citations(text)
resolved_citations = resolve_citations(found_citations)
returns (pseudo):
{
: [FullCaseCitation('1 U.S. 12')],
: [FullCaseCitation('2 F.3d 2'), IdCitation('Id.')]
}
Importantly, eyecite performs these resolutions using only its immanent knowledge about each citation's
textual representation. If you want to perform more sophisticated resolution (e.g., by augmenting each
citation with information from a third-party API), simply pass custom :code:`resolve_id_citation()`,
:code:`resolve_supra_citation()`, :code:`resolve_shortcase_citation()`, and :code:`resolve_full_citation()`
functions to :code:`resolve_citations()` as keyword arguments. You can also configure those functions to
return a more complex resource object (such as a Django model), so long as that object inherits the
:code:`eyecite.models.ResourceType` type (which simply requires hashability). For example, you might implement
a custom full citation resolution function as follows, using the default resolution logic as a fallback:
::
def my_resolve(full_cite):
# special handling for resolution of known cases in our database
resource = MyOpinion.objects.get(full_cite)
if resource:
return resource
# allow normal clustering of other citations
return resolve_full_citation(full_cite)
resolve_citations(citations, resolve_full_citation=my_resolve)
returns (pseudo):
{
: [, , ],
: [, ],
}
Tokenizers
----------
Internally, eyecite works by applying a list of regular expressions to the source text to convert it to a list
of tokens:
::
In [1]: from eyecite.tokenizers import default_tokenizer
In [2]: list(default_tokenizer.tokenize("Foo v. Bar, 123 U.S. 456 (2016). Id. at 457."))
Out[2]:
['Foo',
StopWordToken(data='v.', ...),
'Bar,',
CitationToken(data='123 U.S. 456', volume='123', reporter='U.S.', page='456', ...),
'(2016).',
IdToken(data='Id.', ...),
'at',
'457.']
Tokens are then scanned to determine values like the citation year or case name for citation resolution.
Alternate tokenizers can be substituted by providing a tokenizer instance to :code:`get_citations()`:
::
from eyecite.tokenizers import HyperscanTokenizer
hyperscan_tokenizer = HyperscanTokenizer(cache_dir='.hyperscan')
cites = get_citations(text, tokenizer=hyperscan_tokenizer)
test_FindTest.py includes a simplified example of using a custom tokenizer that uses modified
regular expressions to extract citations with OCR errors.
eyecite ships with two tokenizers:
AhocorasickTokenizer (default)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The default tokenizer uses the pyahocorasick library to filter down eyecite's list of
extractor regexes. It then performs extraction using the builtin :code:`re` library.
HyperscanTokenizer
^^^^^^^^^^^^^^^^^^
The alternate HyperscanTokenizer compiles all extraction regexes into a hyperscan database
so they can be extracted in a single pass. This is far faster than the default tokenizer
(exactly how much faster depends on how many citation formats are included in the target text),
but requires the optional :code:`hyperscan` dependency that has limited platform support.
See the "Installation" section for hyperscan installation instructions and limitations.
Compiling the hyperscan database takes several seconds, so short-running scripts may want to
provide a cache directory where the database can be stored. The directory should be writeable
only by the user:
::
hyperscan_tokenizer = HyperscanTokenizer(cache_dir='.hyperscan')
Debugging
---------
If you want to see what metadata eyecite is able to extract for each citation, you can use :code:`dump_citations`.
This is primarily useful for developing eyecite, but may also be useful for exploring what data is available to you::
In [1]: from eyecite import dump_citations, get_citations
In [2]: text="Mass. Gen. Laws ch. 1, § 2. Foo v. Bar, 1 U.S. 2, 3-4 (1999). Id. at 3. Foo, supra, at 5."
In [3]: cites=get_citations(text)
In [4]: print(dump_citations(get_citations(text), text))
FullLawCitation: Mass. Gen. Laws ch. 1, § 2. Foo v. Bar, 1 U.S. 2, 3-4 (1
* groups
* reporter='Mass. Gen. Laws'
* chapter='1'
* section='2'
FullCaseCitation: Laws ch. 1, § 2. Foo v. Bar, 1 U.S. 2, 3-4 (1999). Id. at 3. Foo, s
* groups
* volume='1'
* reporter='U.S.'
* page='2'
* metadata
* pin_cite='3-4'
* year='1999'
* court='scotus'
* plaintiff='Foo'
* defendant='Bar,'
* year=1999
IdCitation: v. Bar, 1 U.S. 2, 3-4 (1999). Id. at 3. Foo, supra, at 5.
* metadata
* pin_cite='at 3'
SupraCitation: 2, 3-4 (1999). Id. at 3. Foo, supra, at 5.
* metadata
* antecedent_guess='Foo'
* pin_cite='at 5'
In the real terminal, the :code:`span()` of each extracted citation will be highlighted.
You can use the :code:`context_chars=30` parameter to control how much text is shown before and after.
Installation
============
Installing eyecite is easy.
::
poetry add eyecite
Or via pip::
pip install eyecite
Or install the latest dev version from github::
pip install https://github.com/freelawproject/eyecite/archive/main.zip#egg=eyecite
Hyperscan installation
----------------------
To use :code:`HyperscanTokenizer` you must additionally install the python `hyperscan