Generic logic for checking if there are updates

This implements the logic described in Section 5.1, "The client application", of the TUF spec. It checks which of the server metadata has changed, and downloads all changed metadata to the local cache. (Metadata here refers both to the TUF security metadata as well as the Hackage packge index.)

You should pass Nothing for the UTCTime _only_ under exceptional circumstances (such as when the main server is down for longer than the expiry dates used in the timestamp files on mirrors).

Download a package

Variation on downloadPackage that takes a FilePath instead.

Index directory

Entry into the Hackage index.

Read the Hackage index directory

Should only be called after checkForUpdates.

Files that we might request from the index

The type index tells us the type of the decoded file, if any. For files for which the library does not support decoding this will be (). NOTE: Clients should NOT rely on this type index being (), or they might break if we add support for parsing additional file formats in the future.

TODO: If we wanted to support legacy Hackage, we should also have a case for the global preferred-versions file. But supporting legacy Hackage will probably require more work anyway..

Entry from the Hackage index; see withIndex.

Various operations that we can perform on the index once its open

Note that IndexEntry contains a fields both for the raw file contents and the parsed file contents; clients can choose which to use.

In principle these callbacks will do verification (once we have implemented author signing). Right now they don't need to do that, because the index as a whole will have been verified.

Look up entries in the Hackage index

This is in withFile style so that clients can efficiently look up multiple files from the index.

Should only be called after checkForUpdates.

Check if we need to bootstrap (i.e., if we have root info)

Bootstrap the chain of trust

New clients might need to obtain a copy of the root metadata. This however represents a chicken-and-egg problem: how can we verify the root metadata we downloaded? The only possibility is to be provided with a set of an out-of-band set of root keys and an appropriate threshold.

Clients who provide a threshold of 0 can do an initial "unsafe" update of the root information, if they wish.

The downloaded root information will _only_ be verified against the provided keys, and _not_ against previously downloaded root info (if any). It is the responsibility of the client to call bootstrap only when this is the desired behaviour.

File length

Having verified file length information means we can protect against endless data attacks and similar.

File hash

Key threshold

The key threshold is the minimum number of keys a document must be signed with. Key thresholds are specified in RoleSpec or DelegationsSpec.

File information

This intentionally does not have an Eq instance; see knownFileInfoEqual and verifyFileInfo instead.

NOTE: Throughout we compute file information always over the raw bytes. For example, when timestamp.json lists the hash of snapshot.json, this hash is computed over the actual snapshot.json file (as opposed to the canonical form of the embedded JSON). This brings it in line with the hash computed over target files, where that is the only choice available.

File hash

Compute FileInfo

TODO: Currently this will load the entire input bytestring into memory. We need to make this incremental, by computing the length and all hashes in a single traversal over the input.

Compute FileInfo

Compare the expected trusted file info against the actual file info of a target file.

This should be used only when the FileInfo is already known. If we want to compare known FileInfo against a file on disk we should delay until we have confirmed that the file lengths match (see downloadedVerify).

Extract SHA256 hash from FileInfo (if present)

54-bit integer values

JavaScript can only safely represent numbers between -(2^53 - 1) and 2^53 - 1.

TODO: Although we introduce the type here, we don't actually do any bounds checking and just inherit all type class instance from Int64. We should probably define fromInteger to do bounds checking, give different instances for type classes such as Bounded and FiniteBits, etc.

File version

The file version is a flat integer which must monotonically increase on every file update.

Show and Read instance are defined in terms of the underlying Int (this is use for example by hackage during the backup process).

File expiry date

A Nothing value here means no expiry. That makes it possible to set some files to never expire. (Note that not having the Maybe in the type here still allows that, because you could set an expiry date 2000 years into the future. By having the Maybe here we avoid the _need_ for such encoding issues.)

Occassionally it is useful to read only a header from a file.

HeaderOnly intentionally only has a FromJSON instance (no ToJSON).

Location of the various files we cache

Although the generic TUF algorithms do not care how we organize the cache, we nonetheless specity this here because as long as there are tools which access files in the cache directly we need to define the cache layout. See also comments for defaultCacheLayout.

The cache layout cabal-install uses

We cache the index as cache/00-index.tar; this is important because `cabal-install` expects to find it there (and does not currently go through the hackage-security library to get files from the index).

Layout of the files within the index tarball

Files that we might request from the index

The type index tells us the type of the decoded file, if any. For files for which the library does not support decoding this will be (). NOTE: Clients should NOT rely on this type index being (), or they might break if we add support for parsing additional file formats in the future.

TODO: If we wanted to support legacy Hackage, we should also have a case for the global preferred-versions file. But supporting legacy Hackage will probably require more work anyway..

The layout of the index as maintained on Hackage

Layout of a repository

The layout used on Hackage

Layout used by cabal for ("legacy") local repos

Obviously, such repos do not normally contain any of the TUF files, so their location is more or less arbitrary here.

Definition of a mirror

NOTE: Unlike the TUF specification, we require that all mirrors must have the same format. That is, we omit metapath and targetspath.

Full versus partial mirrors

The TUF spec explicitly allows for partial mirrors, with the mirrors file specifying (through patterns) what is available from partial mirrors.

For now we only support full mirrors; if we wanted to add partial mirrors, we would add a second MirrorPartial constructor here with arguments corresponding to TUF's metacontent and targetscontent fields.

Give a human-readable description of a particular mirror

(for use in error messages)

The root of the repository

Repository roots can be anchored at a remote URL or a local directory.

Note that even for remote repos RepoRoot is (potentially) different from Web -- for a repository located at, say, http://hackage.haskell.org they happen to coincide, but for one location at http://example.com/some/subdirectory they do not.

Paths relative to the root of the repository

The root of the index tarball

Paths relative to the root of the index tarball

The cache directory

Anchor a cache path to the location of the cache

The root metadata

NOTE: We must have the invariant that ALL keys (apart from delegation keys) must be listed in rootKeys. (Delegation keys satisfy a similar invariant, see Targets.)

Role specification

The phantom type indicates what kind of type this role is meant to verify.

A list of signatures

Invariant: each signature must be made with a different key. We enforce this invariant for incoming untrusted data (fromPreSignatures) but not for lists of signatures that we create in code.

Create a new document without any signatures

Sign a document

Variation on withSignatures that doesn't need the repo layout

Construct signatures for already rendered value

General FromJSON instance for signed datatypes

We don't give a general FromJSON instance for Signed because for some datatypes we need to do something special (datatypes where we need to read key environments); for instance, see the "Signed Root" instance.

Signature verification

NOTES: 1. By definition, the signature must be verified against the canonical JSON format. This means we _must_ parse and then pretty print (as we do here) because the document as stored may or may not be in canonical format. 2. However, it is important that we NOT translate from the JSValue to whatever internal datatype we are using and then back to JSValue, because that may not roundtrip: we must allow for additional fields in the JSValue that we ignore (and would therefore lose when we attempt to roundtrip). 3. We verify that all signatures are valid, but we cannot verify (here) that these signatures are signed with the right key, or that we have a sufficient number of signatures. This will be the responsibility of the calling code.

File with uninterpreted signatures

Sometimes we want to be able to read a file without interpreting the signatures (that is, resolving the key IDs) or doing any kind of checks on them. One advantage of this is that this allows us to read many file types without any key environment at all, which is sometimes useful.

A signature with a key ID (rather than an actual key)

This corresponds precisely to the TUF representation of a signature.

Convert a pre-signature to a signature

Verifies that the key type matches the advertised method.

Convert a list of PreSignatures to a list of Signatures

This verifies the invariant that all signatures are made with different keys. We do this on the presignatures rather than the signatures so that we can do the check on key IDs, rather than keys (the latter don't have an Ord instance).

Convert signature to pre-signature

Convert list of pre-signatures to a list of signatures

Target metadata

Most target files do not need expiry dates because they are not subject to change (and hence attacks like freeze attacks are not a concern).

Delegations

Much like the Root datatype, this must have an invariant that ALL used keys (apart from the global keys, which are in the root key environment) must be listed in delegationsKeys.

Delegation specification

NOTE: This is a close analogue of RoleSpec.

A delegation

A delegation is a pair of a pattern and a replacement.

See match for an example.

The key ID of a key, by definition, is the hexdigest of the SHA-256 hash of the canonical JSON form of the key where the private object key is excluded.

NOTE: The FromJSON and ToJSON instances for KeyId are ntentially omitted. Use writeKeyAsId instead.

Compute the key ID of a key

Sign a bytestring and return the signature

TODO: It is unfortunate that we have to convert to a strict bytestring for ed25519

Repository

This is an abstract representation of a repository. It simply provides a way to download metafiles and target files, without specifying how this is done. For instance, for a local repository this could just be doing a file read, whereas for remote repositories this could be using any kind of HTTP client.

Repository-specific exceptions

For instance, for repositories using HTTP this might correspond to a 404; for local repositories this might correspond to file-not-found, etc.

Log messages

We use a RemoteFile rather than a RepoPath here because we might not have a RepoPath for the file that we were trying to download (that is, for example if the server does not provide an uncompressed tarball, it doesn't make much sense to list the path to that non-existing uncompressed tarball).

Re-throw all exceptions thrown by the client API as unchecked exceptions

Errors thrown during role validation

Root metadata updated (as part of the normal update process)