Signing a PDF with adbe.pkcs7.detached

Question

I have a PDF generator written in C, and now I want to add digital signatures to it. I started with a minimal PDF, signed it with JSignPdf, and am now trying to get my own program to produce a file that Adobe Acrobat Reader will interpret identically. I've checked the Digitally sign PDF files question, but the comments there seemed to conclude that one should use iText instead of trying to do it yourself. I don't want that.

Update: I have indeed also read both the PDF Reference for 1.7 and the "32000" specification linked below, but sometimes I get a bit lost by the number of references. Starting with a working example is often the easiest way for me to understand how everything fits together. Sorry for not making this clear in my initial post.

I've gotten Acrobat Reader to acknowledge that there is a signature in the file, but something is still wrong. In the Signature Panel it says "Signed by Unknown" instead of using the correct name from the key. When opening "Signature Properties" it says "This signature is invalid because there are errors in the formatting or information contained in this signature". On the "Advanced Signature Properties", the Hash Algorithm is "Not available".

The PDF from JSignPdf is correct, according to Acrobat Reader. After telling it to accept my self signed certificate, it displays a nice green checkbox for the signature. To find the minimal additions needed, I've cleared one PDF tag after another, carefully not changing the offset for the remaining tags. This gives the same "This signature is invalid..." error message as above, but it still shows "The signer's identity is valid", as well as showing the Hash Algorithm as "SHA1".

The question is what the reason is for this difference, and if there are any tools that can give a more detailed explanation of what is wrong?

In the /Type/Catalog dictionary, I have an /AcroForm. I've tried putting it both in place and as a reference, but that makes no difference. The /AcroForm contains /SigFlags 3 and /Fields [ x 0 R ], where x is the id of a /Type/Annot with /Subtype/Widget. (The "endobj" is moved to the ">>" line to save some space here.)

Update: There are some dictionaries, even though I don't remember their name right now, where the "in place" vs "reference" is significant. Especially the implementation notes in the 1.7 spec has a few of these, as well as a few "the spec says this field is optional, but actually it's required".

2 0 obj <<
 /Type /Catalog
 /Pages 3 0 R
 /AcroForm <<
  /Fields [ 8 0 R ]
  /SigFlags 3
  >>
 >> endobj

In the /Type/Page object, I have /Annots [ x 0 R ], which seemed to be required to get Acrobat Reader to accept that there was any signature here at all.

Update: With a working signature, things change a bit. Without this reference, Acrobat Reader does indeed say the signature is valid, but doesn't show any details about it. With it, the "Signature Properties" menu item is enabled again.

4 0 obj <<
 /Type /Page
 /Parent 3 0 R
 /Resources <<
  /ProcSet [/PDF /Text]
  /Font << /F1 6 0 R >>
  >>
 /MediaBox [0 0 595 842]
 /Contents 5 0 R
 /Annots [ 8 0 R ]
>> endobj

The /Annot dictionary contains /T(Signature1), /FT/Sig, /Rect[0 0 0 0], and /V y 0 R, where y is a /Type/Sig object. The JSignPdf version also contains "/F 132" and "/P 4 0 R", but I can't find them in the PDF Specification. They don't seem to be required anyway.

Update: Ah, I had missed the link from section 12.7.1 to 12.5.2.

8 0 obj <<
 /Subtype/Widget
 /T(Signature1)
 /V 7 0 R
 /Type/Annot
 /FT/Sig
 /Rect [ 0 0 0 0 ]
>> endobj

The /Type/Sig object contains /Filter/Adobe.PPKLite, /SubFilter/adbe.pkcs7.detached, /M(D:20160907094326+02'00'), a /ByteRange array and a /Contents string.

Update: I'm using this combination as it was recommended for PDF/A.

7 0 obj <<
 /Contents <3082031f...>
 /Filter/Adobe.PPKLite
 /Type/Sig
 /ByteRange [ 0 904 2907 527 ]
 /SubFilter/adbe.pkcs7.detached
 /M(D:20160907094326+02'00')
>> endobj 

The /ByteArray has for values: 0, offset-of-last-byte-before-"<"-in-Contents, offset-of-first-byte-after-">", and the length of the remainder of the file. If I take the file from JSignPdf, run this (where buf contains the file data):

SHA1_Init(ctx);
SHA1_Update(ctx, buf + offset1, len1);
SHA1_Update(ctx, buf + offset2, len2);
SHA1_Final(digest, ctx);

I get the exact same data as in the PKCS7 data for the ":messageDigest" tag. The same is true for my own file. So, I trust those values to be correct.

Using the same cert and key I get the exact same PKCS7 data, except of course the messageDigest and rsaEncryption hex dumps. However, copying the JSignPdf PKCS7 data to my file (as they are exactly the same length) doesn't work, it still complains about not finding the Hash Algorithm. My PKCS7 data in the JSignPdf works, but of course gives the wrong checksum. So, everything OpenSSL-related is most likely correct, the problem must be in the PDF tags somewhere. Is there a reference I've missed, or some tag or object ordering that must be followed?

Solved: The only remaining thing to play with at this point was the values for the ByteRange tag. The first length was actually ok. However, the second offset was off by one in the implementation, being 1 too small. Adjusting this, I got a green checkbox for the signature!

Answer 1

In short,

you might have a one-off issue, see at the bold paragraph near the end of the answer. If that turns out not to be your issue, please share the files in question for analysis.

To begin with,

please consider reading the specification for a format before attempting to manipulate files in that format.

The PDF specification is ISO 32000-1 (part 2 is under construction), and you can download a free copy with a very few changes (making clear that this is not the ISO copy) on Adobe's web site:

http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf

(In the middle of your question text you show that you do know that this document exists but you also show that you have not properly studied it.)

For a first overview on integrated PDF signatures, have a look at this answer on Information Security Stack Exchange.

In detail,

that been said, let's look at your problems and point you to some appropriate section of the specification:

In the /Type/Catalog dictionary, I have an /AcroForm. I've tried putting it both in place and as a reference, but that makes no difference. The /AcroForm contains /SigFlags 3 and /Fields [ x 0 R ], where x is the id of a /Type/Annot with /Subtype/Widget. (The "endobj" is moved to the ">>" line to save some space here.)

The document catalog is specified in section 7.7.2.

The AcroForm dictionary there is specified as

AcroForm dictionary (Optional; PDF 1.2) The document’s interactive form (AcroForm) dictionary (see 12.7.2, "Interactive Form Dictionary").

In particular it does not specify whether it is to be a direct object or not. Thus, "that makes no difference" indeed.

The interactive form dictionary is specified in section 12.7.2.

In particular,

SigFlags integer (Optional; PDF 1.3) A set of flags specifying various document-level characteristics related to signature fields (see Table 219, and 12.7.4.5, “Signature Fields”). Default value: 0.

...

1 "SignaturesExist" If set, the document contains at least one signature field. This flag allows a conforming reader to enable user interface items (such as menu items or pushbuttons) related to signature processing without having to scan the entire document for the presence of signature fields.

2 "AppendOnly" If set, the document contains signatures that may be invalidated if the file is saved (written) in a way that alters its previous contents, as opposed to an incremental update. Merely updating the file by appending new information to the end of the previous version is safe (see H.7, “Updating Example”). Conforming readers may use this flag to inform a user requesting a full save that signatures will be invalidated and require explicit confirmation before continuing with the operation.

and

Fields array (Required) An array of references to the document’s root fields(those with no ancestors in the field hierarchy).

and the whole section 12.7 describing interactive forms.

In the /Type/Page object, I have /Annots [ x 0 R ], which seemed to be required to get Acrobat Reader to accept that there was any signature here at all.

Section 12.5 describes annotations.

A signature field is a form field. Form fields can have visualizations on some pages. Such visualizations are Widget annotations. If a form field has only one widget annotation, the form field object and the widget object may be merged into a single object.

All the annotations on some page are referenced from the page's Annots array.

But no, you can have invisible signatures (they do appear in the Adobe Reader signature panel, merely not in the document) which do not require an annotation referenced from a page.

The /Annot dictionary contains /T(Signature1), /FT/Sig, /Rect[0 0 0 0], and /V y 0 R, where y is a /Type/Sig object. The JSignPdf version also contains "/F 132" and "/P 4 0 R", but I can't find them in the PDF Specification. They don't seem to be required anyway.

Ah, so you do know the specification. Please use it!

T, FT, and V are form field entries, cf. section 12.7.3.

Rect, Type, F, and P are annotation entries, cf. section 12.5.2.

The /Type/Sig object contains /Filter/Adobe.PPKLite, /SubFilter/adbe.pkcs7.detached, /M(D:20160907094326+02'00'), a /ByteRange array and a /Contents string.

All of these entries are specified ion section 12.8.1 and more extensively in the remainder of 12.8.

The /ByteArray has for values: 0, offset-of-last-byte-before-"<"-in-Contents, offset-of-first-byte-after-">", and the length of the remainder of the file.

It is specified as

ByteRange array (Required for all signatures that are part of a signature field and usage rights signatures referenced from the UR3 entry in the permissions dictionary) An array of pairs of integers (starting byte offset, length in bytes) that shall describe the exact byte range for the digest calculation. Multiple discontiguous byte ranges shall be used to describe a digest that does not include the signature value (theContents entry) itself.

..

This range should be the entire file, including the signature dictionary but excluding the signature value itself (the Contents entry).

(Even though this merely is a recommendation, signatures not following this recommendation usually are not accepted.)

---

Your offset-of-last-byte-before-"<"-in-Contents seems odd, it should be the offset of the "<", i.e. the length of the part before the "<".

Aside from that you seem to have correctly recognized the values in question. If this is not the cause of your problem, therefore, I assume there are still other problems in your PDF or the signature container you inject. Please share the files in question (e.g. via a public dropbox or google drive share) for further analysis.

---

This all being said, depending on the use case you develop your signing code for you probably should look into PAdES style signatures instead of the good ol'fashioned ones.

Answer 2

A certificate is always certified by a certification authority. So, a certificate is signed by an authority. To check a certificate, means to verify this signature and so you need the authority certificate witch must be avalable to process in some root CA. Your signing pdf certifiacte is self certified. You need to put your self-signed certificate in root CA hive in your cert keystore.