{"id":19921,"date":"2021-02-03T20:25:13","date_gmt":"2021-02-03T14:55:13","guid":{"rendered":"https:\/\/valeurbit.com\/blog\/?p=19921"},"modified":"2021-02-12T17:57:09","modified_gmt":"2021-02-12T12:27:09","slug":"googles-certificate-transparency-as-a-data-source-for-attack-prevention","status":"publish","type":"post","link":"https:\/\/valeurbit.com\/blog\/googles-certificate-transparency-as-a-data-source-for-attack-prevention\/","title":{"rendered":"Google’s Certificate Transparency As A Data Source For Attack Prevention"},"content":{"rendered":"\n

We’ve prepared a two-part translation of Ryan Sears’s article on handling Google’s Certificate Transparency<\/a> logs . The first part gives an overview of the structure of the logs and provides a sample Python code for parsing records from these logs. The second part is devoted to obtaining all certificates from the available logs and setting up the Google BigQuery system for storing and organizing searches for the received data.<\/p>\n\n\n\n

Three years have passed since the original was written, and since then the number of available logs and, accordingly, entries in them has increased many times. It is all the more important to correctly approach the processing of logs if the goal is to maximize the amount of data received.<\/p>\n\n\n\n

Part 1. Parsing Certificate Transparency Logs Like a Boss<\/h2>\n\n\n\n

During the development of our first project, phisfinder<\/a> , I spent a lot of time thinking about the anatomy of phishing attacks and the data sources that would allow us to identify traces of upcoming phishing campaigns before they can cause any real damage.<\/p>\n\n\n\n

One of the sources we’ve integrated (and definitely one of the best) is Certificate Transparency Log (CTL), a project started by Ben Laurie<\/a> and Adam Langley at Google. Essentially, a CTL is a log containing an immutable list of certificates issued by a CA, which is stored in a Merkle tree, allowing each certificate to be cryptographically verified if necessary.<\/p>\n\n\n\n

To understand how much data we will have to deal with, let’s see how many entries are contained in each log from the list from the CTL website:<\/p>\n\n\n\n

import requests\nimport json\nimport locale\nlocale.setlocale(locale.LC_ALL, 'en_US')\n\nctl_log = requests.get(\n    'https:\/\/www.gstatic.com\/ct\/log_list\/log_list.json'\n).json()\n\ntotal_certs = 0\n\nhuman_format = lambda x: locale.format('%d', x, grouping=True)\n\nfor log in ctl_log['logs']:\n    log_url = log['url']\n    try:\n        log_info = requests.get(\n            'https:\/\/{}\/ct\/v1\/get-sth'.format(log_url),\n            timeout=3\n        ).json()\n        total_certs += int(log_info['tree_size'])\n    except:\n        continue\n\n    print(\"{} has {} certificates\".format(\n        log_url,\n        human_format(log_info['tree_size'])\n    ))\n\nprint(\"Total certs -> {}\".format(human_format(total_certs)))<\/code><\/pre>\n\n\n\n
At the output we get:<\/p>\n\n\n\n
ct.googleapis.com\/pilot has 92,224,404 certificates\nct.googleapis.com\/aviator has 46,466,472 certificates\nct1.digicert-ct.com\/log has 1,577,183 certificates\nct.googleapis.com\/rocketeer has 89,391,361 certificates\nct.ws.symantec.com has 3,562,198 certificates\nctlog.api.venafi.com has 94,797 certificates\nvega.ws.symantec.com has 200,401 certificates\nctserver.cnnic.cn has 5,081 certificates\nctlog.wosign.com has 1,387,492 certificates\nct.startssl.com has 293,374 certificates\nct.googleapis.com\/skydiver has 1,249,079 certificates\nct.googleapis.com\/icarus has 48,585,765 certificates\nTotal certs -> 285,037,607<\/code><\/pre>\n\n\n\n
285,037,607 at the time of writing. This is not such a large amount of data, but you still have to make some effort to effectively organize storage and retrieval of certificates. More on this in the second part.Translator’s comment<\/p>\n\n\n\n
Anatomy CTL<\/h2>\n\n\n\n
Receiving records from CTL is done over HTTP, which will allow us to easily receive data using modern libraries. Unfortunately, the data in the records themselves are confusing binary structures, which complicates the parsing process somewhat. An example of a log entry:<\/p>\n\n\n\n
\/\/ curl -s 'https:\/\/ct1.digicert-ct.com\/log\/ct\/v1\/get-entries?start=0&end=0' | jq .<\/em>\n{\n  \"entries\": [\n    {\n      \"leaf_input\": \"AAAAAAFIyfaldAAAAAcDMIIG\/zCCBeegAwIBAgI...\",\n      \"extra_data\": \"AAiJAAS6MIIEtjCCA56gAwIBAgIQDHmpRLCMEZU...\"\n    }\n  ]\n}<\/code><\/pre>\n\n\n\n
Each record contains fields leaf_input<\/code>and extra_data<\/code>in base64 format. Referring to RFC6962, we see that leaf_input<\/code>– the encoded structure MerkleTreeLeaf , and extra_data<\/code>– PrecertChainEntry .<\/p>\n\n\n\n
About PreCerts<\/h2>\n\n\n\n
It took me quite a long time to figure out what PreCert is all about (you can try it yourself, read the RFC , and, apparently, I’m not the only one . I will save you a lot of time thinking and searching in Google and formulate the purpose of PreCerts as follows:<\/p>\n\n\n\n
PreCerts are a separate type of certificate issued by a CA before it issues a \u201creal\u201d certificate. In fact, this is a copy of the original certificate, but contains a special x509 v3 extension called poison<\/em><\/strong> and marked critical. Thus, the certificate will not be validated by platforms that recognize this extension and know that it is PreCert, or by platforms that do not recognize this extension.<\/p>\n\n\n\n
My experience in information security suggests that such a measure is not very effective, if only because bugs in x509 \/ ASN.1 parsing are quite common and individual implementations may be vulnerable to various shenanigans that will ultimately allow PreCert to be validated. I understand why this was done, but it seems that completely removing PreCerts and leaving only the certificates actually issued by the CA in the CTL would be much wiser.<\/p>\n\n\n\n
Parse binary structures<\/h2>\n\n\n\n
As a person engaged in reverse engineering and from time to time participating in various CTFs, the task of parsing binary structures is not new to me. Most people refer to the struct<\/strong> module in such cases , but many years ago, while working for Phillip Martin , he introduced me to the excellent Construct library , which makes parsing such structures much easier. Below are the structures I used for parsing, as well as an example of their use to process records:<\/p>\n\n\n\n
from construct import \\\n    Struct, Byte, Int16ub, Int64ub, Enum, Bytes, Int24ub, this, \\\n    GreedyBytes, GreedyRange, Terminated, Embedded\n\nMerkleTreeHeader = Struct(\n    \"Version\"         \/ Byte,\n    \"MerkleLeafType\"  \/ Byte,\n    \"Timestamp\"       \/ Int64ub,\n    \"LogEntryType\"    \/ Enum(Int16ub, X509LogEntryType=0, PrecertLogEntryType=1),\n    \"Entry\"           \/ GreedyBytes\n)\n\nCertificate = Struct(\n    \"Length\" \/ Int24ub,\n    \"CertData\" \/ Bytes(this.Length)\n)\n\nCertificateChain = Struct(\n    \"ChainLength\" \/ Int24ub,\n    \"Chain\" \/ GreedyRange(Certificate),\n)\n\nPreCertEntry = Struct(\n    \"LeafCert\" \/ Certificate,\n    Embedded(CertificateChain),\n    Terminated\n)<\/code><\/pre>\n\n\n\n
import json\nimport base64\n\nimport ctl_parser_structures\n\nfrom OpenSSL import crypto\n\nentry = json.loads(\"\"\"\n{\n  \"entries\": [\n    {\n      \"leaf_input\": \"AAAAAAFIyfaldAAAAAcDMIIG\/zCCBeegAwIBAgIQ...\",\n      \"extra_data\": \"AAiJAAS6MIIEtjCCA56gAwIBAgIQDHmpRLCMEZUg...\"\n    }\n  ]\n}\n\"\"\")['entries'][0]\n\nleaf_cert = ctl_parser_structures.MerkleTreeHeader.parse(\n    base64.b64decode(entry['leaf_input'])\n)\n\nprint(\"Leaf Timestamp: {}\".format(leaf_cert.Timestamp))\nprint(\"Entry Type: {}\".format(leaf_cert.LogEntryType))\n\nif leaf_cert.LogEntryType == \"X509LogEntryType\":\n    # \u0412 \u0441\u043b\u0443\u0447\u0430\u0435, \u0435\u0441\u043b\u0438 \u0437\u0430\u043f\u0438\u0441\u044c - \u043e\u0431\u044b\u0447\u043d\u044b\u0439 X509 \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442<\/em>\n    cert_data_string = ctl_parser_structures.Certificate.parse(\n        leaf_cert.Entry).CertData\n    chain = [\n        crypto.load_certificate(crypto.FILETYPE_ASN1, cert_data_string)\n    ]\n\n    # \u041f\u0430\u0440\u0441\u0438\u043c \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u0443 `extra_data`<\/em>\n    # \u0447\u0442\u043e\u0431\u044b \u043f\u043e\u043b\u0443\u0447\u0438\u0442\u044c \u043e\u0441\u0442\u0430\u0432\u0448\u0443\u044e\u0441\u044f \u0447\u0430\u0441\u0442\u044c \u0446\u0435\u043f\u043e\u0447\u043a\u0438<\/em>\n    extra_data = ctl_parser_structures.CertificateChain.parse(\n        base64.b64decode(entry['extra_data'])\n    )\n    for cert in extra_data.Chain:\n        chain.append(\n            crypto.load_certificate(crypto.FILETYPE_ASN1, cert.CertData)\n        )\nelse:\n    #  \u0412 \u0441\u043b\u0443\u0447\u0430\u0435, \u0435\u0441\u043b\u0438 \u0437\u0430\u043f\u0438\u0441\u044c - PreCert<\/em>\n    extra_data = ctl_parser_structures.PreCertEntry.parse(\n        base64.b64decode(entry['extra_data'])\n    )\n    chain = [\n        crypto.load_certificate(\n            crypto.FILETYPE_ASN1, extra_data.LeafCert.CertData\n        )\n    ]\n\n    for cert in extra_data.Chain:\n        chain.append(\n            crypto.load_certificate(crypto.FILETYPE_ASN1, cert.CertData)\n        )<\/code><\/pre>\n\n\n\n
We get an array of X509 certificates from the chain with the certificate from leaf_input<\/code>as the first element<\/p>\n\n\n\n
As you can see, Construct makes it pretty easy to define binary structures in Python.<\/p>\n\n\n\n
Now that we understand what CTL is and how to parse individual records, we can move on to the second part – getting and saving all records from the logs with the possibility of further searching by certificates.<\/p>\n\n\n\n
Part 2. Retrieving, Storing and Querying 250M + Certificates Like a Boss<\/h2>\n\n\n\n
Collecting certificates<\/strong><\/p>\n\n\n\n
According to the RFC<\/a> , an endpoint is used to retrieve log entries get-entries<\/code>. Unfortunately, the task is complicated by the limitation on the maximum number of records that can be obtained in one request (controlled by the start<\/code>and parameters end<\/code>), and most logs allow only 64 records to be received at a time. However, Google’s CTLs, which make up the majority of all logs, use a maximum query size of 1024 entries.Translator’s comment<\/p>\n\n\n\n
Since the task is simultaneously IO-bound<\/em> (receiving records via http) and CPU-bound<\/em> (parsing certificates), for efficient processing it will be necessary to connect both asynchrony and multiprocessing.<\/p>\n\n\n\n
Since there were no tools that would allow you to easily and painlessly get and parse all CTLs (apart from the not particularly remarkable utility from Google<\/a> , it was decided to spend a little time and write a tool that would meet all our needs. The result was Axeman<\/a> , which uses asyncio<\/a> and the wonderful aioprocessing<\/a> library for loading, parsing and saving certificates to multiple CSV files, limited only by the speed of the Internet connection.<\/p>\n\n\n\n
Cloud exploitation<\/h2>\n\n\n\n
After receiving an instance (_approx.translated_ as VMs are called in Google Cloud) with 16 cores, 32GB of memory and a 750GB SSD (thanks to Google for the free $ 300 on the account for new accounts!), I launched Axeman, which downloaded all the certificates less than a day and saved the results in \/tmp\/certificates\/$CTL_DOMAIN\/<\/code><\/p>\n\n\n\n
Where is all this data stored?<\/h2>\n\n\n\n
Initially, Postgres was chosen to store and search the data, but although I have no doubt that with the correct schema Postgres could easily handle 250 million records (unlike my first attempt, which took about 20 minutes!), I started looking for solutions that:<\/p>\n\n\n\n