Thinking about “context” for web assets: Automating attack surface tagging and classification for scale

This research and article was written by Colin Flaherty as part of a summer internship with assistance from some of the team at Coalition: Beatriz Lacerda (Data Scientist), Florentino Bexiga (EM Data Collections) and Tiago Henriques (GM Customer Security).

Introduction to the problem

Assets can either be identified as domains or IP addresses. A single domain can point to multiple IP addresses, and multiple domains can point to a single IP address. Assets are assigned vulnerability scores through a slew of heuristic methods, discussion of which is beyond the scope of this article. If multiple assets have the same vulnerability score, how can one prioritize which of these assets to examine first?

For example, a login page without SSL is probably more important to address than a static blog post without SSL. One way to prioritize assets that have the same or similar vulnerability scores is by assigning them comparable “contexts” that encapsulate information about how sensitive or otherwise important they might be. One attempt to tackle this problem is “Eyeballer," an open-source machine learning project that uses a convolutional neural network to assign tags to webpage screenshots such as “homepage,” “custom 404,” “login,” and “old-looking.”

For the purposes of this article, I focus on contextualizing HTTP/HTTPS assets, i.e. web pages.

High-level approach

Contexts should be comparable because they will be used to compare assets. Therefore, structured contexts should be favored over free-form contexts. For example, it would be easier to compare two assets if each of their contexts consists of a subset of a lexicon of “context tags” than if each asset’s context consists of a paragraph of free-form text. Similarly, as size of this context tag lexicon increases, comparing contexts becomes more cumbersome. If every web asset is tagged with its top five most frequent words, then the total context tag set across all web assets could quickly become unmanageable as the number of web assets being tagged grows.

In order to create a set of context tags that could be used to contextualize web assets, web assets may be examined from several perspectives: spatio-visually (layout, colors, design) as a screenshot, as structured code (HTML, CSS), and as a document consisting of written language. A convolutional neural network like Eyeballer can be used to generate tags for a webpage based solely on screenshots of its fully-rendered state. This approach is useful for tags that represent complex categories that nevertheless possess distinctive visual elements, such as the category of “blog” web pages or the category of “old-looking” web pages.

However, it requires significant amounts of labeled data to be precise enough for production, and existing publicly available datasets are not sufficiently large.

An example of code-based context tag generation would be examining a web page’s HTML for an tag with type “password” in order to determine if the page has a login feature. Finally, for text-based tags, natural language processing techniques such as bag-of-words frequency analysis can be applied.

For web asset contexts, one area to explore further is combining spatio-visual information and semantic information to generate tags. A simple bag-of-words frequency analysis may not be very effective on web pages because word frequencies on web pages exhibit sparsity. While important words may show up with high-frequency in traditional text (books, articles, etc.), important words might show up only once or twice on a web page if much of their importance is conveyed via positioning or styling. For example, an “About” page for a company might only display the word “About” once, but “About” would probably appear in large size font at the top of the page, possibly with extra embellishments and separated from other text.

Eyeballer

One proposed approach to providing context for web assets is Eyeballer, which uses a convolutional neural network to categorize web page screenshots. It is open-sourced, and its creators (“BishopFox”) have trained and tested it on a set of ~13,000 webpage screenshots. Its architecture consists of a MobileNet layer pre-trained on the popular ImageNet dataset and then a fine-tuning layer consisting of global average pooling and dense neural network layers for classification.

It uses MobileNet as opposed to other image models built for transfer learning because MobileNet is lightweight relative to its peers. It is pre-trained to determine if a web page can be identified with a subset of the following tags: “homepage,” “custom 404,” “login,” and “old-looking.” On a set of approximately 2,000 images (manually labeled by BishopFox), the network exhibits an all-or-nothing accuracy of 73.77%. For specific labels, the following precision and recall percentages were observed.

LabelCustom404LoginHomepageOld-lookingPrecision95.38%82.04%76.61%86.82%Recall66.19%89.89%94.16%66.32%

The convolutional neural network approach should be well-suited towards identifying high-level visual context such as whether a page is a blog or a homepage. However, in its current state Eyeballer may not be accurate enough for context generation in production.

Since it is unclear how complex categories such as that of a “homepage” could be identified from a code or text-based perspective, I turn to how Eyeballer could be improved.

In particular, there are four avenues for improvement: increased data, hyperparameter tuning, improving the model’s ability to extract features specific to web pages, and more precise label choice. Firstly, the most obvious way to improve this model would be to increase the training data set. However, this can be expensive. The default threshold at which a confidence score is interpreted as predicting a label is 50%. Raising this threshold to 90% improved precision for the “old-looking” label to 94.21% but lowered its recall to 39.58%. In general, adjusting this threshold means trading recall for precision.

Secondly, the specific architecture of Eyeballer could possibly be improved in future versions to better learn to extract web page features. The current training paradigm begins with a neural network pretrained on ImageNet (a large corpus of labeled images of objects) and then trains a fine-tuning layer on top of it. The underlying pre-trained neural network is not trained, however, to extract different features of a webpage (such as navigation bars or input fields) but is rather trained to identify more general lines and shapes. Due to Eyeballer’s global average pooling layer, which is immediately applied to the output of MobileNet, Eyeballer does not have a layer of convolutions that learn to extract features specific to web pages such as navigation bars or comment sections.

Exploring additional convolutional feature extraction layers (in conjunction with additional training data) could engender nontrivial improvements. In addition to training these additional layers with the existing dataset provided by BishopFox, transfer learning could also be applied. By temporarily replacing the Eyeballer classification layer with another classification layer that outputs whether a webpage contains common web page elements such as a navigation bar or comment section, and training this modified neural network on a dataset of web screenshots annotated with whether they include each of these web page elements, the intermediary convolution layers might learn to extract these web page elements.

Finally, label choice might help to improve the model. For example, it might be the case that the model is struggling to achieve a >90% precision for homepage classification because the visual particulars of homepages are so varied. 8,000 training screenshots may not be enough to capture this diversity sufficiently. However, note that homepages are of interest primarily because they typically link to many other pages, which may themselves possess sensitive assets, and then note that these links usually appear in some form of a navigation bar. Therefore, it might be just as useful to label web assets by whether or not they have a navigation bar as it is to label them by whether or not they are a homepage. Since navigation bars are less varied in form than homepages (they usually are at the tops of pages, display rows of words or phrases, etc.) a convolutional neural network may be able to learn to identify them more easily than they could learn to identify homepages in general.

HTML and CSS-informed keyword extraction

A bag-of-words approach can be applied to a webpage pretty simply. Firstly, the domain in question is visited. Then, the response’s HTML is parsed for all words that are not HTML tags or arguments. This will yield a multiset of words that can be cleaned with standard natural language processing techniques such as tokenization, removing stop words, removing punctuation, only preserving adjectives and nouns, generating n-grams, and lemmatization. This cleaning can be achieved easily with a programming package such as NLTK (see this demo code).

Finally, the resultant cleaned set of words forms a multiset, or “bag of words,” that can be further filtered for keyword extraction.

While the bag-of-words approach is useful for keyword extraction in general, the simplest way of determining word importance (frequency analysis) is not particularly well-suited for web page analysis. On a web page, the most important words may only appear infrequently, and visual layout and design is an important means through which information is conveyed. To circumvent this limitation, words can be tagged with features that signify their importance beyond frequency, and this expanded feature set can be used to determine relative word importance.

Firstly, words can be analyzed for spatial-visual importance. One heuristic that could be explored is the number of words that exist within a circle around a particular instance of a word. For example, suppose the word “About” appears on a web page twice. Then, an algorithm can be employed to take as input a screenshot of the webpage in question and annotate each instance of a word with a dot at its center. For each instance of the word “About,” a second algorithm can count the number of words (represented by dots) within a circle of X radius. If there are very few neighboring words for one of the instances of “About,” it could be concluded that this instance of “About” is a title and that furthermore this web page is an “About” page.

Practically, this approach is limited by the fact that not all “About” pages will display the word “About” as a title. They might instead display the words “Our Story” or “Meet the Owners.” While this approach can look for these synonym titles as well, it will not exhaustively identify all possible titles for an “About” page. In other words, the precision would be high (if a page has “About” or “Our Story” as a title, then it most likely is an “About” page) but recall might be lower (if an “About” page has a different title, then it might not be properly identified).

Secondly, words can be analyzed via a web page’s HTML. For example, a word is probably more important if it is included within the