VAST — Visibility Across Space and Time

VAST is a network telemetry engine for data-driven security investigations. It ingests high-volume streams of network and logs data, indexes it for later retrieval, and offers several ways to access the data efficiently. The goal is to make network forensics more productive and enable data scientists to tap into the rich world of network event data.

Architecture

Features

• Built for network forensics: VAST is purpose-built for SecOps (incident responders, threat hunters) at the intersection of data science. Security investigations are data investigations, and VAST puts the analyst back in charge, helping to get to the right subset of telemetry for the problem at hand.
• Interactive queries: VAST’s multi-level indexing delivers sub-second response times over the entire telemetry set—perfect for explorative threat-hunting workflows.
• High-throughput streaming: VAST relies on end-to-end streaming to ingest massive amounts of data. Dynamic backpressure ensures that the system does not keel over when stuffing too much data into it.
• Rich Data Model: VAST’s type-rich data model helps to retain domain semantics with a flexible schema and query language. All types support meaningful operations, e.g., IP address support top-k prefix search and containers membership queries. Moreover, VAST’s typed expression syntax allows you to search over fields having a particular type.
• Unfederated data access: VAST defines a portable framing for messages and files to enable access to the data from various platforms. The zero-copy export mechanism makes data sharing with downstream analytics applications incredibly efficient. This empowers data scientists to work on their analytics, as opposed to building tools for parsing and plumbing.

VAST stands for Visibility Across Space and Time to reflect the key benefit for users: make it easy to express temporal and spatial event relationships to illuminate your network analysis.

Changelog v4.2

Changes​

• The long option name --emit-file-header of the pcap parser is now called --emit-file-headers (plural) to streamline it with the nic loader and the new capability to process concatenated PCAP files. #3513
• The decapsulate operator no longer drops the PCAP packet data in incoming events. #3515

Features​

• The new s3 connector enables the user to import/export file data from/to S3 buckets. #3496
• The new zmq connector ships with a saver and loader for interacting with ZeroMQ. The loader (source) implements a connecting SUB socket and the saver (sink) a binding PUB socket. The --bind or --connect flags make it possible to control the direction of connection establishment. #3497
• The new gcs connector enables the user to import/export file data from/to GCS buckets. #3498
• The new connectors http, https, ftp, and ftps simplify using remote files in pipelines via HTTP(S) and FTP(S). #3499
• The new lines parser splits its input at newline characters and produces events with a single field containing the line. #3511
• The pcap parser can now process a stream of concatenated PCAP files. On the command line, you can now parse traces with cat *.pcap | tenzir 'read pcap'. When providing --emit-file-headers, each intermediate file header yields a separate event. #3513
• The nic loader has a new option --emit-file-headers that prepends a PCAP file header for every batch of bytes that the loader produces, yielding a stream of concatenated PCAP files. #3513
• You can now write show nics to get a list of network interfaces. Use show nics | select name to a get a list of possible interface names for from nic. #3517

Bug Fixes​

• Pipelines now show up in the “stopped” instead of the “created” state after the node restarted. #3487

Install & Use

Copyright (c) 2014, Tenzir GmbH