The Lucid / DPT (data plane threads) language

The Lucid / DPT (data plane threads) language

Lucid is a high-level language for programming high-performance data planes. Its designed to feel more like a regular programming language than P4, with simple, general, and modular abstractions. At the same time, Lucid is carefully designed so that its compiler can translate programs into efficient code optimized for real platforms.

An important idea in Lucid is pipeline parallelism. Unlike other languages, Lucid guarantees that a program can be pipelined in its type-and-effect. In other words, any type-checking program is guaranteed to always access persistent objects in the same order. This is necessary on targets like the Tofino that are physical pipelines. On other targets it allows the compiler to use pipelining to exploit parallelism safely (without changing program semantics).

Lucid also has an interpreter with support for the entire language and fast, multi-node simulations. This makes it a convenient modeling language for data-plane applications.

The Lucid codebase is also organized for extensibility, with a modular micropass design and frontend / midend / backend components that use a few common core IRs throughout.

Getting Started

The best way to get started with Lucid is to compile it from source (instructions below), download the (Visual Studio syntax highlighting extension), and then follow the Tutorials

Installation

Compiling Lucid from source requires installing ocaml and a few dependencies from its package manager.

  1. install opam

Opam is ocaml’s package manager. Install it from source with:

bash -c "sh <(curl -fsSL https://opam.ocaml.org/install.sh)"

Or follow the instructions here: opam install

  1. add a ocaml 4.12 switch

Lucid is pinned to ocaml 4.12 due to a few minor API changes in later versions.

opam switch create 4.12.0
opam switch 4.12.0
  1. install the Lucid dependencies

From the root of this repo, run:

opam install --deps-only .
  1. build lucid with make
make

Install script

On a linux machine (with sudo access), you should be able to do:

./install_dependencies.sh
make

to install all the dependencies from scratch (i.e., including opam, ocaml, etc.) and build the Lucid interpreter (dpt) and tofino compiler (dptc).

Docker

There is also a small docker image for Lucid. Its designed to be as simple as possible to install, but not always up to date.

1. Install docker

  • if you are on a laptop/desktop, just install the docker desktop app: docker desktop
  • if you are on a server… you can probably figure out how to install docker

2. Clone this repository and pull the lucid docker container

Run this in your terminal:

git clone https://github.com/PrincetonUniversity/lucid/
cd lucid
./docker_lucid.sh pull

This will download about 400MB of data and should take < 5 minutes.

Once finished, you can run ./docker_lucid.sh interpret to run the interpreter or ./docker_lucid.sh compile to run the Tofino compiler. The docker_lucid script takes care of forwarding all arguments, files, and directories to / from the docker image.

Syntax highlighting

There is a VSCode syntax highlighter for Lucid here: (https://github.com/benherber/Lucid-DPT-VSCode-Extension)

Lucid also renders okay as C (besides polymorphic size arguments with the single quote symbol).

What to look at next

The tutorials covers basic language features and using the interpreter and compiler.

The interpreter test programs has examples that cover every language feature, with some documentation.

There are re-usable data structure libraries that may be useful for your application, and also illustrate how to package Lucid libraries as modules with clean interfaces.

There are a few larger example applications that give you a sense of some of the cool things you can do with Lucid.

The examples directory also has some other subdirectories with misc examples. Some may be out of date, though.

Finally, Lucid’s wiki documents all of Lucid’s well-supported features.

Projects that used Lucid

Lucid has been used in a number of projects. Here is a listing of those that we know about: projects

Publications

This is a list of the main papers/articles/dissertations about core lucid features.

About Lucid or its components:

  • Lucid, a Language for Control in the Data Plane (SIGCOMM 2021) – The SIGCOMM 2021 artifact is in the sigcomm21_artifact branch). This paper is a good place to start, particularly if you have a networking / systems focus.

  • Safe, modular packet pipeline programming (POPL 2022). This covers modularity and polymorphism.

  • Lucid, a high-level easy-to-use dataplane programming language (Devon K. Loehr’s PhD Thesis). This is a great general overview and snapshot of Lucid as of late 2024.

  • Automated Optimiation of Parameterized Data-Plane Programs with Parasol (arxiv preprint). This is about Lucid’s features for automated program tuning, which is not yet well documented.

  • Sequence Abstractions for Flexible, Line-Rate Network Monitoring. (NSDI 2024). This adds stronger pattern matching to Lucid, and higher-level abstractions that have not yet been integrated.

Citation

For citations, please use the SIGCOMM 2021 paper:

Sonchack, John, Devon Loehr, Jennifer Rexford, and David Walker. "Lucid: A language for control in the data plane." In Proceedings of the 2021 ACM SIGCOMM 2021 Conference, pp. 731-747. 2021.

@inproceedings{lucid,
  title={Lucid: A language for control in the data plane},
  author={Sonchack, John and Loehr, Devon and Rexford, Jennifer and Walker, David},
  booktitle={Proceedings of the 2021 ACM SIGCOMM 2021 Conference},
  pages={731--747},
  year={2021}
}