Dirkjan Ochtman: writing

rustls

rustls is a pure Rust implementation of the TLS protocol.

Substantial effort last month went into migrating support for PEM decoding from the rustls-pemfile crate (backed by the base64 crate) to the rustls-pki-types crate. While the new decoder is slightly slower, it avoids timing side channel attacks (only used for PEM items containing private keys). We iterated a number of times on making the new API easy to use. Adopting the new APIs should shrink dependency graphs by getting rid of both rustls-pemfile (which got a semver-compatible bump to use the new implementation) and the base64 crate.

We went through a number of iterations reviewing a PR from Bart Dubbeldam and Adolfo Ochagavía adding support for RFC 7250 raw public keys.

ComplexSpaces added deployment considerations to the rustls-platform-verifier README. We now recommend (almost) everyone use the platform verifier for their server certificate verification needs.

Joe has been doing interesting work to improve performance:

• We can now send flights of handshake messages in a single message.
• He eliminated an allocation in signature verification.
• And eliminated large copies in resumption code paths.

I contributed some smaller improvements last month:

• I finally converted one of the rustls examples from the long-dead docopt to clap. Joe kindly offered to take care of the others.
• I finished a contributed PR to enable signing without a private key in rcgen.
• Clarified error handling in rustls-native-certs.

Hickory DNS

Hickory DNS is a project to build a comprehensive suite of Rust libraries to build DNS services on top of. Because the project is nearing a (fairly large) feature release, I've been trying to make a number of improvements to the project, cleaning up the API and moving code around to reduce complexity. Here are some of the things I worked on:

• I moved the RuntimeProvider trait from hickory-resolver down into hickory-proto. This abstraction was created to allow the resolver to abstract over different async runtimes (and their I/O traits and types), but it did not actually depend on the resolver infrastructure itself. Meanwhile, at the lower -proto level we had a somewhat parallel set of abstractions, so I set out to move the RuntimeProvider into -proto and try to make it the focal point of abstracting over runtimes and I/O. This work was split over two PRs, one for UDP and one for TCP. I should probably review whether there is potential for further simplification in the other protocols (H2, QUIC, H3).
• Hickory DNS depends on the resolve-conf crate in order to find the system DNS configuration (on Unix systems). Unfortunately the resolv-conf crate has been unmaintained for a while. While attempts to contact the maintainer didn't lead to any results, Ferrous team member Andrei Listochkin mentioned that he knew the original maintainer and could talk to them via Telegram. Soon, the repository and crate ownership had been transferred to us, so that we can start fixing some of the outstanding issues soon.
• After reviewing a PR to allow customization of the DNS over HTTP URL path I noticed that the configuration for the main Hickory DNS binary crate was distributed over the -server library crate and the hickory-dns binary crate. Some of this didn't really make sense, so I submitted a collection of config tweaks to improve on the situation.
• While reviewing another PR, I noticed that the hickory-server crates had a bunch of instances of Arc<Box<dyn AuthorityObject>>, which seemed unnecessarily complex. I cleaned these up so that we use Arc<dyn AuthorityObject> instead.
• After we noticed that some tests against the Quad9 name servers kept causing issues in CI, I decided to kill the quad9 tests. We still run tests against Google and Cloudflare to help validate our code.
• A user reported that the Cargo features for the server were pulling in OpenSSL even when that should be necessary. I submitted a PR to clean up server features which improved the situation.
• While working on KumoMTA (see below), I noticed that RecordSet::new() took a &Name argument only to clone it. I changed it to take a Name directly so the caller can hand over ownership where possible.
• Made some minor recursor tweaks.

I reviewed 25 PRs Hickory DNS last month, here are the more notable ones:

• Chained authority implementation
• CNAME resolution support for the recursor
• NSEC3 validation

rustup

rustup is the tool most Rust developers to use manage their compiler toolchains; I joined the team last year after some of the previous members had left, reducing the available capacity. We're currently gearing up for the beta of rustup 1.28 which contains some larger changes, including:

• No implicit toolchain installs
• Default to rustls with the rustls-platform-verifier for downloads
• Improved logging based on tracing-rs

In September I recruited Chris to the team. He had already been helping out a bunch with the Windows parts of rustup in particular, and we decided it made sense to ask him to join the team, which he happily agreed to.

Notable changes from last month:

• We finally merged a contributed PR to replace winreg with windows registry, after we coordinated with the windows-rs team and Chris on the best way forward. This included scrapping some unsafe code from the rustup code base because we can now build on better upstream abstractions.
• Improve the logging setup to better integrate tracing.

pyrtls

I launched a new project called pyrtls last month. pyrtls provides Python bindings to rustls, seeking to expand access to rustls' great performance and security track record as an alternative to the OpenSSL-based ssl module that is a part of the standard library.

I started working on this intermittently back in 2021 and finally decided to write up some documentation and "ship" it by announcing the project on social media. At the end of the month, it earned a mention in last month's issue of the Feisty Duck Cryptography and Security Newsletter. I was happy to receive my first contributed PR adding a typing stub a few days later.

Although the project has managed to earn 70 stars over the past 4 weeks, I've been a little disappointed with the lack of feedback so far. Given that I don't have a use case for this project myself (since I don't really use Python anymore), I probably won't spend much more time on it until there is some uptake.

KumoMTA

I've been doing some consulting work for KumoMTA, a startup building a mail transfer agent (MTA) for enterprise senders. I had connected with their cofounder Wez (of wezterm fame) through issues against rustls and Hickory (both of which KumoMTA uses to build their MTA).

• My first issue was to build a Redis rate limiting implementation that doesn't use the redis-cell Redis extension. This integrated a Lua script-based implementation of the generic cell rate algorithm which can be used in deployments where custom Redis extensions are not viable.
• Then, I started working on an implementation of the Sender Policy Framework (SPF) standard. This will be used in KumoMTA to help detect spam. Wez had already started on an implementation, so I picked up from where he had left off and started implementing the core protocol.
• I noticed that the KumoMTA codebase was using both lazy-static and once_cell, so I decided to spend a little time (after getting approval from Wez, of course) cleaning this up and migrating the entire project to the newly available OnceLock and LazyLock types stabilized in 1.70.0 and 1.80.0.

Quinn

Quinn is the most popular Rust implementation of the QUIC transport protocol. Some interesting PRs I reviewed last month:

• Enabling building quinn-proto for WASM targets from the folks at n0 computer, who are building Iroh on top of Quinn.
• Faster UDP I/O on Apple platforms, from the Mozilla folks who are in the process of adopting quinn-udp for Firefox's UDP needs.

I also submitted some PRs of my own:

• Fixed bookkeeping for datagrams in response to an issue.
• Cleaned up some new clippy lints.
• Avoided a potential panic in TLS server config setup.
• Published 0.11.7 and 0.11.8 releases for quinn-proto and quinn. We also investigated a DoS vulnerability that was introduced in quinn-proto 0.11.0 and published a security advisory after 0.11.7 was released.
• Published 0.5.5 for quinn-udp.

Askama

Askama is a compile-time template engine using a Jinja-like syntax.

Reviewed some changes to:

• Fix include statements within block fragments.
• Improve CI to run all tests and packages to completion even if some fail.
• Accept a trailing comma in arrays.

I also:

• Improved the error in case a Rust operator is used as a delimiter in a custom syntax, in response to an issue report.
• Applied clippy suggestions from Rust 1.81 and fixed dependencies after a new version of Axum was published that uses tower 0.5.

I should really reserve some time to finish the 0.13 release.

tokio

• I prepared a tokio-stream release to support some work I was doing in tonic to improve newly public API.
• Although I am only peripherally involved with the tracing-rs project, I spent some time cleaning up all the warnings that had accumulated over time.
• Reviewed a tracing-opentelemetry upgrade to opentelemetry 0.25 and published the 0.26.0 release.

instant-acme

instant-acme is a RFC 8555 client for provisioning TLS certificates.

Reviewed a contributed PR to avoid bad nonce errors which the Pebble test CA generates to exercise client code. As suggested by the spec, we now retry requests that trigger a bad nonce error a few times before giving up.

On sustainability

Although I have been reaching out to organizations that I'd hoped would be interested in funding this kind of work across the Rust ecosystem, so far I have not succeeded in finding any. Although I'm very grateful to my current sponsors, I'd like to find some companies that can help make my work more sustainable. Happy to get on a call to discuss how I could help out your team!

Many thanks to these sponsors (5 USD/month or more):

• astral-sh
• codecov
• thomaseizinger
• stepfunc
• MJDSys
• repi
• sourcegraph
• eightseventhreethree
• paolobarbolini

Finance without fear

I believe the Rust project must figure out how accept donations for the benefit of the Rust language, compiler and ecosystem. Whether this involves a new legal entity or not, we should use the corporate money on the table where it could enable the community to move the ecosystem faster than otherwise possible.

Even if other communities have run into trouble with this, the focus of the Rust project on positive-sum outcomes and careful involvement with the community should make it possible to come up with proposals that are widely supported. Things like improving the state or scope of CI, having a procedural macro book written or even building custom tooling might all be good ideas.

So far, the Rust project has preferred existing tools over building our own. However, I think we run into the limitations of this strategy both in the example of CI and in the limitations of GitHub issues for some of the more involved discussions. This is a classic build vs buy discussion -- if we consider CI and RFC discussions to be a "core business" of the Rust project, then it might make sense to invest in improving these parts of the project's process.

(Is "tooling without terror" too terrible a pun?)

Foundation without fear

Setting up a legal entity and governance structure for the Rust project appears to be a controversial topic in the Rust community. Of course, it could be a home within an existing foundation: joining Thunderbird under the umbrella of the Mozilla Foundation, or another entity like the Software Freedom Conservancy. I would like the increased transparency and accountability that might come with a more explicit setup where it concerns how the leadership of the project evolves.

Since it keeps coming up, it would be good to have an RFC about the foundation idea; to have a thorough conversation on the benefits and downsides. If the latter end up outweighing the former, the RFC can serve as negative space documentation (per Graydon) for everyone thinking about it.

In a different vein, I think PEP 8016 is worth exploring as a carefully considered alternative we could crib ideas from. As an experiment, we could have part of the core team be elected by a (large cross-section of) the community.

Asynchronous alignment

I think the Rust project could benefit from an increased sense of the direction of the project. While the language design and library evolution processes "just" some adjustments to the increased scale at which we're operating, other parts of how the project executes are not so well-defined, or happen in synchronous meetings that are opaque to the community. I'm used to open source projects where decision making happens exclusively in asynchronous venues, and the current way Rust teams and working groups function is quite different -- with a notable exception for the way the compiler team documents their work.

The yearly roadmap building effort is one great way of doing this, but the yearly cycle makes it a very coarse-grained check-in. One suggestion I've been thinking about is to have a community event (maybe with video?) every release cycle or two where a group of core team members answers crowd-sourced questions from the community (along the lines of an AMA or Google's TGIF event).

This could also help cut down on the "Why don't you just" and "Be Heard" sentiments that I've seen core project team members lament over the past year. I think those sentiments come from a feeling of powerlessness and being underinformed. In my mind, more deliberate and continuous communication about the state of the project could go a long way towards improving on this.

Careful conduct

Although I wholly support the code of conduct and appreciate many aspects of the way we interact online, there's some things that I think could be improved. While I've seen people expressing their personal frustration with a project decision get quickly moderated, I've also seen users who keep posting vague forum threads or exert significant stop energy on others' proposals. I wish we could be more openly critical of decisions (not people) while at the same time more strict towards net-negative participants.

On a more subtle note, I've felt that core project team members cast the community as an angry mob at times. Although I realize it is a big job, I think it's fair to ask project leaders to bring more empathy to the table in these cases -- to me, that is an important part of leadership. If a significant part of the community is up in arms about something, there likely is a kernel of truth to their argument, and the leadership should respond to the sentiment to prevent it from festering.

Update: interesting discussion on these topics on Reddit.

Taking the long view

To my taste, too much focus in 2018 was spent on releasing the edition. We could have spent more time addressing technical and organizational debt and released the edition 3 or 6 months later to little consequence. The Rust release process has been carefully engineered to optimize for cautious movement towards long-term goals -- and the edition process, in setting a fixed scope and an arbitrary hard deadline, subverted a number of these goals. This caused issues around the release of the website as well as problems with tools when 1.31 was released.

On the one hand, the cautious, deliberate approach to language design and compiler development have served us well, and will likely continue to do so. However, we should not forget that this process functions well in part due to an imposed feedback cycle that gives enough time and space for people to experiment before casting the new feature in (stable) stone per Hyrum's law. In a similar vein, we cannot expect to get our organizational changes right on the first try, and so we should not be afraid to experiment and iterate with the way the community works on the Rust project, rather than taking a long time to get it just right. This is especially true because the experiment period allows a wider community to participate and interact with the proposed change, thus reinforcing community alignment and improving feedback.

In this sense, we might learn from Jeff Bezos when he talks about a bias to action, saying that we should not spend too much time discussing decisions that are easy to change after the fact. For Rust, too, we want to remain at Day 1 for as long as we can -- let's remember the "without stagnation" part.

Contemplating Cargo

On a somewhat more technical note, I hope for more attention given to Cargo. While Cargo is a great tool in many ways, it also suffers from a number of usability cliffs where the behavior is hard to predict or understand. When I tried to contribute, I found that there's also a decent amount of technical debt, and my attempts to improve this through refactoring were met with such reluctance that I stopped contributing. From recent reports, it seems I'm not the only one.

Since Cargo is so central to the Rust experience, I think we should not undervalue it; as one example, the interactions between Cargo and rustc also influence the perception of those compile times people keep talking about. I was very happy to see the leadership changes and increase of the team's size, and I hope those changes will lead to further Cargo improvements this year.

Unused inventory, redux

Last year, I wrote at some length about the concept of unused inventory. This is still very much on my mind, and I was happy to see Niko talk about it in very similar terms. I hope we can improve things here.

Closing thoughts

I'm still as bullish on the Rust language as ever. Non-lexical lifetimes and the module system changes have made an already great language even better, async/await is going to be great. What I most want from Rust, though, is that more of the software in my life can enjoy its reliability and efficiency (in performance vs footprint), and for that we need adoption and sustainability. While the technical side of the project is working well, I've been concerned about the functioning of the project's governance and processes. A number of Rust 2019 posts have addressed similar concerns, which makes me hopeful for 2019.

Unused inventory

Avery Pennarun recently wrote an essay called An epic treatise on scheduling, bug tracking and triage; it's long, but if you're interested in software engineering process, I think it'll be worth your time. One of the points that really stuck with me is where he talks about unreleased software as inventory.

Avery discusses how Kanban was invented in the Japanese car industry, where minimizing inventory was one of the driving factors. As Avery explains, this very much applies to shipping software: unreleased code means you have spent the time to design and implement the feature, but now you have to pay for maintenance of this code (in terms of complexity in implementing other bugs or features) without realizing the value of shipping the actual feature to your customers. And that's not even discussing the opportunity costs of how you could have spent the time spent on unreleased feature A to ship released feature B sooner, increasing the value brought to your customers.

If you doubt the analogy, please go and read Avery's essay, since I cannot do it justice here. My point is this: rustc has a lot of unused inventory.

So I found myself agreeing with Nick Cameron's Rust 2018 post, where he describes his wish for 2018 to be "boring", because we should just finish up what we've got in the pipeline instead of starting new things. Pascal Hertleif's take on Rust 2018 is even stronger, calling for consolidation and rapidly reducing the number of in-flight unstable features. Currently there are 113 in the language and 155 in the library; see also my analysis of open library tracking issues.

If you think about that in terms of unused inventory: how much time has the community spent on designing and implementing these features? How much value, in return, has made it into the stable Rust compiler that most people use? How much time has been spent on maintaining these features while we were shipping other things? How many of the commits going into the master branch during any given release cycle actually affect stable Rust users' life six weeks later? When is an unstable feature actually used so much that it has effectively prematurely stabilized? For example, if there is agreement that the design can be improved, are we still willing to break the codegen features Rocket uses?

One challenge here is that Rust is an open source community, not a hierarchically-driven top-down organization where the leaders can just tell people what to do. Still, if the community can come together and agree on priorities, we can focus more on shipping features in stable Rust. We could adopt a rule that features cannot stay in nightly if no progress on stabilization is being made for more than 4 cycles. Or agree as a community that no more than 25 language and 25 library features can be in-flight at any time.

My other conclusion is that stabilizing features is a bottleneck in our factory for shipping Rust. So while we work on reducing inventory, we should at the same time try to increase the capacity of bottlenecks in the stabilization process.

Meta-ergonomics

I've been thinking about metacognition recently as a nice word for a useful concept. Similar to how metacognition means the awareness and analysis of one's own learning or thinking processes, and playing off of the 2017 year theme of ergonomics, maybe 2018 should be the year of going meta: meta-ergonomics, where we focus on the process of improving language ergonomics.

In order to scale the number of people that can help design, implement and stabilize new features and fix bugs, how can we best connect the high-level goals to the lower-level implementation process? Can we highlight the current blockers and accessible "good first bugs" where mentoring is available?

I subscribed to some issues in the GitHub roadmap issue tracker last year, but did not find myself deriving a lot of value over the course of the year. There were good write-ups of what needed to get done, but few updates over time and little connection to the ongoing implementation effort. The best way I've found to keep track of the non-lexical lifetime effort, for example, was just to check out links from This Week in Rust, which mostly talked about stuff that was completed rather than upcoming opportunities for contribution.

One important part has already been kicked off by Niko Matsakis: the Rust Compiler Book will hopefully become an important resource this year for helping people hack on the compiler. When I tried my hands at one small language feature this year, documentation like that was sorely lacking.

Deep docs

As with missing documentation for the compiler, the other area where Rust can improve next year is documentation. While The Rust Programming Language is a great resource for people just starting with Rust, I ran into a number of cases where it didn't fulfill my needs and I had to ask around for help.

The community is wonderful in providing such help, but at the same time it felt frustrating when there was no documentation to better describe the language's syntax and semantics: what edge cases are allowed or not, what parts remain unimplemented, why certain restrictions are there and what relevant RFCs are in the pipeline. I'm not sure whether this is intermediate-level documentation or reference documentation, or maybe those really are the same thing.

Web development

The big ticket item here is WebAssembly. I believe that WebAssembly is about to take off in a big way, particularly once it gets access to DOM APIs. My personal end goal here is to be able to write web apps where both the backend and the frontend are written in Rust. Ideally, the UI would leverage functional reactive programming (see Yew), so that the app's state lives on the client and the server just ships state updates as required. (Elm, but in a rustic way.) Lots of progress was made with WASM support in 2017, but making that really polished would make Rust a top contender for greenfield WASM projects (that is, not using legacy C/C++), which seems like an important use case going forward.

Even if you just want to write Rust on the backend, the infrastructure is still maturing. Rocket has many cool ideas, but only works on nightly and doesn't yet support asynchronous programming. Gotham seems to be the next viable option, on stable and with support for futures, but it seems to be in the very early stages (starting with the documentation). For simpler API services, it looks like hyper (hopefully soon with built-in HTTP 2 support) and serde work well together, although something with more polish and less boilerplate would be nice.

So far Askama, my take on templating for Rust, hasn't taken off in a big way (although I've been very happy with contributions!). I'm not sure why exactly; I'll keep iterating as I haven't seen any competition that makes more sense to me. I would like to explore how it can fit in with functional reactive programming -- if that ends up working out, it may also draw more people in.

In general, it doesn't feel like Rust is web yet, so I hope this will improve in 2018.

Paper cuts

And then there's a list of things I've ran into over time that I'd like to see some traction on somehow. Note that I do pretty much all of my development on stable Rust, and that contributes to some of these problems.

• When I tried to expose a parser result for imap-proto, I was surprised to find out you cannot access enum variants through type aliases. I wrote up a pre-RFC to help fix that; stabilizing that in 2018 seems like a challenge.
• In one case I wanted to use Vec::resize_default(), which has been waiting for stabilization for about 8 months now without any signs of progress.
• Installing clippy or rustfmt (as a stable user) and keeping them running takes hard work and troubleshooting time. Updating these tools means I have to update nightly, and the other way around. On the latest update, rustup warned me I should delete the separate installations to allow it to install the rustup-managed ones, but that actually didn't work.
• Writing a Tokio network protocol client (tokio-imap) took a lot of time, since the documentation focused much more on writing servers at the time. It feels like Tokio made little progress in 2017; I hope 2018 will be better.
• The ergonomics around futures are not where they should be. I hope impl Trait and async/await can make enough progress in 2018 to make this better.

Conclusion

I deeply believe in Rust; I've been trying to articulate that in another blog post (still in progress). I hope 2018 will be another great year for Rust, and I am eager to participate more in the Rust community over the coming year.