Engineering Update: The All Requests Page

The engineering team at NextRequest has been hard at work to bring a brand new NextRequest into the world.

We’ve been working in, on, and with the NextRequest codebase for almost five years and while we’re super proud of it and how it’s helped governments and other institutions across the country fulfill public records requests, the truth is it was time for a change. After we launched RapidReview last year, we took the lessons we learned in supporting complicated, technically demanding workflows and decided to apply them to the rest of the NextRequest application.

First, we focused on the Documents page. This page helps the public find documents that have already been uploaded and released, potentially finding what they’re looking for without filing a request. Our customers have uploaded millions and millions of files ranging from emails to pdfs to video, so we needed to make sure the technology we relied upon was up to the task. Like other companies from Cisco to Netflix, we had already been using Elasticsearch to power our searches, but we leveraged their functionality even further to support complex filters and help ease the burden on our database. This means faster and more accurate search results.

We’re happy with the result of our efforts, and the numbers don’t lie: in the months since the refreshed Documents page launched, we’ve seen throughput in the application increase by 117%! Throughput is a measure of the number of requests sent to the server by users in a given minute. Since throughput is a measure of the number of requests sent to the server by our customers in a given minute, this increase demonstrates that our customers are working more efficiently and effectively in the app, since they are not wasting time waiting for the page to load.
‍

As you can see, we also reworked the look & feel of the page, based around our new NextRequest design system: Pnyx. We’ll talk more about Pnyx in a future blog post, but it’s been a huge part of the refresh so far, guiding our design choices, unifying accessibility and usability across the application, and helping us move faster as a development team. Specifically, Pnyx has a robust reusable component library that means we aren’t reinventing the drop-down every time. This has helped us get these pages out to our customers faster, and to be able to quickly respond to the valuable feedback they’ve provided.

But we’re far from done with the Documents page. This new architecture and design system means we will be able to more rapidly iterate on this page, bringing improvements to all our customers and their constituents who rely on NextRequest to access important public information.

Once the Documents page was released to the public, we turned to the All Requests page. We knew this page was going to be a challenge since so many people came to that page with different needs. From intrepid reporters looking to see the goings on within their local government to city clerks supervising dozens of requests at once, the All Requests page is a crucial part of the NextRequest experience. And so we started by talking to our users. We performed research, sent out surveys, and collected past feedback on the page to make sure our next steps were grounded in the real-world experience of people using this page. We continued this research⇒design⇒improve cycle throughout the weeks of development, right up to the release to production.

Just like the Documents page, the All Requests page has to be able to handle large datasets. Our customers have fulfilled hundreds of thousands of requests in NextRequest, and finding the right one quickly can make or break a workflow. Again we turned to Elasticsearch, not only to help our users search the requests themselves, but also to power the search filters that our power-users employ to immediately identify the request they’re looking for.
‍

We were also able to reuse many of the visual patterns from the Documents page, again by relying on the reusable components in Pnyx. As an added bonus, as we improved the components for use on the All Requests page, those improvements were incorporated into the Documents page. It was a great feeling to experience this positive feedback loop while developing the All Requests page, and we’re excited to continue to bring more of those improvements to everyone who uses the NextRequest application.

And although the All Requests page has been live for less than a week, we’ve already started seeing performance improvements. Even our 95th percentile load times (load times for the largest queries with the most data) are being returned by our search engine in under 0.025 seconds, more than an order of magnitude faster than the old architecture could perform under similar conditions.

What’s next for the NextRequest engineering team? We’re going to take a minute to celebrate a successful release, then get right back to work! We plan to bring even more new pages in the coming weeks and months. Our most ambitious goal is to refresh the Request page itself. This is the page at the heart of the NextRequest experience, and we intend to be deliberate and intentional about how we work to improve it. We’ve already started researching and planning, and just like with the previous two pages, we’re going to be relying heavily on feedback from our users throughout the process.

Big thanks to everyone who helped us, especially all out beta testers who gave us such great feedback along the way! If you want to see these new pages in action, check out Oakland’s Documents and All Requests pages.

Yohanan, Ruben, Rachel, Jordan, Imanol, Eric, Andy
-NextRequest Engineering team