Featured Image

We built a COVID-19 Data Portal

October 9, 2020

Launch of the COVID-19 Data Portal

Today, we launched the COVID-19 Data Portal of the Lancet Commission on COVID-19. It provides access to the latest data available on COVID-19, such as new cases, deaths, positive test rate, and other valuable data to gauge the transmission and response to the pandemic. It features 3.5 million data points, covering 216 countries over a period of 284 days (and counting).

The data is pooled from different sources, including Johns Hopkins University, Our World in Data, Oxford, YouGov, and Google. You can download the full database (CSV) or see a list of the indicators and their descriptions. The data portal and database are updated every day (Monday to Friday). Over time, more heat maps and graphs will be added, including data to track the socio-economic impacts of the pandemic.

A screenshot of the indicator COVID-19 Overall Transmission on the data portal

COVID-19 Overall Transmission

Professor Jeffrey Sachs developed an assessment of the overall transmission rate of COVID-19 in countries. It takes into account two key measures of the pandemic, new cases per million and the positive test rate (the percentage of tests that are positive), to give an indication of where the pandemic is spreading and where it is being suppressed. The methodology is described in detail in the Lancet COVID-19 Commission Statement on the occasion of the 75th session of the UN General Assembly.

The assessment is available in the interactive map below. The map can also be accessed via the COVID-19 Data Portal: https://data.covid19commission.org/map/overall-transmission

How We Built It

We developed the data portal in collaboration with Professor Jeffrey Sachs and my colleagues Guillaume, Ismini, and Julie from the Sustainable Development Solutions Network. We have two codebases that power the data portal: A Python-based repository for scraping, processing, and compiling the database and a React-based repository for visualizing the data in our data portal.

Database

The Python scripts extract the latest data from a variety of sources (Johns Hopkins University, Our World in Data, Oxford COVID-19 Government Response Tracker, etc...), normalize country names and dates into a consistent format, and assemble everything into one overall database in CSV format.

The code is available under GPL-3.0 on https://github.com/sdsna/lancet-covid-19-database. The entry point is extract-indicator.py, which reads in the codebook and then triggers the corresponding pipeline from the pipelines folder for each indicator.

Data Portal

Our data portal is built with React and Next and hosted on Netlify. All content is pre-rendered, so that content is served fast and at the lowest possible cost. When used a GitHub on-push hook, that triggers a rebuild of the portal on Netlify, pulling in the latest data.

Like our Python scripts, the code for the data portal is available under GPL-3.0 on https://github.com/sdsna/lancet-covid-19-dashboard. We launched the data portal with six interactive heat maps (on transmission rate, new cases per million, new deaths per million, and positive test rates) and more will be added over the coming weeks.

About the Lancet COVID-19 Commission

The Lancet COVID-19 Commission has been created to help speed up global, equitable, and lasting solutions to the pandemic.
Professor Jeffrey Sachs, Chair of The Lancet COVID-19 Commission

The Lancet Commission on COVID-19 is an interdisciplinary initiative of leading thinkers, researchers, and practitioners, created to tackle the following four issues:

  1. Recommendations on how to best suppress the epidemic.
  2. Addressing the humanitarian crises arising from the pandemic.
  3. Addressing the financial and economic crises resulting from the pandemic.
  4. Rebuilding an inclusive, fair and sustainable world.

The work of the Commission will be supported by several task forces, in areas ranging from vaccine development, to safe workplaces, to global economic recovery. The Secretariat of the Commission is jointly hosted at the Center for Sustainable Development (CSD) at Columbia University, and the UN Sustainable Development Solutions Network (SDSN).

You can learn more about the Lancet COVID-19 Commission here: https://covid19commission.org/