Daniel Lemire

Daniel Lemire is a full professor in computer science. His research is focused on software performance, indexing techniques and data science. For example, he works on bitmap indexes and integer-compression techniques. He also works on database design and probabilistic algorithms (e.g., universal hashing). He likes to reflect on technology and its effect on our civilization. 

His work on bitmap indexes is used by companies such as eBay, Facebook, LinkedIn and Netflix in their data warehouses within big-data frameworks such as Google Procella (YouTube's database engine), Apache Hive, Druid,  Netflix Atlas, LinkedIn Pinot, Apache Spark and Apache Kylin. Git, the ubiquitous version control system uses his techniques to accelerate queries. Some of his compression software is used by Apache Arrow and Apache Impala. His indexing techniques are used to accelerate medical research. Some of his techniques have been adopted by Apache Lucene, the search engine behind sites such as  Wikipedia and frameworks such as Solr and Elastic. One of his hashing techniques can be found in Google's Tensorflow where it improves performance. His Slope One algorithm is a reference in recommender systems. In 2019, his research was featured in Quebec Science magazine: L’intelligence artificielle pour rendre les logiciels plus rapides (Making Software Faster with AI); it was also the subject of an interview at the Années Lumières show (CBC), Des serveurs informatiques plus rapides et moins énergivores (Faster and More Efficient Servers). With his collaborators in 2019, he wrote the fastest JSON parser in the world (simdjson): JSON is the de facto standard data interchange format on the Internet. The simdjson library showed for the first time that is possible to parse JSON at gigabytes per second correctly and completly. His simdjson library is one of the top hundred most popular C++ projects of all time on GitHub. The simdjson library is used by Facebook, by Shopify, by Intel, by Microsoft, by Apache Doris and by several other important systems such as Node.js.

He has written over 90 peer-reviewed publications, including more than 50 journal articles. His scientific work has been cited over 5,000 times.  He has held competitive grants for over fifteen years. He regularly serves as an expert in prestigious program committees (e.g., ACM CIKM, ACM WSDM, ACM SIGIR, ACM RecSys). 

He works regularly on open source software libraries. He programs regularly in C, C++, Java, JavaScript, Python, Swift and Go. He works primarily in an open-source setting (e.g., Linux). In 2012, he was rewarded by the Google Open Source Peer Bonus Program. In February 2019, he was ranked in second position among the most popular developers on GitHub and most popular in C++ (ahead of Microsoft, Google, Facebook). In 2020, Daniel Lemire was one of the top 100 developers most followed on GitHub. The GitHub site has 28 million developers.

The algorithm described in his article Fast Random Integer Generation in an Interval was adopted to accelerate random number generation by the Swift standard library, by the Go language, by the Julia language, by the GNU C++ library (Linux) and by Numpy (Python). The algorithm described in his article Faster Base64 Encoding and Decoding using AVX2 Instructions is used within PHP. The algorithm described in his article Faster Remainder by Direct Computation: Applications to Compilers and Software Libraries is used within the C# standard library to accelerate the Dictionary class.

Daniel Lemire and his colleagues wrote the Ada library which is maybe the fastest URL parser in the world. Ada improved the performance of the popular JavaScript runtime Node.js:

"Since Node.js 18, a new URL parser dependency was added to Node.js — Ada. This addition bumped the Node.js performance when parsing URLs to a new level. Some results could reach up to an improvement of 400%. As a regular user, you may not use it directly. But if you use an HTTP server then it’s very likely to be affected by this performance improvement." (State of Node.js Performance 2023)

He is a long-time social media user: his blog has tens of thousands of readers and was featured on slashdot, reddit and hacker news. He was one of the first Twitter users, he is followed by over 16,000 people : @lemire.

For years, he organized live talks open to the public in Montreal: tribalab and technolab. Following the Spring 2020 events, the talks have been moved mostly online.

Talks (YouTube)

NodeConf EU 2023

BID 2023

SPIRE 2021

Go Systems (San Francisco, 2020)

Performance Summmit III (Seattle, 2020)

QCon San Francisco 2019 (best voted talk!)

Spark Summit East 2017

Laboratory

We are lucky to have a fully equipped laboratory with a dedicated technician. We have a server farm that has been used worldwide for experiments in software performance (e.g., by researchers such as Agner Fog). Some of our machines have the following specifications:Microarchitecture Icelake : Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz

• Icelake microarchitecture: Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
• Haswell microarchitecture: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
• Knights Landing microarchitecture: Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz (64 cores)
• Skylake microarchitecture: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
• Skylake-X microarchitecture: Intel(R) Xeon(R) W-2104 CPU @ 3.20GHz
• IBM POWER9 2.2 GHz, 4-core
• Cannonlake microarchitecture: Intel Core i3-8121U CPU @2.20GHz
• Skylark microarchitecture (ARMv8): Ampere eMAG CPU 32 cores @ 3.3 GHz

We also have several workstations and space in our laboratory to explore virtual reality as a tool in data science.

Students and post-doctoral fellows

We are recruiting students and postdoctoral fellows. If you love writing crazily fast software and want to come to Montreal, drop us a line. Link to an impressive GitHub profile is an asset. Speaking French is necessary if you want to pursue an academic program with me at the Université of Québec, except maybe at the Ph.D. level where allowances can be made for strong students.  Some of our best students are women. We offer scholarships for graduate studies in software performance for data engineering (in French).

If you are a Canadian undergraduate student with at least a B average, you might be interested in coming to work with Daniel Lemire under an NSERC Undergraduate Student Research Awards. The awards help pay for a full time research project in our Montreal labs. The application deadlines are:

• March 1st for the Summer term;
• July 1st for the Fall term;
• November 1st for the Winter term.

It is an ongoing competition: we receive applications for every term. It is ok if you do not speak French. Please allow at least a week to put together an application with my help.

If you are interested in pursuing a master in information technology full-time under the supervision of Daniel Lemire in Montreal and you know some French, I take applications for NSERC Graduate Scholarships. You need to have a strong academic profile to apply. You should be a Canadian citizen or permanent resident of Canada. The deadline is December first of each year. You must plan ahead. We take applications every year.

If you are interested in pursuing a Ph.D. in cognitive computing full-time under the supervision of Daniel Lemire in Montreal and you know some French, we take applications for NSERC graduate scholarships. You need to have a strong academic profile to apply. You should be a Canadian citizen or permanent resident of Canada. The deadline is November 1st of each year. You must plan ahead. We take applications every year.

Moreover, all students finishing an M.Sc. thesis in information technology with us receive a scholarship, automatically. All students making progress on a doctorate in cognitive computer science receive automatic scholarships. Enrolment in a PhD program implies a waiver of tuition fees for foreign students. 

Daniel Lemire regularly supervises students, from the undergraduate to the Ph.D. level. He works primarily with students who love to program and who prefer an open source setting (e.g., Linux). Many of his students contribute to open-source projects on sites such as GitHub.

He recently supervised the following Ph.D. students: 

• Pierre Marie Ntang, cognitive computer science (graduated in 2023);
• Gary Germeil, cognitive computer science (graduated in 2022);
• Tarek Khei, cognitive computer science (graduated in 2020);
• Xueping Dai, environmental science (gradauted in 2019);
• Erick Aokou Koffi, cognitive computer science (graduated in 2018);
• Badis Merdaoui, cognitive computer science (graduated in 2017);
• Jing Li, computer science (graduated in 2016);
• Samy Chambi, computer science (graduated in 2016);
• Hazel Webb, computer science (graduated in 2010).

Several of his students occupy key positions: e.g., Maxime Boisvert (M.Sc., 2017) is Production Engineering Manager at Shopify, Shany Carle (M.Sc., 2017) is computer science professor at cégep de Victoriaville, Shira Smith is engineer at Discord in Californie.

Education

• Postdoctorate (Institute of Biomedical Engineering)
• Engineering Mathematics Ph.D. (University of Montreal and Polytechnique Montréal)
• Master in Mathematics (University of Toronto)
• Bachelor degree in Mathematics (University of Toronto), with High Distinction

Research Interests

• Data Science
• Data Indexing
• Data Engineering
• Software Performance
• Vectorization (SIMD)