We are a research group at the University of Chicago, led by Prof. Junchen Jiang and located at John Crerar Library.

We work on machine learning for performance improvements in across different hardware and software systems. In this area, we have tackle areas in video streaming systems, tradtional networking systems like datacenters and satellite networks. We also work on building efficient systems for serving the latest applications such as Large Language Model (LLM) inference by improving for both lower latency and user perceived quality.

demo

Meet the members

Latest News

Hanchen presented CacheGen paper at SIGCOMM'24.
Junchen's talk on Knowledge-Delivery Networks at ByteDance and APNet'24.
Yuhan Liu presented ChameleonAPI at OSDI'24.
Zhengxu Xia successfully completes his PhD candidacy exam at UChicago.
Yihua Cheng starts his internship at Conviva.
See all posts

Ongoing Projects

Using Cached Knowledge for Large Language Model Serving with CacheFuse
Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input.
May 21, 2024
Using Cached Knowledge for Large Language Model Serving with CacheFuse
Resource allocation for Multi-Tenant Retrieval Augmented Generation Systems
Retrieval Augmented Generation (RAG) is the recent state-of-the art paradigm which lets Large Language Models (LLMs) generalise to text generation tasks in new domains, for which they have not been trained.
May 1, 2024
Resource allocation for Multi-Tenant Retrieval Augmented Generation Systems
Knowledge Streaming from LLMs to Environments
Nowadays, not only can we chat with chatGPT, but we can also let it write and execute code to do more complex tasks (e.g. letting chatGPT plot the boundary of America using Python).
Apr 1, 2024
Knowledge Streaming from LLMs to Environments
See all