Helping Delivery Robots Find Your Front Door [Prof. Mohit Bansal was interviewed for the story]

November 14, 2019

With a new navigation system from MIT, robots can decipher common landscape features, even in an unfamiliar environment

By Emily Matchar
November 13, 2019 12:03PM

elivery robots, once a sci-fi fantasy, became a reality this year, rolling along university campus sidewalks and suburban California streets, bringing pizza and Amazon packages right to customers’ front doors. They’re increasingly being seen as a solution for “last-mile delivery”—the part of the supply chain where goods are moved from a local transportation hub or warehouse to their final destination. This last leg is notoriously inefficient, causing traffic congestion and releasing outsize amounts of pollution. Robots, many think, could be a solution.

But how do robots find the door? It’s not always simple. GPS can take the robot to the right address, but it can’t tell it whether the door is to the left of the garage or at the end of the garden path.

That’s why researchers at MIT have developed a new robot navigation system. The system involves training the robots to recognize environmental features like driveways and mailboxes and to learn which features are likely to lead to a door.

“It’s kind of unreasonable to expect you’d have a detailed map of every single environment your robot was going to operate in,” says Michael Everett, a graduate student in MIT’s department of mechanical engineering who worked on the research. Instead, the team asked, “how do you drive around and find objects when you don’t have a map ahead of time?”

The answer involves using an algorithm that pulls features—”door” or “stairs” or “hedge”—from pictures and makes new maps of the environment as the robot moves. The maps use both the semantic label (ie, “door”) and a depth image. The algorithm allows the robots to make decisions based on the maps, which helps them reach their destination more quickly.

The researchers trained the algorithm on satellite maps from Bing. The maps showed 77 houses from three suburban neighborhoods and one urban one. Everett color-coded the maps based on feature—sidewalks yellow, driveways blue, hedges green, doors gray. He trained the program using both complete images of the landscape and images that were partly covered, since a moving robot will often have its view partially obscured by street features, cars or pedestrians.

Everett and his team then developed a “cost-to-go estimator” algorithm for choosing a path of maximum efficiency (and thus minimum “cost”). This algorithm created a second map, this one in greyscale. On the map, darker locations are farther from the goal, lighter locations are closer. A road or sidewalk might be darker, while a driveway would be lighter and lighter the closer it gets to the front door. The front door—the destination–is the lightest. This cost-to-go estimator map helps a robot make informed decisions on the fly.

The team tested the algorithms using a simulation of a house that hadn’t appeared on the training images. They found that their technique helped find the front door 189 percent faster than traditional navigation algorithms, which rely on complete maps and specific GPS coordinates. While the algorithms that currently drive most delivery robots do generally get them to the destination, they’re not always efficient.

“This MIT navigation system is an important step in this overall direction of faster real-time navigation and delivery,” says Mohit Bansal, a professor of computer science at the University of North Carolina at Chapel Hill who was not involved in the research.

Bansal says the next hurdle for developers of delivery robot systems will be to enable robots to handle longer commands, including commands with negation (such as “don’t go to the side door”). Another challenge will be developing robots that can ask questions if they get lost or confused.

The MIT team hopes that their algorithm could one day be used to help robots find things in completely unfamiliar environments. Imagine a robot that could understand the command “find my shoes” or “take this letter to the nearest post office.”

“My vision there is that all our robots are going to be able to just understand really casual human instructions like, ‘hey, robot, go grab a coffee for me,’” Everett says.

Everett presented his findings earlier this month at the International Conference on Intelligent Robots and Systems in Macau. It was a finalist for a “best paper award” in cognitive robotics, a prize given to promote “advancements of cognitive robotics in industry, home applications, and daily life.” The work is partially funded by the Ford Motor Company, which is developing its own delivery robots programs.

Currently, the navigation system works best in environments with a lot of structure. The suburban neighborhoods on the training maps tend to have predictable features–sidewalks leading to driveways leading to front doors.

“If you’ve been to one house, you have a pretty good idea of what the other houses look like,” he says.

This means the navigation system would likely work well in ordered environments like hotel corridors or airport terminals, but perhaps would have more trouble in, say, a historic city center where buildings are built in dramatically different styles.

“At the end of the day, we want to see if the algorithm can handle the uncertainties and noise that the real world has,” Everett says.

We’ll be waiting right here for that robot-fetched cup of coffee.

UNC CS demos autonomous vehicle research for Safe Systems Transportation Research Forum

November 13, 2019
Sridhar Duggirala and students Charlotte Dorn and Abel Karimi congressional staff as part of the UNC-Chapel Hill Safe Systems Transportation Research Forum
Sridhar Duggirala and students Charlotte Dorn and Abel Karimi show an autonomous vehicle to congressional staff as part of the UNC-Chapel Hill Safe Systems Transportation Research Forum
Graduate student Abel Karimi demonstrates an autonomous vehicle's ability to avoid obstacles
Graduate student Abel Karimi demonstrates an autonomous vehicle’s ability to avoid obstacles

On October 11, 2019, UNC welcomed congressional staff as part of the UNC-Chapel Hill Safe Systems Transportation Research Forum. UNC Computer Science faculty member, Sridhar Duggirala, alongside colleagues from the UNC Highway Safety Research Center and RENCI, modeled how they design and test autonomous vehicle safety software.  During the presentation, Duggirala and his students talked about virtual testing and evaluation of autonomous vehicles, explaining how they use the data gathered to make advancements. Staffers were able to share information about ongoing works on developing test tracks for autonomous vehicles and bridging the gap with virtual testing and real-life testing. This opportunity to share current research with lawmakers was part of the larger Forum, but is representative of the many stakeholders eager to see the progress of research at UNC.

HackNC 2019 celebrates largest hackathon yet

October 25, 2019
HackNC competitors work on projects in Woollen Gym
HackNC 2019 competitors work on projects in Woollen Gym. Photo by Austin Wang.

Hundreds of undergraduate students gathered in Woollen Gym on October 12 for HackNC 2019, the largest hackathon held annually at UNC-Chapel Hill. 

Hackathons are time-limited, team-based coding competitions involving both software and hardware. Students at HackNC form teams, come up with an idea, and spend 24 hours building that idea, learning skills from free workshops, eating food from several local restaurants and spending time with their fellow coders. The 2019 event was the largest yet, featuring 650 hackers, 96 project submissions, 50 industry representatives at the sponsor fair, a women in tech panel, an inflatable Mobile Planetarium exhibit from the Morehead Planetarium, a basketball shootout with Capital One, multiple palettes of free succulents and more.

A team of students at HackNC 2019 works to solve a problem
A team of students works to solve a problem at HackNC. Photo by Austin Wang.

Workshops at HackNC 2019 were led by representatives from sponsor companies and by volunteer mentors. Topics included backend development in Python, pitching research projects, cloud architecture for healthcare, tips for getting a job at a tech company, machine learning techniques for chemistry, user experience advice and more.

This year’s winning hack was SafeWallet, an application that allows parents and caregivers to set limits on how much money their dependents can spend, as well as where they can spend it. Using the web interface, users can add funds and adjust spending constraints on the fly. The project was developed by UNC students Vibhu Ambil, George Dimitrov, Sahil Patel and Tyler Youngberg, who said they had to learn a lot during the hackathon to pull the project together.

Tyler Youngberg, George Dimitrov, Vibhu Ambil and Sahil Patel accept first place prizes from organizer Shreya Gullapalli at HackNC 2019
Tyler Youngberg, George Dimitrov, Vibhu Ambil and Sahil Patel accept prizes from organizer Shreya Gullapalli at HackNC 2019. Photo by Austin Wang.

Youngberg said the team members were shocked when they heard their name called as the first prize winner. The team wants to continue working on the project after HackNC, and their next step is to enable smartphone payment using near field communication (NFC) in a secured mobile app.

Another impressive project was RxTranslate, which took 2nd place, as well as the Wolfram Award, Best Health Hack, and Best Hack Empowering Minorities. RxTranslate is an application that addresses health literacy by helping users consolidate and document their prescriptions, translate directions for use into other languages and avoid negative drug interactions. It was developed by Tiffany Nguyen and Maria Ortiz, two students at East Carolina University, after Ortiz witnessed a family member suffer the effects of a drug interaction. Ortiz said it was an amazing experience learning experience and growth opportunity to work on the project.

Tiffany Nguyen, Maria Ortiz, and Shreya Gullapalli at HackNC 2019
Tiffany Nguyen and Maria Ortiz accept prizes from organizer Shreya Gullapalli at HackNC 2019. Photo by Austin Wang.

“It was an inspiration to know that we can use our knowledge to aid people in avoiding these problems,” Ortiz said. “Computer science is a great field, and I wonder what we can accomplish with all this knowledge and experience.”

Nguyen said that she and Ortiz hope to expand RxTranslate to use machine learning techniques to automatically extract prescription information from pill bottles and search the web for known drug interactions with a categorization of the danger posed. They want to build it into a mobile application that allows users to make an account and store things like known allergies, while also conforming to HIPAA standards for the storage and flow of healthcare information.

While Nguyen is a veteran of hackathons, Ortiz and all four developers of SafeWallet were competing for the very first time. One of the goals of HackNC is to be a welcoming environment for computer science students of any experience level to push themselves and hone their skills, while building community and growing together with other students.

“I had no idea what to expect going in, but I really enjoyed it,” Youngberg said. “It’s not at all intimidating. Even if you’re brand new, you can learn a lot and have a good time.”

Ambil wanted to give credit to the environment that the organizers had created at HackNC.

“The resources and workshops were really great, and we were able to learn a lot on the fly,” Ambil said. “You could come to the event just to learn and still have a great time. We were able to learn a lot from each other and push each other to new levels, because of that.”

For more information about HackNC visit

2019 Carolina Data Challenge draws 250 students for third annual data science hackathon

October 21, 2019
Carolina Data Challenge participants gather in Chapman Hall for the opening ceremony
Carolina Data Challenge participants gather in Chapman Hall for the opening ceremony
a team of students works on their project in the lobby of Sitterson Hall
A team of students works on their project in the lobby of Sitterson Hall

Sitterson Hall played host to more than 250 students during the third annual Carolina Data Challenge, which took place October 5-6, 2019. This year’s event was the largest yet, with participation up 60 percent from 2018.

Carolina Data Challenge is a 24-hour data science hackathon run by the Carolina Analytics and Data Science (CADS) Club and hosted by the UNC Department of Computer Science. Participants have the opportunity to “hack” on a dataset from either the financial, technology or non-profit sector, and prizes are awarded to the teams who provide the best data visualization, most valuable insights and best use of outside data, as well as to the top beginner team.

In addition to being a competition, the event is also a valuable opportunity to develop data science skills and experience, build community with fellow students and network with potential employers. No prior experience in data science is necessary, and there are numerous resources for beginners, including industry mentors and workshops and practice problems to develop skills. Prior to the event, CADS coordinated an introductory workshop and resume review session that was open to all students.

Carolina Data Challenge is one manifestation of the rapidly growing interest in data science at UNC-Chapel Hill. As the university plans to launch a new data science initiative, the 2019 hackathon featured 51 total submissions, and more than 70 percent of the participants were UNC students.

Carolina Data Challenge 2019 was sponsored by CapTech, Valassis, the Renaissance Computing Institute (RENCI) and the National Consortium for Data Science, Fidelity, SAS, Barings, the Institute for Advanced Analytics, EY and Quantworks. The keynote address was given by Elliot Inman of SAS.

UNC CS goes to 2019 Grace Hopper Celebration

October 17, 2019

Women from UNC CS pose for a photo at Grace Hopper Celebration 2019 in Orlando, FloridaThe 2019 Grace Hopper Celebration (GHC), “the world’s largest gathering of women technologists”, hosted by, was held October 1-4 in Orlando, Florida. Over 50 UNC CS women attended the conference, including 40 who received scholarships from the department to attend. The group was also joined by UNC CS alumni who attended and even presented workshops at the event.

The Grace Hopper Celebration offers students the opportunity to engage with potential employers, interview for internships and full-time positions, attend workshops and other sessions from key industry experts, and network with fellow women in tech.

Shannon Goad, an attendee and senior majoring in computer science and mathematics, shared, “The most impactful part of Grace Hopper Celebration for me was seeing all of the attendees uplift each other during the intense networking and interviewing. There was a spirit of mutual support instead of competition, and everyone was truly celebrating each other’s successes. It was great to feel so welcomed by a community in tech, and know that everyone was genuinely cheering each other on.” 

Conference attendees gained important resources to find success and further a career in tech. More than 80 percent of attendees are offered a full-time job or a first or second internship as a result of attending the conference. 

Those in attendance also have the opportunity to attend workshops and lectures from some of the women leaders in tech. GHC 2019 attendee and junior computer science major Sarah Bost, was excited and inspired by attending a fireside chat with CEO Brenda Darden Wilkerson and COO Jacqueline Bouvier Copeland.

“These women have dedicated their careers to promoting diversity in technology, and it was so inspiring to hear about their experiences and their vision for the future,” Bost said. “As a young woman starting my career, the future of technology is really important to me. Being at a conference where all of the top tech companies and students were passionate about’s commitment to achieving a workplace that is 50/50 by 2025 allowed me to feel excited about my future career in a more gender-neutral workplace.”

Attending the Grace Hopper Celebration can be quite costly and out-of-reach for many college-aged students. In 2016, the Department of Computer Science, with support from corporate partners, began offering scholarships for students to attend the Grace Hopper Celebration. Recognizing the importance of providing future leaders and women in tech with the opportunity to connect with industry representatives that are supportive of a more open and accepting tech community, there was a commitment to ensure UNC CS was represented as a leader committed to the development of women in tech. Opportunities such as these can truly be life changing for students. With the help of partners and donors, the department has sent more than 160 women to four Grace Hopper Celebrations.

Thanks to CapTech and CoStar Group for supporting UNC CS at the 2019 Grace Hopper Celebration!

UNC researchers receive $941,000 NIH R01 grant to utilize voice assistant devices in post-treatment cancer care

October 3, 2019
Clockwise from top: Shahriar Nirjon, Lixin Song, Mohit Bansal
Clockwise from top: Shahriar Nirjon, Lixin Song, Mohit Bansal
Examples of decision-making and natural language feedback from AURA to patients
Examples of decision-making and natural language feedback from AURA to patients

A team of UNC-Chapel Hill researchers led by assistant professor Shahriar Nirjon (Computer Science) was awarded a National Institute of Health (NIH) R01 grant worth $941,000 over four years. The grant supports research to monitor and provide feedback to post-treatment cancer patients, using a combination of radio and audio sensing technologies.

The project, dubbed “AURA” (a combination of AUdio and RAdio), is a collaboration between Nirjon and Mohit Bansal in the Department of Computer Science and Lixin Song in the School of Nursing. Existing audio-based voice assistant devices, such as Amazon Echo and Google Home, will be augmented with radio-based RF sensing technologies, enabling the devices to record more information from their surroundings and provide contextually relevant feedback to the patient. The device will combine audio and radio sensing to gather relevant patient information automatically and interactively, and store patient’s health records into an electronic system used by the entire care team, including the patient, family members, caregivers and healthcare providers at remote sites. The project will study the performance of AURA when deployed in the homes of post-surgery colorectal and bladder cancer patients, but AURA’s design is generic and extendable to meet the requirements of a wide range of post-treatment self-care scenarios.

The grant is awarded through a cross-agency program called the National Science Foundation Smart and Connected Health (NSF SCH) program. NSF SCH is an extremely competitive program, the purpose of which is to support the development of technologies, analytics and models supporting next-generation health and medical research through high-risk, high-reward advances in computer and information science, engineering and technology, behavior and cognition. Throughout the four-year project, Nirjon will lead research on radio-frequency-based human activity recognition, Bansal will lead research on natural language feedback and Song will lead the study of post-treatment colorectal and bladder cancer patients using the developed technology.

Department of Computer Science opens doors for 2019 Middle School/High School Open House

September 27, 2019
Undergraduate student Victor Murta shows the Nao robot during the 2019 MS/HS Open House
Undergraduate student Victor Murta demos the Nao robot to a group of visitors in the Computational Robotics Lab

The UNC Department of Computer Science welcomed hundreds of visitors to its fourth annual Middle School and High School Open House on Saturday, September 14. The outreach event allows attendees to see demonstrations of computer science research and hardware and learn about how to prepare to study computer science in college.

Inside Sitterson Hall and Brooks Building, guests could experience 22 unique demos spread across three floors, including seven different laboratory spaces. These included virtual and augmented reality devices, a humanoid robot, automatic object detection and language generation from video, an autonomous RC car, a hackable arcade game used in undergraduate computer security courses, a 3D printer, a data visualization tool utilized by radiologists, and much more. Demos were staffed by undergraduate students, graduate students, and faculty members from the Department of Computer Science, giving visitors the opportunity to ask questions about department life as well as research.

High school students and their parents could sign up for one of three 45-minute information sessions to learn about computer science at UNC and ask questions about the major and preparing for college. The information sessions were led by Department Chair Kevin Jeffay and featured a panel of undergraduate student ambassadors.

The Middle School and High School Open House is one of the department’s two open house outreach events held each year. If you are interested in learning more about computer science at UNC, we invite you to visit Sitterson Hall and Brooks Building during the UNC Science Expo on April 4, 2020. The next Middle School and High School Open House will be held on a Saturday in the fall of 2020. You can learn more at

For information on applying to UNC-Chapel Hill as an undergraduate student, visit

Focus Carolina: Mohit Bansal

September 24, 2019

Mohit Bansal is an assistant professor in the computer science department in the College of Arts & Sciences at the University of North Carolina at Chapel Hill. He is also director of Carolina’s Natural Language Processing Lab.

Jane Calloway, Sunday, September 22nd, 2019

Mohit Bansal is an assistant professor in the computer science department in the College of Arts & Sciences at the University of North Carolina at Chapel Hill. He is also director of Carolina’s Natural Language Processing Lab. The lab’s research interests are in natural language processing and machine learning, with a particular focus on developing human-like language generation and dialogue models, and multimodal artificial intelligence agents that combine language with vision and robotics.

Read the transcript of this episode

Peter Hase Awarded Prestigious Royster Society Fellowship

August 26, 2019
Peter Hase

Peter Hase

First-year computer science doctoral student Peter Hase was awarded the Royster Society of Fellows Recruitment Fellowship by the UNC Graduate School. The fellowship provides full tuition cost, health insurance, and funds for professional travel for five years, as well as a $24,000 stipend for two years and a partial-stipend for three to encourage external fellowships. Beyond funding support, key benefits of membership in the society of fellows also include interdisciplinary learning, networking, and professional development and social opportunities. Among these is the opportunity to teach an interdisciplinary seminar for first-year undergraduates.

The Royster fellowship program selects students with the highest academic potential and the most impressive record of achievement in undergraduate education and work and life experiences. It is the University’s most selective and prestigious interdisciplinary fellowship program.

Hase’s research interests lie in developing interpretable machine learning methods, with a focus on the domain of natural language processing (NLP). An important goal of this research is the design of machine learning systems that make decisions in a way that is transparent to people and open to scrutiny. He will be working on these kinds of problems at UNC with professor Mohit Bansal in the UNC-NLP Lab.

Hase previously received a bachelor’s degree in statistical science from Duke University, where he did research on interpretable computer vision techniques and algorithmic poetry generation with professors Cynthia Rudin and Sayan Mukherjee. His work on interpretable computer vision will be published this year at the 2019 AAAI Conference on Human Computation and Crowdsourcing (HCOMP). He attended Duke as a recipient of the AJ Tannenbaum Trinity Scholarship, a four-year merit scholarship. As an undergraduate student, Hase was also heavily involved with the school’s Effective Altruism club, a student group whose mission is to build an enabling community for students who are trying to find careers where they can improve the world. After leading Duke’s club for two years, he now plans to work with UNC’s chapter.

For a full list of current fellows, visit the Graduate School website. For more information about Peter, visit his webpage.

Four NIBIB grantees win prestigious Presidential early career awards

July 23, 2019

Four NIBIB grantees are among more than 300 recipients of the prestigious Presidential Early Career Award for Scientists and Engineers (PECASE) announced by President Donald J. Trump on July 2, 2019, and to be awarded at a ceremony on July 25.  The PECASE is the highest honor bestowed by the U.S. government to outstanding scientists and engineers who are beginning their independent research careers and who show exceptional promise for leadership in science and technology. Recipients announced this year hail from universities in 38 states across the country.

“The PECASE award is a national honor that puts a spotlight on an exceptionally talented group of NIBIB grantees at a time of unprecedented breakthroughs in advancing human health,” said NIBIB Director Bruce J. Tromberg, Ph.D. “These promising young scientists span the country and a fascinating spectrum of bioengineering research, from biomaterials to biomedical devices. Each awardee represents promising talent whose commitment to their innovative projects will engineer the future of health.”

Darren J. Lipomi, Ph.D., associate professor of nanoengineering, University of California, San Diego, is a 2015 PECASE nominee, just awarded this year. His NIBIB grant is an NIH Director’s Pioneer Award (DP2EB022358) supporting scientists who undertake novel approaches to major challenges in biomedical research. Lipomi develops wearable and implantable medical sensors, including stretchable, biodegradable, and self-healing semiconducting polymer materials. Stretchable electronics are designed to seamlessly integrate with the body contours to monitor vital signs, muscle activity, metabolic changes, and organ function. His project aims to create a new class of semiconducting polymer material that has the mechanical properties of human skin. This transparent electronic skin will be soft and elastic, sense contact, absorb blunt force, and will self-heal when damaged—all the while providing continuous and wireless health-monitoring.

Ron Alterovitz, Ph.D., professor of computer science, University of North Carolina at Chapel Hill, is a 2017 PECASE nominee. His NIBIB grant (R01EB024864) aims to improve the survival rate for lung cancer by enabling earlier stage diagnosis using a novel robotic device. The project is creating a new robotic system that deploys a needle that can semi-automatically steer through lung tissue to safely biopsy nodules. Currently used instruments cannot accurately access many nodules. The innovative robotic system will enable access to nodules throughout the lung, increase targeting accuracy, and avoid major bleeding by steering the needle around larger blood vessels. The project brings together a multidisciplinary team that spans expertise in interventional pulmonology, cardiothoracic surgery, radiology, mechanical engineering, and several subareas of computer science, including artificial intelligence and medical image analysis.

Xudong Wang, Ph.D., professor and associate chair of materials science and engineering, University of Wisconsin – Madison, is a 2016 PECASE nominee. His NIBIB grant (R01EB021336) supports the development of self-powered implantable biomedical devices for continuous, real-time sensing, monitoring, and other vital health functions. A variety of energy sources in the human body, such as limb movement, respiration, and heartbeat can provide sufficient power for small biomedical devices. The project explores innovative nanotechnology to create self-sufficient power supplies for implantable devices used in areas such as wound healing and weight control.

Angela K. Pannier, Ph.D., professor of biological systems engineering, University of Nebraska-Lincoln, is a 2017 PECASE nominee. Her NIBIB grant (DP2EB025760) is an NIH Director’s Pioneer Award. Her lab is developing more than 10 projects related to biomaterials and gene delivery systems. The award will support development of novel methods that improve use of adult stem cells in gene therapy, a promising tool for treating a variety of diseases.

Established in 1996, the PECASE acknowledges the contributions scientists and engineers have made to the advancement of science, technology, education, and mathematics (STEM) education and to community service as demonstrated through scientific leadership, public education, and community outreach. The White House Office of Science and Technology Policy coordinates the PECASE with participating departments and agencies.The PECASE awards ceremony will take place the morning of July 25, 2019, at Daughters of the American Revolution, Constitution Hall.  NIBIB-nominated recipients will be celebrated at an NIBIB seminar on the NIH campus on the same day.

Read the July 2, 2019, White House announcement here.