3D Display Could Bring Touch to the Digital World

Andrew Corselli

Imagine an iPad with a surface that can morph and deform, allowing you to draw 3D designs, create haiku poems that jump out from the screen, and even hold your partner’s hand from an ocean away.

Such is the vision of a team of researchers from the University of Colorado Boulder. In a new study — funded by the National Science Foundation and published in Nature Communications — the group has created a one-of-a-kind shape-shifting display that fits on a card table. The device is made from a 10-by-10 grid of soft robotic “muscles” that can sense outside pressure and pop up to create patterns. It’s precise enough to generate scrolling text and fast enough to shake a chemistry beaker filled with fluid.

“As technology has progressed, we started with sending text over long distances, then audio and now video,” said co-lead author Brian Johnson. “But we’re still missing touch.”

The project has its origins in the search for a different kind of technology: synthetic organs. In 2017, researchers led by Professor Mark Rentschler developed what they call sTISSUE — squishy organs that behave and feel like real human body parts but are made entirely out of silicone-like materials.

In developing that technology, however, the team landed on the idea of a tabletop display. The research is part of the Materials Science & Engineering Program.

Other research teams have developed similar smart tablets, but the CU Boulder display is softer, takes up a lot less room, and is much faster. Each of its robotic muscles can activate as much as 50 times per second.

Here is an exclusive Tech Briefs interview — edited for length and clarity — with Rentschler.

Tech Briefs: I’m sure there were too many to count, but what was the biggest technical challenge you faced while developing this technology?

Rentschler: A couple of the key challenges really were the integration of everything into a form factor that really was able to help us demonstrate the capabilities of this approach. I’ll put that as 1A, and 1B would really just be developing the novel sensing capabilities and integrating that into the system.

Tech Briefs: Would you mind explaining in very simple terms how it works?

Rentschler: Yes, absolutely. So, at a basic level, we have a system that’s about the size of a board game — a couple of feet by a couple of feet wide, and it’s only roughly two to three inches tall. And it’s an array of pixels really, so 10 pixels by 10 pixels, but we've set this up as a modular system. So, each pixel has a number of actuators below it that are stacked together, and that allows the pixel to go up and down. It’s a small cell, and that small cell is roughly two inches by two inches wide, and that cell can go up and down based on what it is sensing and based on the control scheme that we have under it. So, within that cell, we have the actuators that go up and down, but we also have sensing embedded in it.

We have magnetic particles across the surface of the top of it, so we can measure the deformation either vertical as it is, expanded upwards, or also if anything pushes down on it, we can sense that. So that's an individual cell, and then we have a row of them, so a 1x10 array of these cells in a row, and that’s what we consider a module. And then we have 10 of those rows stacked together, or 10 of those modules together in the display that we have in the paper.

And then we have a soft silicone layer, that's the black layer that is spread across the whole thing. It’s a very thin sheet, but it allows it to be a continuous surface. And so we can manipulate a continuous medium, one being liquids on the surface using these more discrete actuators that are underneath of it.

Tech Briefs: You’re quoted as saying you could use these artificial organs to help develop medical devices and or surgical robotic tools for much less cost than using real animal tissue. In developing that technology, however, the team landed on the idea of a tabletop display. How did you decide on the tabletop display?

Rentschler: Originally we were thinking that we could make these individual cells — and we knew we were going to end up making them much larger than a human cell to begin with, and they're not biological, but we were trying to replicate the muscles and sort of the skeletal material and also the nerves or the sensing that are in human tissues — and ultimately try to make it smaller so that we could replicate organs from the body to help you develop robotic systems or medical devices. And that really comes from my background working with clinicians to design medical devices or robots to help them solve their problems. So that's where we started, and the whole concept for us was really to begin with looking at organs that had sort of sense and react on their own.

For us, one of the interesting things was that the gastrointestinal tract; there you’re looking at the colon or the small bowel, and when those organs sense things inside of them, there's peristaltic forces where they clamp down on the material to move it through the body. And so that was the original concept. As we started to develop these individual cells, we started to realize all these other applications, whether that was haptic feedback and sort of consumer electronics or manufacturing, being able to manipulate objects sort of on a conveyor belt or sorting or even some interesting things that we hadn't thought of to begin with — sort of shape morphing for aerodynamics.

So, you can think of either cars or jet airplanes with things moving quickly through the air that need to change their shape to enhance the performance going into a turn or things like that. That really led us to making what I’d say is a more generalized display to demonstrate a number of these different capabilities before we go off into individual directions to solve specific problems.

Tech Briefs: You mentioned the team is now focusing on shrinking the actuators to increase the resolution of the display, almost like adding more pixels to a computer screen. And you also said that the group is working to flip the display inside out. That way engineers can design a glove that allows you to feel objects in virtual reality. How are those two projects coming along?

Rentschler: From an actuator standpoint, a lot of that research is definitely moving forward. There is sort of a sweet spot of how much effort to push into making the individual pixels a lot smaller or making them, you know, I'll say half that size so that it’s still not an integration nightmare just yet, but we can definitely make the actuators at least half the size they are, if not a quarter of the size they are, and still get really nice performance today.

There’s a whole other area of research that the team has been working on just to make the individual pixels a lot smaller and then integrating them into specific applications. So, I don’t have any specific updates right now. Those things will be coming; it’s an exciting time for us.

Technology Enables Digital Communication Through Touch

Applying Friction Principles to Finger-Device Interaction Mechanics

Tech Briefs: What are your next steps? Any other future research work, etc., on the horizon?

Rentschler: It's really everything I just mentioned. We’re looking at ways that we can integrate this into sort of consumer electronics and spin it out into products or work with partners on that. We’re looking at some interesting things, I would say, from a medical standpoint of really trying to take that next step toward creating these simulated systems.

Then, I think a really exciting area for us is the shape morphing. The ability to integrate this into surfaces — whether that’s an airplane jet, a wind turbine, things that are moving through the air — could benefit from sort of sensing and reacting, and continuous shape morphing is a really powerful area for us as we go forward.

Tech Briefs: Do you have any advice for engineers aiming to bring their ideas to fruition?

Rentschler: We always talk about innovation and novelty, but I would argue the innovation is just getting it done. You have to make some decisions along the way that that really might close some doors temporarily, but you have to pick and choose just to get it done. And that integration and the completion of it, I think, is always the biggest challenge of bringing innovative things all the way to fruition.