The software used to control a robot is normally highly adapted to its specific physical set up. But now researchers have created a single general-purpose robotic control policy that can operate robotic arms, wheeled robots, quadrupeds, and even drones.

One of the biggest challenges when it comes to applying machine learning to robotics is the paucity of data. While computer vision and natural language processing can piggyback off the vast quantities of image and text data found on the Internet, collecting robot data is costly and time-consuming.

To get around this, there have been growing efforts to pool data collected by different groups on different kinds of robots, including the Open X-Embodiment and DROID datasets. The hope is that training on diverse robotics data will lead to “positive transfer,” which refers to when skills learned from training on one task help to boost performance on another.

The problem is that robots often have very different embodiments—a term used to describe their physical layout and suite of sensors and actuators—so the data they collect can vary significantly. For instance, a robotic arm might be static, have a complex arrangement of joints and fingers, and collect video from a camera on its wrist. In contrast, a quadruped robot is regularly on the move and relies on force feedback from its legs to maneuver. The kinds of tasks and actions these machines are trained to carry out are also diverse: The arm may pick and place objects, while the quadruped needs keen navigation.

That makes training a single AI model for robots on these large collections of data challenging, says Homer Walke, a Ph.D. student at the University of California, Berkeley. So far, most attempts have either focused on data from a narrower selection of similar robots or researchers have manually tweaked data to make observations from different robots more similar. But in research to be presented at the Conference on Robot Learning (CoRL) in Munich in November, they unveiled a new model called CrossFormer that can train on data from a diverse set of robots and control them just as well as specialized control policies.

“We want to be able to train on all of this data to get the most capable robot,” says Walke. “The main advance in this paper is working out what kind of architecture works the best for accommodating all these varying inputs and outputs.”

How to control diverse robots with the same AI model

The team used the same model architecture that powers large language model, known as a transformer. In many ways, the challenge the researchers were trying to solve is not dissimilar to that facing a chatbot, says Walke. In language modeling, the AI has to to pick out similar patterns in sentences with different lengths and word orders. Robot data can also be arranged in a sequence much like a written sentence, but depending on the particular embodiment, observations and actions vary in length and order too.

“Words might appear in different locations in a sentence, but they still mean the same thing,” says Walke. “In our task, an observation image might appear in different locations in the sequence, but it’s still fundamentally an image and we still want to treat it like an image.”

UC Berkeley/Carnegie Mellon University

Most machine learning approaches work through a sequence one element at a time, but transformers can process the entire stream of data at once. This allows them to analyze the relationship between different elements and makes them better at handling sequences that are not standardized, much like the diverse data found in large robotics datasets.

Walke and his colleagues aren’t the first to train transformers on large-scale robotics data. But previous approaches have either trained solely on data from robotic arms with broadly similar embodiments or manually converted input data to a common format to make it easier to process. In contrast, CrossFormer can process images from cameras positioned above a robot, at head height or on a robotic arms wrist, as well as joint position data from both quadrupeds and robotic arms, without any tweaks.

The result is a single control policy that can operate single robotic arms, pairs of robotic arms, quadrupeds, and wheeled robots on tasks as varied as picking and placing objects, cutting sushi, and obstacle avoidance. Crucially, it matched the performance of specialized models tailored for each robot and outperformed previous approaches trained on diverse robotic data. The team even tested whether the model could control an embodiment not included in the dataset—a small quadcopter. While they simplified things by making the drone fly at a fixed altitude, CrossFormer still outperformed the previous best method.

“That was definitely pretty cool,” says Ria Doshi, an undergraduate student at Berkeley. “I think that as we scale up our policy to be able to train on even larger sets of diverse data, it’ll become easier to see this kind of zero shot transfer onto robots that have been completely unseen in the training.”

The limitations of one AI model for all robots

The team admits there’s still work to do, however. The model is too big for any of the robots’ embedded chips and instead has to be run from a server. Even then, processing times are only just fast enough to support real-time operation, and Walke admits that could break down if they scale up the model. “When you pack so much data into a model it has to be very big and that means running it for real-time control becomes difficult.”

One potential workaround would be to use an approach called distillation, says Oier Mees, a postdoctoral research at Berkley and part of the CrossFormer team. This essentially involves training a smaller model to mimic the larger model, and if successful can result in similar performance for a much smaller computational budget.

But of more importance than the computing resource problem is that the team failed to see any positive transfer in their experiments, as CrossFormer simply matched previous performance rather than exceeding it. Walke thinks progress in computer vision and natural language processing suggests that training on more data could be the key.

Others say it might not be that simple. Jeannette Bohg, a professor of robotics at Stanford University, says the ability to train on such a diverse dataset is a significant contribution. But she wonders whether part of the reason why the researchers didn’t see positive transfer is their insistence on not aligning the input data. Previous research that trained on robots with similar observation and action data has shown evidence of such cross-overs. “By getting rid of this alignment, they may have also gotten rid of this significant positive transfer that we’ve seen in other work,” Bohg says.

It’s also not clear if the approach will boost performance on tasks specific to particular embodiments or robotic applications, says Ram Ramamoorthy, a robotics professor at Edinburgh University. The work is a promising step towards helping robots capture concepts common to most robots, like “avoid this obstacle,” he says. But it may be less useful for tackling control problems specific to a particular robot, such as how to knead dough or navigate a forest, which are often the hardest to solve.

Where’s your flying car? I’m sorry to say that I have no idea. But here’s something that is somewhat similar, in that it flies, transports things, and has “car” in the name: it’s a flying cart, called the Palletrone (pallet+drone), designed for human-robot interaction-based aerial cargo transportation.

The way this thing works is fairly straightforward. The Palletrone will try to keep its roll and pitch at zero, to make sure that there’s a flat and stable platform for your preciouses, even if you don’t load those preciouses onto the drone evenly. Once loaded up, the drone relies on you to tell it where to go and what to do, using its IMU to respond to the slightest touch and translating those forces into control over the Palletrone’s horizontal, vertical, and yaw trajectories. This is particularly tricky to do, because the system has to be able to differentiate between the force exerted by cargo, and the force exerted by a human, since if the IMU senses a force moving the drone downward, it could be either. But professor Seung Jae Lee tells us that they developed “a simple but effective method to distinguish between them.”

Since the drone has to do all of this sensing and movement without pitching or rolling (since that would dump its cargo directly onto the floor) it’s equipped with internal propeller arms that can be rotated to vector thrust in any direction. We were curious about how having a bunch of unpredictable stuff sitting right above those rotors might affect the performance of the drone. But Seung Jae Lee says that the drone’s porous side structures allow for sufficient airflow and that even when the entire top of the drone is covered, thrust is only decreased by about 5 percent.

The current incarnation of the Palletrone is not particularly smart, and you need to remain in control of it, although if you let it go it will do its best to remain stationary (until it runs out of batteries). The researchers describe the experience of using this thing as “akin to maneuvering a shopping cart,” although I would guess that it’s somewhat noisier. In the video, the Palletrone is loaded down with just under 3 kilograms of cargo, which is respectable enough for testing. The drone is obviously not powerful enough to haul your typical grocery bag up the stairs to your apartment. But, it’s a couple of steps in the right direction, at least.

We also asked Seung Jae Lee about how he envisions the Palletrone being used, besides as just a logistics platform for either commercial or industrial use. “By attaching a camera to the platform, it could serve as a flying tripod or even act as a dolly, allowing for flexible camera movements and angles,” he says. “This would be particularly useful in environments where specialized filming equipment is difficult to procure.”

And for those of you about to comment something along the lines of, “this can’t possibly have enough battery life to be real-world useful,” they’re already working to solve that, with a docking system that allows one Palletrone to change the battery of another in-flight:

One Palletrone swaps out the battery of a second Palletrone.Seoul Tech

“The Palletrone Cart: Human-Robot Interaction-Based Aerial Cargo Transportation,” by Geonwoo Park, Hyungeun Park, Wooyong Park, Dongjae Lee, Murim Kim, and Seung Jae Lee from Seoul National University of Science and Technology in Korea, is published in IEEE Robotics And Automation Letters.

This is a sponsored article brought to you by Khalifa University of Science and Technology.

A total of eight intense competitions to inspire creativity and innovation along with 13 forums dedicated to diverse segments of robotics and artificial intelligence will be part of the 36th edition of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024) in Abu Dhabi.

These competitions at the Middle East and North Africa (MENA) region’s first-ever global conference and exhibition from 14-18 October 2024 at the Abu Dhabi National Exhibition Center (ADNEC) will highlight some of the key aspects of robotics. These include physical or athletic intelligence of robots, remote robot navigation, robot manipulation, underwater robotics, perception and sensing as well as challenges for wildlife preservation.

This edition of IROS is one of the largest of its kind globally in this category because of active participation across all levels, with 5,740 authors, 16 keynote speakers, 46 workshops, 11 tutorials, as well as 28 exhibitors and 12 startups. The forums at IROS will explore the rapidly evolving role of robotics in many industry sectors as well as policy-making and regulatory areas. Several leading corporate majors, and industry professionals from across the globe are gathering for IROS 2024 which is themed “Robotics for Sustainable Development.”

“The intense robotics competitions will inspire creativity, while the products on display as well as keynotes will pave the way for more community-relevant solutions.” —Jorge Dias, IROS 2024 General Chair

Dr. Jorge Dias, IROS 2024 General Chair, said: “Such a large gathering of scientists, researchers, industry leaders and government stakeholders in Abu Dhabi for IROS 2024 also demonstrates the role of UAE in pioneering new technologies and in providing an international platform for knowledge exchange and sharing of expertise. The intense robotics competitions will inspire creativity, while the products on display as well as keynotes will pave the way for more community-relevant solutions.”

The competitions are:

• 2nd AI Olympics With RealAIGym: Is AI Ready for Athletic Intelligence in the Real World? focusing on the physical or athletic intelligence of robots

• The Earth Rovers Challenge on remote robot navigation

• Robotic Construction Challenge and the euROBIN Manipulation Skill Versatility Challenge (MSVC) on robot manipulation

• Underwater Robotics Challenges on innovation and problem-solving in underwater robotics and artificial intelligence

• ROAD-IROS: Automatic Data Annotation Challenge for ROAD Dataset on perception and sensing

• 21st F1Tenth Autonomous Racing Competition

• IEEE RAS Quadruped Robot Challenge (QRC), focusing on navigation

In addition to these competitions, the Falcon Monitoring Challenge (FMC) will focus on advancing the field of wildlife tracking and conservation through the development of sophisticated, noninvasive monitoring systems.

Khalifa University

IROS 2024 will also include three keynote talks on ‘Robotic Competitions’ that will be moderated by Professor Lakmal Seneviratne, Director, Center for Autonomous Robotic Systems (KU-CARS), Khalifa University. The keynotes will be delivered by Professor Pedro Lima, Institute for Systems and Robotics, Instituto Superior Técnico, University of. Lisbon, Portugal; Dr. Timothy Chung, General Manager, Autonomy and Robotics, Microsoft, US; and Dr. Ubbo Visser, President of the RoboCup Federation, Director of Graduate Studies, and Associate Professor of Computer Science, University of Miami, US.

The forums at IROS 2024 will include:

• Robotics in Africa Forum

• Marine Robotics in Ocean Decade Initiative for Sustainable Development

• Robots for Sustainability and Sustainable Robots

• Government Forum: Funding for Robotics Research, Human-Avatars Symbiosis: Can you imagine a future society where you can remotely control multiple avatars?

• Moonshot R&D Program Goal 3 Forum. Envisioning a Future of Human-Robot Co-living: Potential for Robotics to Transform Human Lives

Other forums include:

• Sustainable Medical and Surgical Robotics

• Empowering Diverse Voices in Robotics, Robotics & AI in the UAE: Research Innovation and Entrepreneurship

• Europe Regulates Artificial Intelligence: the Challenge for Robotics, Industrial Opportunities and Socio-Economic Impact of Medical Robotics

• The Future of Work: AI-Enhanced Robotics and Human Interaction Research in M3S

• Robots for a Better Tomorrow: Wellbeing Through Advanced Technology.

One of the largest and most important robotics research conferences in the world, IROS 2024 provides a platform for the international robotics community to exchange knowledge and ideas about the latest advances in intelligent robots and smart machines. A total of 3,344 paper submissions representing 60 countries, have been received from researchers and scientists across the world. China tops the list with more than 1,000 papers, the US with 777, Germany with 302, Japan with 253, and the UK and South Korea with 173 each. The UAE remains top in the Arab region with 68 papers.

One of the largest and most important robotics research conferences in the world, IROS 2024 provides a platform for the international robotics community to exchange knowledge and ideas.

For eight consecutive years since 2017, Abu Dhabi has remained first on the world’s safest cities list, according to online database Numbeo, which assessed 329 global cities for the 2024 listing. This reflects the emirate’s ongoing efforts to ensure a good quality of life for citizens and residents. With a multicultural community, Abu Dhabi is home to people from more than 200 nationalities, and draws a large number of tourists to some of the top art galleries in the city such as Louvre Abu Dhabi and the Guggenheim Abu Dhabi, as well as other destinations such as Ferrari World Abu Dhabi and Warner Bros. World™ Abu Dhabi.

Because of its listing as one of the safest cities, Abu Dhabi continues to host several international conferences and exhibitions. Abu Dhabi is set to host the UNCTAD World Investment Forum, the 13th World Trade Organization (WTO) Ministerial Conference (MC13), the 12th World Environment Education Congress in 2024, and the IUCN World Conservation Congress in 2025.

IROS 2024 is sponsored by IEEE Robotics and Automation Society, Abu Dhabi Convention and Exhibition Bureau, the Robotics Society of Japan (RSJ), the Society of Instrument and Control Engineers (SICE), the New Technology Foundation, and the IEEE Industrial Electronics Society (IES).

More information at https://iros2024-abudhabi.org/

Elon Musk has joined the chorus of conservative and MAGA voices online backing Sen. Rick Scott (R-Fla.) for Senate GOP leader - after calling Sen. John Thune (R-S.D.) the "top choice of Democrats."

EA FC 25 players have a whole new Team of the Week to grind for, and there are some fantastic cards among them, including Sporting CP's hotshot striker Viktor Gyokeres.

Destiny 2's Exotic vendor, Xur, is back again this week. Here's how to find him, what he's selling, and whether we recommend his offerings before he vanishes again.

EA FC 25 players can snag some big freebies this week, with EA Sports celebrating the Ballon d'Or in style for all Ultimate Team players with some of the best players around.

Black Ops 6 is here, and fans have been itching to know what's included in Season 1 - and now we have an answer. Here's what's included, and when you can play

Call of Duty fans are impressed by Black Ops 6 Season 1 already, calling it the 'best Season 1 roadmap' since 2019's Modern Warfare reboot and you can find out below why they're saying it

Black Ops 6 is here, and it's Black Ops 2's anniversary, so what better time than to rank the Call of Duty Black Ops titles? Here's our ranking of every mainline version.

If you wanted to return to GTA Trilogy, now might be the best time as the game has finally got an update after three years which makes it closer to the much better mobile version

Comet Nishimura (C/2023 P1) was discovered in August and is now whizzing by Earth, but finding it in the sky will be a challenge.

House Republicans on Tuesday demanded that each member of President Biden's cabinet preserve all relevant documents and communications, a move that signals future investigations into the Biden administration.

According to a report Tuesday, the league deliberately decided not to impose an immediate fine on San Francisco 49ers defensive end Nick Bosa for wearing a "Make America Great Again" hat during a postgame television interview.

House Speaker Mike Johnson is seeking a smooth re-election to another term wielding the gavel, but a small group of discontent conservatives are again vying to shake things up at the top.

Kurt "CyberGuy" Knutsson presents the cities targeted most by robocalls, why this is happening and what you can do about it to protect your privacy.

An updated Android trojan called FakeCall hijacks bank calls. Tech expert Kurt “CyberGuy" Knutsson says Android phone manufacturers and Google need to step up their game on security.

Apple has introduced a native call recording feature as part of Apple Intelligence with its release of iOS 18.1. It's easier than ever to document conversations.

Stay up to date on the latest AI technology advancements and learn about the challenges and opportunities AI presents now and for the future.

Arsenal and England midfielder Jordan Nobbs will be a columnist for 90min & BBC Sport pundit at the Women's World Cup in France this summer, as Phil Neville's Lionesses' look to secure a maiden World Cup triumph. Two wins from our first two games is obviously what we were all hoping for. The first game of a World Cup, or any other major tournament, is never going to be one where you're playing at your absolute best. But the 2-1 win over was Scotland was great given the pre-match hype, the..

Liverpool already have one song synonymous with the club but they did attempt to officially do likewise with another. The Reds tried to trademark 'Allez Allez Allez'..

The future and all that is possible was on full display this week at MWC Barcelona 2024 in Hall Six of Gran Via, the site of an innovation zone that featured hands-on and immersive demos of things ranging from devices that enhance the growing of fruit to what has been described as the “world’s first […]

With competition heating up in the foundry business – India this week approved three new semiconductor plants involving Tata Group and CG Power,, and is looking to achieve dominance in the industry – existing foundries have to up their game. Chief among them is Intel, which has been trying to recover from historical missteps that […]

The annual Perseid meteor shower is set to peak on Sunday night into early Monday morning, giving stargazers the chance to see hundreds of meteors.

A stadium-sized asteroid is passing relatively close to Earth on Tuesday, NASA announced. Its distance from Earth and its massive size makes it a "potentially hazardous object."

Human cognitive performance can be significantly influenced by the presence of audience members.

The post Study: Chimps Perform Better on Challenging Computer Tasks When They Have Audience appeared first on Sci.News: Breaking Science News.

Dis-Ease by Mariam Ghani uses strong visuals and compelling interviews to argue that how we see and describe disease affects how we deal with it, says Simon Ings

The World Health Organization considers "gaming disorder" a condition, but researchers have now found that playing video games can boost well-being

Red and processed meat, and even poultry, seem to raise the risk of developing type 2 diabetes, according to a study of nearly 2 million adults, but not everyone is convinced

A single dose of a smallpox vaccine seems to lower the risk of catching mpox by around 60 per cent, and two doses would probably be even better

Dopamine was long thought to play a part in the placebo effect for pain relief, but a new study is questioning its true role

Male pattern baldness could soon be a thing of the past, with new hair loss treatments beginning to show tantalising results

Filters on commercial flights seem to stop peanut particles from circulating around aircraft, making the risk of a serious allergic reaction from inhaling the allergens very low

Cases of prostate cancer are surging alarmingly around the world. Thankfully, we are developing more accurate tests that can catch the condition early

Everything you need to know about Rwanda's outbreak of Marburg virus, which has been described as one of the deadliest human pathogens

As more of us take up vaping and concerns rise about the long-term effects, we now have enough data to get a grip on the health impact – and how it compares to smoking

Diets that claim to control excess oestrogen or stress hormones are all the rage on Instagram and TikTok. They could be good for us, just not for the reasons claimed

Our metabolism's response to food is highly idiosyncratic and there are hints that tailoring our diet to these personal differences can deliver health benefits