Control Engineering Practice 10 (2002) 1293–1299

A flexible architecture for telemanipulator control

A. Webera,*, H. Breitwieserb, J. Bennerb

aUniversitaet Karlsruhe (TH), Institut fur Mechanische, Verfahrenstechnik und Mechanik, Max-Eyth-Schule, 73230 Kirchheim/u.Teck,

76128 Karlsruhe, GermanybForschungszentrum Karlsruhe, Technik und Umwelt, Postfach 3640, 76021 Karlsruhe, Germany

Received 14 January 2002; accepted 1 March 2002

Abstract

Many automation tasks require human intervention and guidance. A control system architecture combining work units and

master units by software interfaces is described. OSCAR1 is incorporated as a universal transform service. It is regarding dynamic

properties of work units. Implementing the transform at the master site eases integration of new robot types. The dynamic

behaviour when crossing singular points is compared with a closed form standard transform. The control system has also been

applied in medical manipulator systems. r 2002 Published by Elsevier Science Ltd.

Keywords: Telemanipulation; Distribution control; Control algorithms; Optimisation; Medical applications; Manual control

1. Introduction

Aside traditional usage there are many applicationareas of robotics needing high flexibility like e.g.minimally invasive surgery (Hill, Green, Jensen, &Shah, 1994; Salcudean, Ku, & Bell, 1997; Mitshuishiet al., 1997; Schenker & Charles, 1995; Voges et al.,1977). One problem is to provide for integration of asuitable human machine interface. This is also a crucialpart of the numerous simulators being developed e.g.(K .uhnapfel, Krumm, Kuhn, H .ubner, & Neisius, 1995;Jambon, Dubois, & Karpf, 1997). So it might bedesirable to interconnect different control and teachingdevices with or without haptic feedback like thoseknown from telemanipulation for nuclear applicationsor such used for virtual reality like the Phantom(SenSable Devices, Inc.). The best way probably is toprovide for a flexible, open software interface betweenthe controllers. From the point of view of the controldevice, it is desirable to integrate any robots asteleoperated work units and also to control work unitsof different design from one master unit applying theuniversal master control concept (Bejczy & Handlykken,

1981). An approach to both aims is the MONSUN2

control system architecture (Breitwieser & Weber,1996).This paper sketches the main distribution and control

concepts in Section 2. The universal transform OSCAR(see footnote 1) has been incorporated. OSCAR’smathematical principles and merits are covered inSection 3. Experimental results especially the behaviournear singularities for the control of an industrial robotas well as of a medical work unit are shown in Section 4.

2. Concept of distribution and control

2.1. MONSUN system structure

Robots or (tele)manipulator slaves are called workunit (WU). A WU is assumed to have an exchangeableeffector. Additionally, a force-torque sensor may beincluded. In principle there is at least one example of amaster unit (MU), e.g. a master arm, preferably capablefor haptic feedback. But other control devices like e.g. aspace mouse are also admissible. The MU is the mainmeans to manually control a work unit out of a set ofdifferent work units, if teleoperation or teaching is

*Corresponding author. Tel.: +49-721-608-6559; fax: +49-721-608-

6563.

E-mail address: weber@iai.fzk.de (A. Weber).1OSCAR=Optimization Strategy for Control of Articulated

Robots.

2MONSUN=Manipulator Control System Utilizing Network

technology.

0967-0661/02/$ - see front matter r 2002 Published by Elsevier Science Ltd.

PII: S 0 9 6 7 - 0 6 6 1 ( 0 2 ) 0 0 0 7 8 - 3

desired. The selection of a distinct WU from the MU-site is termed coupling. The main technical means forcoupling MONSUN units and providing control in-formation exchange is a common communicationstandard for message exchange. Such a standard isprovided by the TCP/IP transport protocols.

2.2. Basic concepts defining operation modes

One can distinguish operation modes into stand aloneoperation modes refering to a WU or MU and intocoordinated operation modes combining a MU and aWU as seen in Fig. 1. Mixed operation modes rely onoperator guided operation modes and on automaticmodes as well.User guided operation modes according to Fig. 1 are

completely user controlled. These guided operationmodes may be further divided into classical teleopera-tion i.e. master slave operation and into teaching mode.The teaching mode may include additional informationsupply and may have restrictions of some kind. AsMONSUN does not require any similarity among thekinematical structures of a WU to be controlled by aMU, motions in most cases have to be defined on acartesian basis. Additionally, auxilliary axles like agripper are assumed to be controllable. The cartesiancontrol principle requires a definition of standardisedcoordinate systems to which a motion refers to as well asthe mappings defining the steering modes (Breitwieser,1997). For reasons of safety and from a global point ofview MONSUN foresees only position steering modes,i.e. only position increments are to be transferred. Thisis due to the assumption, that the connecting commu-nication system is not 100% reliable and transmissionsmay be delayed or interrupted for some periods of timeor even crash. Of course one can define rate controlschemes by integration of rate device inputs. Otherfeatures of a MU may include e.g. indexing, scalingposition/force, forced orientation alignment, mixedposition-rate steering, and spring effects.

2.3. Protocols and controller implementation structures

Depending on the services and operation mode andhow many MONSUN units are involved, severalcontrollers have to communicate by the exchange ofmessages. The coupling of MU to a WU may bechanged at any time by the operator.The simple protocols (Breitwieser & Weber, 1996) are

defined by local automatons, by message exchangesequences, and the format of the messages. Accordingto Fig. 2 there are three channels with reliable messagetransfer by a TCP-connection. One bidirectional con-nection, Com Do, is for protocol information exchangebetween the MONSUN automatons which are includedwithin the units’ control tasks. The both other unidirec-tional connections are for extensive exchange of positionincrements and force information. A change of thesteering mode (world, tool, joint) can be chosendynamically. Therefore, the steering mode informationhas been included within the position incrementinformation.Thus the cycle tasks are responsible for implementing

the control modes. The receive tasks are responsible forefficient asynchronous communication smoothing pos-sible time delay jitter. The functional blocks of the cycletasks when running manual operation mode are shownin Fig. 3.For unilateral position control, the feedback loop

with force information may be rudimentary. The trans-form blocks ‘forward transform’ and ‘backward trans-form’ are also responsible for the steering modedependent calculations regarding increments and posesand other path planning.

2.4. Extension of control concept

The above described controller structure assumeseach WU to have its own backward transform. This isnormally the case when integrating existing robot

operation modes

coordinated operation modes standalone operation modes

user guided operation modes

mixed operation modes

autonomous operation

sensorguided

non sensorguided

special operation modes

referencing ...synchronous

interleaved

tracking

Fig. 1. Classes of operation modes in MONSUN.

WU -

control

task

WU-cycletask

MU -sendtask

MU - cycle

receive

task

MU -cycletask

MU Controltask

MMI /comm.task

DAT_UP DAT_DO

COM_DO

WU controller MU controller

WU -cycle

receivetask

WU -receive

task

Fig. 2. Task scheme for MU and WU controllers.

A. Weber et al. / Control Engineering Practice 10 (2002) 1293–12991294

designs. Very often, new designs are created withoutconsidering the backward transform problem. Some-times, no closed solution is available. Such designs mayresult from special requirements of an application (cf.part 4). Additionally, a WU’s kinematical design canhave singularities within its work volume. Near thosepoints with low manipulation index, cartesian control isproblematical for reasons of dynamics even if thereexists a backward transform. In general, the dynamiccapabilities are to be respected in manual control.Therefore it is desirable to incorporate a universaltransform method e.g. at the WU site. The universaltransform service OSCAR has been integrated withinthe HDS07-WU of the MONSUN testbed. OSCARrealises a universal transform service for all (evenredundant) open kinematic chain structured robots.One may move the block ‘backward transform’ from

the WU site to the MU site and use only the joint-steering mode at the WU controller. For universalbackward transform service, at the MU site a smalldatabase is necessary to maintain the kinematic descrip-tions of WUs being candidates for MONSUN coupling.This database must also contain information aboutdynamical properties. A consequence of this extension isthat it is now rather easy to integrate new or alreadyexisting work units into a MONSUN system. Newcontrol modes can be implemented without involvementof a WU controller. The above described extension hasalso an advantage in artificial borderline control. Thisfeature, described in Breitwieser and Weber (1996)provides the operator with the intuitive simulation of a

borderline having a suitable force reflecting master. Notime delay is inferred and thus a natural promptness canbe guaranteed.

3. OSCAR

OSCAR (Weber, Benner, & Haffner, 1994) replacesthe conventional dedicated backward transformationroutine by a configuration optimisation procedure(Noble & Daniel, 1977), determining joint angle changesdq which are optimal in the actual situation. Thisoptimisation takes into account two aspects:

* configuration control criteria, which have to beregarded as well as possible, and

* constraints, which must be respected completely.

The quadratic optimisation problem with linearconstraints used in OSCAR can generally be formulatedby

minx

Q ¼Xq

i¼1

pi � ðwi;target � wiÞ2; ð1Þ

respect to

A1 � v ¼ b1 ð2Þ

A2 � vpb2; ð3Þ

vminpvpvmax: ð4Þ

Fig. 3. Functional diagram of MU- und WU-cycle task.

A. Weber et al. / Control Engineering Practice 10 (2002) 1293–1299 1295

Assuming that matrix A1 is regular, this optimisationproblem can be solved by a simplified version of thegradient projection algorithm, described in Noble andDaniel (1977), Rosen (1957, 1960), Wolfe Philip (1962).The optimisation vector v of iteration k þ 1 is calculatedby

vkþ1 ¼ vk � g � Pk � gk ð5Þ

where P is the projection matrix, g the gradient of costfunction (1), k the iteration count and g the step size.In case a new constraint is activated during iteration

k þ 1; the projection matrix may be calculated itera-tively by

Pkþ1 ¼ Pk � c � cT= cj j; c ¼ Pk � aj ; ð6Þ

where aj is the vector of activated constraint. In eachcontrol cycle a target move for the robot end effector toa new position and orientation dxtarget is given byoperator or path planner. Thus a cost function (7) forthe movement of tool coordinate system Ktcp is built.This movement is constrained by admissible deviationsD from target position (9). The joint movement vectordq also is constrained by joint angle boundaries, speedand acceleration limits. These joint constraints dqminand dqmax are part of (9). The optimisation vector is nowdefined as v¼ ½dq; dx: For realizing this robot controltask the optimisation problem is formulated by

min½dq;dx

Q ¼X6

i¼1

pi � ðdxi;target � dxiÞ2 ð7Þ

with respect to

½J;�I6 � ½dq; dxT ¼ 0; ð8Þ

½dqmin; dx� DTp½dq; dxTp½dqmax; dxþ DT ; ð9Þ

where J is the Jacobian, I6 the identity matrix, pi theweighting factors.When optimisation starts, the equality constraints (8)

are part of the projection matrix. Due to identity matrixI6 in (8) the condition for a regular matrix A1 is alwaysfulfilled. Thus the algorithm even works near singula-rities. All parameters of the optimisation vector, reach-ing their limits get part of the projection matrix too,until the number of active constraints is equal to thenumber of parameters in the optimisation vector. Ineach case the demanded end effector movement will beachieved as far as the guaranteed limits of jointmovement will allow for this. If there does not exist asolution for the optimisation vector inside its limits (9)an exception code will be given.Other constraints may be added to Eqs. (2)–(4):

* workspace limits;* obstacles in workspace;* joint constraints as dq5 ¼ dq6 ¼ dq7 ¼ dq8 for con-trolling an instrument with a multi-link joint asshown in Fig. 10;

* fixpoints with respect to robot links as shown inFig. 4.

Especially, fixpoints are important for medical robotapplications, namely to guaranty the invariance of theincision point (Fig. 11). First the actual tool center pointposition at the incision point is defined as fixpoint.Moving the tool center point now, the following robotlinks are forced to touch this fixpoint position by thealgorithm.Using geometrical relations the distance between the

fixpoint position F and the cutting point of the robotlink with the fixpoint plane, discribed by the fixpointitself and a normal n; can be calculated by

D ¼ F� xi � mðxiþ1 � xiÞ ð10Þ

with

m ¼ n � ðF� xiÞ=ðn � ðxiþ1 � xiÞÞ

xiþ1;xi : robot joint positions:

The differentiation dD=dq ¼ 0 leads to linear con-straints which can be added to Eq. (3).OSCAR applied for a six-axis robot and needing 12

iterations takes about 6ms on the HDS-controller.

4. Experiences

The main WU within the MONSUN testbed is theHDS07 (see Fig. 5) industrial robot from NokiaMechatronics. It is equipped with an electrical gripperas a seventh axis adding a payload of 7 kg. Its handaxlesare arranged in a R–P–R-configuration. Thus it employsthe singularity of the configuration, when gripperapproach line and lower arm are aligned. Unfortunately,this singularity is located within the standard workvolume for manual operation.

nxi

xi+1

F

Fig. 4. Fixpoint constraint definition.

A. Weber et al. / Control Engineering Practice 10 (2002) 1293–12991296

The HDS controller is based on a Multibus-II system.Except of some IO all control is implemented on onei486/100MHz based board running the real timeoperating system iRMX-III.3 A model based positionclosed loop control is implemented.In the sequel experiments when crossing the streched

singularity configuration in manual mode are compared,using OSCAR and the standard transform. Position andforce information were exchanged at a rate of 33Hzfrom EM2 to HDS and back, respectively.As MU a master arm of a conventional telemanipu-

lator design, an EM2-Master (see Fig. 6), employingforce/torque capabilities on 6+1 axles has been used.The MU controller runs also on a i486/100MHz board.Tilting the gripper with constant rate from a

horizontal level down by 901 to a vertical downarrangement, the robot is driven through or nearlythrough the above-mentioned singularity. The standardtransform (STD) calculates its position set pointswithout regarding acceleration capabilities of theHDS07. Thus very often, stability of position controllacks in this zone because of amplifier saturation. Theadmissible maximal tracking error of the handaxlesmust be put to values of about 2 rad and more for notshutting down the robot when crossing this area.The HDS steered in world coordinates thus has to

cross the stretched configuration. The resulting devia-

tions of the position of the tool coordinate system Ktcp;which should have remained unchanged as well as theangular deviations of the ez unit vector of Ktcp areshown in Fig. 7.As one can easily see, huge deviations (0.22m and

0.55 rad) occurred in this example. This of course cannot

Fig. 5. HDS07 work unit.

Fig. 6. EM2 master.

STD real deviations

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5

sec

STD real pos distance [m]

STD real p-j-angular distance [rad]

Fig. 7. Position and angular deviations of the HDS07 passing through

singularity.

3RadiSys.

A. Weber et al. / Control Engineering Practice 10 (2002) 1293–1299 1297

be tolerated in any and especially in manual operationmode. OSCAR has been implemented as a separate taskwithin the HDS-WU and operates in parallel with theSTD transform. Position set points then can be takenfrom OSCAR. With a very same motion from the MUsite and the WU starting from the same horizontalgripper position, a completely other behaviour can beseen from Fig. 8. Position deviations4 are negligiblecompared with STD transform. The angular deviation is0.14 rad with OSCAR compared with 0.55 rad takingSTD transform. Fig. 8 also contains the nominaldistances taken by OSCAR to lower dynamical require-ments.The specified maximal values for position deviation

were 0.007m and 0.1 rad, respectively. One shouldmention, that these experiments show differences whichare not precisely repeatable in normal manual operation

mode. Sometimes depending on the exact configuration,a good behaviour with the STD transform is possible,but the operator cannot predict that this will actuallyhappen. He must take into account a behaviour like thatshown in Fig. 7. Taking OSCAR the singularityproblem can always be mastered. As a consequencesmaller limit values for admissible tracking errors checkroutines can be applied.As OSCAR has also been implemented and integrated

within the EM2-MU controller, the HDS work unit canbe controlled e.g. in WW-steering mode by the MUwhere the WU is supplied only with joint increments asis the case in the joint-steering mode. This extension hasbeen applied in steering TISMIKA (see Figs. 9 and 10)in cartesian steering mode.TISMIKA is the work unit within ARTEMIS5 (Voges

et al., 1977). The multi-link joint of the medical workunit is mechanically coupled. Mathematically, each jointof the multi-link joint sets the same joint angle. By thisforced coupling the STD transforms are no moreapplicable. Another fact is that the number and alsothe joint ranges of such a multi link joint may differfrom one another depending on the application fieldswhich may require different medical instruments.

OSCAR real and nominal distances

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5

sec

[m]

or

[ra

d]

real pos distance OSCAR

real angular distance OSCAR

nominal pos distance OSCAR

nominal angular distanceOSCAR

Fig. 8. Nominal and real deviations applying OSCAR for backward

transform.

effector

multi link joint instrument shaft

Fig. 9. Instrument with multi-link joint.

instrument

drive box instrument

effector

carrier

Fig. 10. TISMIKA-WU.

4Maximal 14mm at 3.6 s.

5Advanced Robotics and Telemanipulator system for Minimally

Invasive Surgery.

A. Weber et al. / Control Engineering Practice 10 (2002) 1293–12991298

OSCAR can cope with forced coupling of that type asdescribed in part 3.

5. Conclusion

This paper presented an open architecture fordistributed manipulator systems called MONSUN.Either different work units or master control unitsmay be combined arbitrarily if they regard theMONSUN protocols. By the extension of the distribu-tion concept of functionality, the integration of newwork units is simplified. The general backward trans-form is solved by OSCAR. Aside user defined restric-tions it takes into account dynamic capabilities of workunits and is able to provide additional information forthe master unit which may be conveyed by haptic meansto the operator. OSCAR has been used successfully forthe control of a surgical work unit which includes amechanically coupled multi-link joint. Its ability to copewith singularities has been discussed for an industrial

robot. OSCAR can handle fixpoint restrictions neededfor minimally invasive surgery (Fig. 11).

References

Bejczy, A., & Handlykken, M. (1981). Experimental results with a six-

degree-of-freedom force-reflecting hand controller. In: Proceedings

of the 17th Annual Conference on Manual Control. LA.

Breitwieser, H. (1997). The universal master concept and different

control coordinate systems. In: IARP Second Workshop on Medical

Robotics. Heidelberg, Forschungszentrum Karlsruhe.

Breitwieser, H., & Weber, W. (1996). MONSUN a distributed

manipulator control system utilizing network technology. In P.

Kopacek (Ed.), Human-oriented design of advanced robotics systems

(pp. 159–167). Oxford: Pergamon.

Hill, J. W., Green, P. S., Jensen, J. Y., & Shah, A. (1994). Telepresence

surgery demonstration system, In: Proceedings of the IEEE

International Conference on Robotics and Automation, pp. 2302–

2307.

Jambon, A. C., Dubois, P., & Karpf, S. (1997). A low-cost training

simulator for initial formation in gynecologic laparoscopy. In:

Proceedings of the CVRMed-MRCAS’97, Springer, Lecture Notes

in Computer Science 1205.

K .uhnapfel, U., Krumm, H. G., Kuhn, C., H .ubner, M., & Neisius, B.

(1995). Endosurgery simulations with KISMET, IDG-Conference

documentation Virtual Reality World ’95. Computerwoche Verlag,

M .unchen.

Mitshuishi Mamoru et al. (1997). Dexterity enhancement for a tele-

micro-surgery System. In: J. Troccaz, et al. (Eds.), CVRMed-

MRCAS’97, Springer, Lecture Notes in Computer Science 1205.

Noble, B., & Daniel, J. W. (1977). Applied linear algebra, Prentice-Hall

International Editions, NJ.

Rosen, J. B. (1957). Nonlinear programming. The gradient projection

method. In: American Mathematical Society, 63 (pp. 25–26),

Abstract No.80.

Rosen, J. B. (1960). The gradient projection method for nonlinear

programming. Journal of the Society for Industrial and Applied

Mathematics, 8(1).

Salcudean, S. E., Ku, S., & Bell, G. (1997). Performance measurement

in scaled teleoperations for microsurgery. In: J. Troccaz, et al.

(Eds.), CVRMed-MRCAS’97, Springer Lecture Notes in Computer

Science 1205.

Schenker, P. S., & Charles, S. T. (1995). Development of a

telemanipulator for dexterity enhanced microsurgery, In: Proceed-

ings of the MRCAS’95.

SenSable Devices, Inc. PHANToM haptic master, www.sensable.com.

Voges, U., et al. (1977). Evaluation of ARTEMIS, the advanced

robotics and telemanipulator system for minimally invasive

surgery. In: IARP Second Workshop on Medical Robotics.

Heidelberg, Forschungszentrum Karlsruhe.

Weber, A., Benner, J., & Haffner, H. (1994). OSCAR—optimization

strategy for control of redundant articulated robots. In: RI/SME

Fifth World Conference on Robotics Research, Cambridge, MA,

USA.

Wolfe Philip (1962). Methods of nonlinear programming. Recent

Advances in Mathematical Programming, McGraw-Hill Book

Company, Inc.

Fig. 11. Modular medical robots.

A. Weber et al. / Control Engineering Practice 10 (2002) 1293–1299 1299