Remote procedure calls, issues, implementation

1. CS 704DAdvanced Operating System(RPC) Debasis Das

2. Remote ProcedureCalls(RPC) 2

4. RPC became popular IPC mechanisms for distributed applications

5. Simple call syntax

7. Ease of use, clean & Simple

8. Generality & efficiency

9. Same IPC mechanism can be used for procedure on different machines as well as on the same machine11/21/2009 IT703D: Dirstibuted Computing Debasis Das 3

11. Make a procedure call, place parameters in a known location

12. Transfer control to the procedure

13. Procedure executes with copies of the parameters/arguments

14. Control is returned to calling procedure with result, if any11/21/2009 IT703D: Distributed Computing Debasis Das 4

15. RPC Model(Graphically) Caller Client Process Request message & parameters Send reply with result, if any Server Process 11/21/2009 IT703D: Distributed Computing Debasis Das 5

17. If concurrency required, calling processes can leave messages asynchronously, server process keeps serving requests

18. Server process can have threads. Starts a thread on request, server is free to field further calls if they come along11/21/2009 IT703D: Distributed Computing Debasis Das 6

20. Syntactic transparency

21. Semantic transparency

22. Syntactic transparency is easy to achieve but not semantic transparency11/21/2009 IT703D: Distributed Computing Debasis Das 7

24. No shared memory: passing addresses as arguments is meaningless, call by reference pointers are useless, structures referenced by pointers are difficult.

25. More vulnerable to failures: two different processes, two different machines, intervening network. Processor and communication failures need to be taken care of.

26. RPCs can take much more time(x100 to x1000) due to communication congestions. Long delays need to be handled.11/21/2009 IT703D: Distributed Computing Debasis Das 8

27. Implementing RPC Mechanism Server Client Return Call Call Return Execute Stubs Unpack Pack Pack Unpack RPC Runtimes Receive Send Receive Send Wait 11/21/2009 IT703D: Distributed Computing Debasis Das 9

29. On call request, packs specs of remote procedure and arguments

30. Asks RPC Runtime to transmit the message to the server

31. Server stub

32. After receipt, the RPC Runtime passes the message to it. It then unpacks the message and makes a usual call to the procedure.

33. On receipt of result, packs the result and asks the runtime to send it back to the client.11/21/2009 IT703D: Distributed Computing Debasis Das 10

35. No need to dabble in send & receive primitives11/21/2009 IT703D: Distributed Computing Debasis Das 11

37. Write code to implement translation functions provided by the RPC implementer

38. Automatically

39. By using IDL. Procedures supported by the interface, types of arguments and results

40. IDL compiler will ensure compilation, type checking etc.

41. Optimizes data to be sent over the network. Interface definition usually will have information as to which arguments are input/output/both.11/21/2009 IT703D: Distributed Computing Debasis Das

43. Interface import: calls procedures from the interface

44. Define an interface using the IDL, write client program that imports the interface and server program that exports the interface

45. IDL compilers generate client stub procedure, server stub procedure, header files. Compile & link the stubs.

46. Interface code thus generated could be combined with client/server procedures done in different languages11/21/2009 IT703D: Distributed Computing Debasis Das

48. Sent by client to server with request to execute a remote procedure

49. Reply messages

50. Sent by server to the requesting client to indicate completion of a request and related results if generated

51. The messages follow RPC protocols and has nothing to do with network transport protocol11/21/2009 IT703D: Distributed Computing Debasis Das

53. Id information for the remote procedure to be executed

54. Required arguments to execute the procedure

55. Additional fields

56. Message id field, essentially a sequence number

57. Message type field identifying if a call/reply message

58. Client id field, identifying the requesting client and for authentication11/21/2009 IT703D: Distributed Computing Debasis Das

59. RPC Call Message Structure Remote Procedure id Client id Message id Message type Program number Version number Procedure number Arguments 11/21/2009 IT703D: Distributed Computing Debasis Das 16

61. Client id not authorized, failure message

62. Procedure id not available on server, failure message

63. Parameters not successfully decoded, failure message

64. Exception(e.g. divide by zero) happens, failure message

65. Procedure executes successfully, return result(s)11/21/2009 IT703D: Distributed Computing Debasis Das

66. RPC Reply Message Structure Message id Message type Reply Status success Result(s) Message id Message type Reply Status failure Failure reason 11/21/2009 IT703D: Distributed Computing Debasis Das 18

68. This is the marshalling process and involves

69. Take the arguments/result

70. Encoding of data on sender’s machine

71. Decoding of message data on receiver’s machine

72. Representation method(tagged or untagged) should be known, type safety can be ensured

73. Must reflect structure of all types; primitive types, structured types and user defined types11/21/2009 IT703D: Distributed Computing Debasis Das 19

74. Marshalling Procedure Types Provided as part of the RPC support. Procedures for scalar types and compound ones built from Scalar types are included Defined by users of RPC systems. Includes marshalling procedures for user defined data types 11/21/2009 IT703D: Distributed Computing Debasis Das

76. Stateful servers

77. Stateless servers

78. Why stateless servers

79. Server creation semantics

80. Instance-per-call servers

81. Instance-per-session servers

82. Persistent servers11/21/2009 IT703D: Distributed Computing Debasis Das

84. For example a Stateful file server will remember that a particular client opened a file(given file id) previously and will let consecutive reads /writes and other operations on that file to be done properly

85. Two consecutive reads such as Read(fid,100,buf) will read two consecutive 100 bytes of data from the open file, opened in an earlier procedure call.11/21/2009 IT703D: Distributed Computing Debasis Das

87. It is the responsibility of the server to ensure the operations are carried out correctly

88. For a similar example; you may have to open the file, seek to the correct location and then read to have a Read correctly.

89. Read(filename, position, bytes, buffer) or Write(filename, position, bytes, buffer)11/21/2009 IT703D: Distributed Computing Debasis Das

91. Stateless servers protect against failures

92. Server crashes, state information is lost

93. Client may be unaware of crash & restart creating inconsistent results.

94. Client crashes and restarts, server has useless state information that cannot be easily rolled back11/21/2009 IT703D: Distributed Computing Debasis Das

96. Independent lifetimes

97. Independent locations

98. Independent address spaces

99. Servers may be created prior to the client processes or on demand11/21/2009 IT703D: Distributed Computing Debasis Das

101. Deleted when the call has been serviced

102. Not a very common method because

103. Will have to be a stateless server and state information to be maintained by the client process or the OS. OS involvement is expensive.11/21/2009 IT703D: Distributed Computing Debasis Das

105. Server managers are registered with a binding agent

106. When a client needs a service, contacts binding agent

107. Binding agent sends address of server manager

108. Client sends request to server manager, who spawns a server and sends its address

109. Client interacts with server until it no longer requires the server

110. Server manager destroys the serverWhat should be type of server; stateful or stateless? 11/21/2009 IT703D: Distributed Computing Debasis Das

112. Shared by more than one client

113. Created and installed prior to clients

114. Servers register with binding agent

115. Client request binding agent for a server

116. Binding agent selects one randomly or as per some policy

117. Client requests are interleaved and several sets of state are maintained

118. Multiple servers improves load sharing/failure tolerance11/21/2009 IT703D: Distributed Computing Debasis Das

120. Send value of arguments along with invocation message

121. Can be complicated and wasteful for complex data structures

122. Call by reference

123. Does not make sense in separate address spaces

124. Systems with shared distributed memory would be ok

125. Call by object reference

126. Call by move

127. Call by visit11/21/2009 IT703D: Distributed Computing Debasis Das

129. Call message can get lost

130. Reply message can get lost

131. Caller node can crash and get restarted

132. Callee node can crash and get restarted

133. The issue is to use call semantics that can protect against this

134. Application procedure is obviously not the place to handle these issues

135. Best place would be the RPC runtime system11/21/2009 IT703D: Distributed Computing Debasis Das

137. Last one call semantics

138. Last-of many call semantics

139. At-least-once call semantics

140. Exactly-once call semantics11/21/2009 IT703D: Distributed Computing Debasis Das

142. Call ack. or results are not guaranteed

143. Weakest semantic, may be ok for situations where no response is really required.

144. May be within a LAN which has high probability of transmission success11/21/2009 IT703D: Distributed Computing Debasis Das

146. Repeat until one call succeeds

147. This can work well only when only two nodes on which the processes work are involved.

148. Nested calls to different nodes will create problems

149. Orphan processes create problems

150. Let the orphans complete or exterminate them11/21/2009 IT703D: Distributed Computing Debasis Das

152. Ignore orphans

153. Attach call ids to calls, response returns the same id

154. Calls repeated has new id

155. Ignore response if it does not match the latest call id11/21/2009 IT703D: Distributed Computing Debasis Das

157. Implemented by the time-out based method

158. Takes the result of the first response without bothering if the result is from an orphan11/21/2009 IT703D: Distributed Computing Debasis Das

160. Programmers will have to ensure indempotency by creating appropriate interfaces

161. Whereas if the callee executes exactly once, indempotency can be guaranteed

162. Implementation uses timeouts, re-transmissions, call id, same id for repeats and reply cache for callee11/21/2009 IT703D: Distributed Computing Debasis Das

164. Request/Reply protocol

165. Request/reply/Acknowledge-Reply protocol11/21/2009 IT703D: Distributed Computing Debasis Das

167. Callee process does not send any reply

168. Also called an asynchronous protocol

169. Improves client and server performance

170. Example, remote X-window system11/21/2009 IT703D: Distributed Computing Debasis Das

172. RPC runtime does not take responsibility of retransmission

173. Some applications may still manage with this simple protocol and unreliable delivery

174. Time update for example

175. Receivers may request specific update after fixed period if connection lost for a long time11/21/2009 IT703D: Distributed Computing Debasis Das

177. A simple protocol is one I which the request, arguments, results can all fit into a single packet

178. Duration of call and interval between calls are small, smaller than transmission time of the packet

179. Implicit acknowledgements used

180. A reply is considered as an ack. To the earlier request

181. A request is considered as ack. For earlier reply

182. Timeout & retires used for failure handling11/21/2009 IT703D: Distributed Computing Debasis Das

184. Exactly once semantics on server possible with reply cache11/21/2009 IT703D: Distributed Computing Debasis Das

186. Request, reply followed by a reply ack. Completes the cycle

187. The reply ack. Itself may get lost

188. Use sequence numbers, the ack. carries the sequence number original request, as does the reply message11/21/2009 IT703D: Distributed Computing Debasis Das

190. RPCs involving arguments and/or results that cannot be fitted into one datagram11/21/2009 IT703D: Distributed Computing Debasis Das

192. Request message to server, periodic probes with the same request number, server generates ack.

193. If original request was not received, server says so and client is intimated of failure

194. Periodic generation of acknowledgments by server

195. Server generates ack. Prior to timeout. So the client knows the server is working on request.

196. If ack. or reply not received within timeout, failure of node or comm. is assumed and user informed.11/21/2009 IT703D: Distributed Computing Debasis Das

198. Use multiple physical RPC for each logical RPC. Each physical RPC handles data for one datagram

199. Inefficient due to fixed overheads with each physical RPC

200. Use multi datagram messages, handle missing or out of sequence datagrams as discussed in IPC 11/21/2009 IT703D: Distributed Computing Debasis Das

202. The process of associating a client & a server is binding

203. Server “exports” services it is willing to undertake

204. Client imports these services via RPC runtime11/21/2009 IT703D: Distributed Computing Debasis Das

206. How does the binding process locate the required server?

207. When is it proper to bind a client to a server?

208. Can the client change the binding during execution?

209. Can a client get bound to multiple server with similar service?11/21/2009 IT703D: Distributed Computing Debasis Das

211. Type includes a version number

212. When client is not bothered, it does not specify an instance

213. User decides semantics of interfaces, RPC package has no contribution

214. Example, file_server is a type, which may be provided by many instances11/21/2009 IT703D: Distributed Computing Debasis Das

216. Broadcasting

217. Client node broadcasts request to all

218. First response received from a server is given the client process

219. Binding Agent

220. Maintains a mapping table of interface name and location information

221. Servers register themselves, may deregister also

222. Identification information and a handle

223. Binding agent check periodically and deregister non responding servers11/21/2009 IT703D: Distributed Computing Debasis Das

224. Primitives used: register, deregister and lookup Server registers Client request for server Server information Client calls the server Binding Agent 2 1 3 Server Process Client Process 4 11/21/2009 IT703D: Distributed Computing Debasis Das Binding Process

226. Server address is compiled into code at compile time

227. Inflexible

228. Binding at link time

229. Server registers with binding agent

230. Client requests import of service, binding agent return handle

231. Handle retained by client for subsequent calls

232. Binding at call time

233. By indirect call11/21/2009 IT703D: Distributed Computing Debasis Das

234. Binding Agent Client passes server interface name name & arguments 2. Agent does RPC call to server 3.Server returns result 4. Agent returns result & handle 5.Client makes subsequent calls directly 1 3 2 4 Server Process Client Process 5 11/21/2009 IT703D: Distributed Computing Debasis Das Call Time Binding(Indirect call)

236. Client may want to change binding to another server if the earlier call failed

237. Server may want to change binding if it has to change to another node or upgrade to another version

238. When a server wants to change binding, the current state data may have to be transferred to the alternate server11/21/2009 IT703D: Distributed Computing Debasis Das

240. For example update of multiple copies of a file at different nodes

241. It is advantageous then to bind the client to multiple servers

242. Bind the client to all the servers that has copies of the file at different nodes11/21/2009 IT703D: Distributed Computing Debasis Das

244. Define exception conditions and raise exceptions

245. Can be handled by programming languages that have exception handling constructs

246. Alternatively(like in Unix) return a value -1 to indicate error and a global variable errno that contains a number indicating error type( a legitimate result could be -1)11/21/2009 IT703D: Distributed Computing Debasis Das

248. Considerations are

249. Is authentication of server required by the client?

250. Is authentication of server required when result is returned?

251. Is it ok if results and arguments of RPC are required or they could be shared by other users? (Other than caller & callee)11/21/2009 IT703D: Distributed Computing Debasis Das

253. Interactive programs may need several client actions before completing the original call

254. Broadcast RPC

255. Client request broadcast on network, several server may respond to the request

256. Batch Mode RPC

257. Send a batch of RPC calls. High call rate(50-100 remote calls per second) applications can use this11/21/2009 IT703D: Distributed Computing Debasis Das

259. Provide server with client’s handle

260. Making client process wait for callback RPC

261. Handling callback deadlock11/21/2009 IT703D: Distributed Computing Debasis Das

263. Program number sent to server as part of RPC request

264. Server then can initiate a RPC to client

265. A handle also may be sent so that further calls can be made directly11/21/2009 IT703D: Distributed Computing Debasis Das

267. To do that the client usually calls a svc-routine. The svc-routine waits for a callback and intimates client when a callback is received11/21/2009 IT703D: Distributed Computing Debasis Das

268. Waiting for reply From P2 Process P1 Process P2 Process P3 Waiting for Reply from P3 Waiting for reply From P1 11/21/2009 IT703D: Distributed Computing Debasis Das Handling Callback Deadlocks

270. Agent sends it to servers registered for that service

271. The broadcast call will find only the ones registered with the binding agent

272. 2. Get a binding for broadcast port

273. Broadcast a RPC

274. o/one or m out of n may be adopted

275. Servers with successful results respond, ones with error may remain silent11/21/2009 IT703D: Distributed Computing Debasis Das

277. Reduces overheads in sending individual calls

278. Clients with high call rates can use it, needs reliable transport

279. Entire queue requests are sent to server when

280. A predetermined interval elapses

281. Predetermined requests have been queued

282. Amount of batched data exceeds buffer size

283. A call is made which needs a reply, one must be able to distinguish between the batch calls and non-queueing RPC request11/21/2009 IT703D: Distributed Computing Debasis Das

285. Participation machines may have different data representation

286. Transport protocol

287. RPC mechanism should be transparent of network protocol

288. Control protocol

289. Should be transparent of network control protocols

290. Delay the decision. Binding agent sends the necessary data conversion and protocol routines to the client stub at binding time11/21/2009 IT703D: Distributed Computing Debasis Das

292. Needs communications of two types

293. Cross domain

294. Cross machine

295. LPRC is about cross domain communication, techniques used

296. Simple control transfer

297. Simple data transfer

298. Simple stubs

299. Design for concurrency11/21/2009 IT703D: Distributed Computing Debasis Das

301. When calling, client supplies an argument stack and the thread

302. Trap to kernel

303. Kernel validates caller, creates call linkage

304. Server starts executing the thread

305. When complete send the results and control back11/21/2009 IT703D: Distributed Computing Debasis Das

307. One copy: copy from client stack to the shared argument stack

308. In contrast regular RPC has to do 4 copies

309. Client stack to RPC message buffer

310. Message in client domain to kernel domain

311. Kernel domain to server domain

312. Server domain buffer to server stack11/21/2009 IT703D: Distributed Computing Debasis Das

314. Stub to stub implemented by the stubs themselves

315. Domain to domain implemented by the kernel11/21/2009 IT703D: Distributed Computing Debasis Das

317. Shared data structures minimized to reduce lock contention on domain transfer path

318. Reduce context switching latency by caching domains on idle processors

319. LRPC increases throughput by a factor of three!11/21/2009 IT703D: Distributed Computing Debasis Das

321. Serving multiple requests simultaneously

322. Reducing per-call workload of servers

323. Reply caching of indempotent remote procedures

324. Proper selection of time out values

325. Proper design of RPC protocol specification11/21/2009 IT703D: Distributed Computing Debasis Das

327. Addressing in the underlying protocol needs to be rich enough to provide correct routing of results

328. Early reply method

329. Split call, client sends parameters, server sends a tag

330. Client call again with tag when convenient to get result

331. Problem is server will have hold on to the result until required

332. Call buffering (via a call buffer server)

333. Sends call with parameters to call buffer server

334. Gets the result when it is ready

335. A variation of that method

336. Use a variable promise(can be blocked or ready) operation ready and claim on that11/21/2009 IT703D: Distributed Computing Debasis Das

338. Server may have to wait for a shared resource

339. Server calls a remote process that takes time or involves transmission delays

340. While waiting, server should be able to service other calls

341. Implement threads11/21/2009 IT703D: Distributed Computing Debasis Das

343. One way to reduce the load is to use stateless servers. Clients can keep track of the state. It is responsible for the processing anyway11/21/2009 IT703D: Distributed Computing Debasis Das

345. A cache helps11/21/2009 IT703D: Distributed Computing Debasis Das

347. Too large, reduction in throughput

348. Using retransmission time back off method is one solution11/21/2009 IT703D: Distributed Computing Debasis Das

350. Reduces encoding/decoding time

351. Reduces transfer time

352. One could use an existing general purpose protocol; in fact many RPC systems use TCP/IP and UDP/IP

353. General purpose protocols may be wasteful

354. For example RPC does not need the checksum field. It has to filled and takes computation time11/21/2009 IT703D: Distributed Computing Debasis Das