15 app-report RM allocates resources with itself for running AM 17 startContainer ( mr-container-spec, serialized-tokens ) tokens: deleg, shuffle, client, app 18 write( serialized-tokens ) Includes client-token, protected by OS permissions 19 exec ( job-dir, serialized-tokens-loc) Application Master (am) Provided the location of the token file via env 20 register ( app-token ) Map Task Execution 21 allocateResources (spec) : ContainerInfo(nm-token,container-token) 22 startContainer ( spec=map, tokens byte buffer A&C,D ) 23 exec () Application Container (ac) setuid? Is AC exec a setuid program? 24 extractFiles? Does AC extract the job files from HDFS? 25 exec () Task (t) Map Task 26 readFile ( split-file ) : blk-id, blk-loc, blk-access-tkn 27 readblk ( blk-id ) : split-blk-data 28 jobStatus () How does AC access AM? What does AC do now? Status Query 29 statusQuery () How does C know how to access AM? Reduce Task Execution 30 negotateResources () : ContainerInfo 31 launchContainer ( spec=reduce, tokens ) 32 exec () Application Container (ac) Are different containers used for Map and Reduce Tasks? 33 extractFiles? Does AC extract the job files from HDFS? 34 exec () Task (t) Reduce Task 35 writeFile ( reduce-file ) : blk-id, blk-loc, blk-access-tkn Need to identify where c-deleg-tkn came from and when it was refreshed. Should this be a token? 14 generateApplicationToken(): app-token For AM to talk to the RM. Oozie 6 createDelegationToken(): c-hs-deleg-token For completed job status 7 createShuffleSecret(): shuffle-secret Used to fetch data from NM 8 mkdir ( job-dir ) loop 9 writeFile () : blk-id, blk-loc, blk-access-tkn 10 writeBlock ( blk-id ): blk-data 11 serializeTokens( TODO ): serialized-tokens Job Submission 12 submitApplication ( job-dir, serialized-tokens ) Tokens also includes c-rm-deleg-token 13 generateClientSecret(): client-secret For the C to talk to AM. Test Diagram U NN DN JT TT 1 getNewJobId(): job-id 2 getDelegToken(): deleg-token 3 writeJobFiles( files, deleg-token ): void 4 writeJobFiles( files, deleg-token ): void 5 submitJob(): job-id 6 heartbeat() : work 7 refreshDelegToken( deleg-token ): void 8 extractJobFiles(): data 9 extractJobFiles(): data 10 new() T 11 exec() 12 status(): void Secure Map Reduce 2 Client (c) Resource Manager (rm) History Server (hs) Node Manager (nm) File System (fs) Name Node (nn) Data Node (dn) Install Bootstrap 1 heartbeat ( status ) : blk-tkn-secret 2 heartbeat () : nm-secret-key, container-secret-key Cycled daily? Job Definition 3 createJob(): job-id Used to identify job-dir 4 createDelegationToken(): c-nn-deleg-token 5 createDelegationToken(): c-rm-deleg-token Optional, e.g. sysl file type is optional.Text above Sample Client Client Server Server Test (A description of each one is shown on the. Input blackboxes in the format App Input sysl file that contains the system specifications. In the dialog box, select the blank template or one of the three starter diagrams.
0 Comments
Leave a Reply. |