The following diagram presents a high-level overview of the components and services in the .
Platform Components and Services
- User Interface: Depending on the product, users interact through one of the available interfaces through a secure connection with the platform.
- : The following services interact with flow and dataset metadata in the , a proprietary database.
- Compiler: The compiler renders the steps created or selected by the user into a sequence of commands that can be executed against the specified target. See Wrangle Execution Layers.
- UDF: As needed, developers can create user-defined functions (UDFs), which can be made available to end-users through the UI. For more information, see User-Defined Functions.
- Scheduling: The Scheduling service enables management of asynchronous execution of jobs on a cron-based schedule. For more information, see Overview of Automator.
- Machine Learning: The executed recipes are analyzed for patterns, which provide more focused recommendations to end-users in the environment.
- Metadata Server: This server manages the read/write operations of the .
- Workload Management: This layer, which includes job execution and monitoring functions, is managed by Batch Job Runner, a proprietary service of the .
- Execution Environment: The can execute recipes through a variety of running environments.
- Storage: The unified storage layer provides brokerage services for read/write operations against HDFS, Hive, S3, and other clustered storage.