Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version next

...

  • User Interface: Depending on the product, users interact through one of the available interfaces through a secure connection with the platform. 
  • D s item
    itemServices
    :
    The following services interact with flow and dataset metadata in the
    D s item
    itemdatabase
    , a proprietary Postgres database.
    • Compiler: The
      D s lang
       compiler renders the steps created or selected by the user into a sequence of commands that can be executed against the specified target. See Wrangle Execution Layers.
    • UDF: As needed, developers can create user-defined functions (UDFs), which can be made available to end-users through the UI.
    • Scheduling: The Scheduling service enables management of asynchronous execution of jobs on a cron-based schedule. 
    • Machine Learning: The executed
      D s lang
       recipes are analyzed for patterns, which provide more focused recommendations to end-users in the environment. 
    • Metadata Server: This server manages the read/write operations of the
      D s item
      itemdatabase
      .
  • Workload Management: This layer, which includes job execution and monitoring functions, is managed by Batch Job Runner, a proprietary service of the
    D s platform
  • Execution Environment: The
    D s platform
     can execute
    D s lang
    recipes through a variety of running environments.
  • Storage: The unified storage layer provides brokerage services for read/write operations against HDFS, Hive, S3, and other clustered storage.