apache/dolphinscheduler

apache/dolphinscheduler repository overview

⁠NOTICE: This Docker repository is deprecated since 3.0.0, please use the following ones

Apache DolphinScheduler(incubator) is a distributed and extensible workflow scheduler platform with a powerful DAG visual interface. It is dedicated to solving complex job dependencies in data pipeline, making multiple types of jobs available out of the box

⁠Docker Deployment

please refer the official Docker deployment document: https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/docker-deployment.html⁠

⁠Dolphin Scheduler Official Website

⁠dolphinscheduler.apache.org⁠

⁠Features:

Associate the tasks according to the dependencies of the tasks in a DAG graph, which can visualize the running state of the task in real-time.
Support many task types: Shell, MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Sub_Process, Procedure, etc.
Support process scheduling, dependency scheduling, manual scheduling, manual pause/stop/recovery, support for failed retry/alarm, recovery from specified nodes, Kill task, etc.
Support the priority of process & task, task failover, and task timeout alarm or failure.
Support process global parameters and node custom parameter settings.
Support online upload/download of resource files, management, etc. Support online file creation and editing.
Support task log online viewing and scrolling, online download log, etc.
Implement cluster HA, decentralize Master cluster and Worker cluster through Zookeeper.
Support the viewing of Master/Worker CPU load, memory, and CPU usage metrics.
Support presenting tree or Gantt chart of workflow history as well as the statistics results of task & process status in each workflow.
Support backfilling data.
Support multi-tenant.
Support internationalization.
There are more waiting for partners to explore...

⁠What's in Dolphin Scheduler

Stability	Easy to use	Features	Scalability
Decentralized multi-master and multi-worker	Visualization process defines key information such as task status, task type, retry times, task running machine, visual variables, and so on at a glance.	Support pause, recover operation	Support custom task types
HA is supported by itself	All process definition operations are visualized, dragging tasks to draw DAGs, configuring data sources and resources. At the same time, for third-party systems, the API mode operation is provided.	Users on Dolphin Scheduler can achieve many-to-one or one-to-one mapping relationship through tenants and Hadoop users, which is very important for scheduling large data jobs.	The scheduler uses distributed scheduling, and the overall scheduling capability will increase linearly with the scale of the cluster. Master and Worker support dynamic online and offline.
Overload processing: Overload processing: By using the task queue mechanism, the number of schedulable tasks on a single machine can be flexibly configured. Machine jam can be avoided with high tolerance to numbers of tasks cached in task queue.	One-click deployment	Support traditional shell tasks, and big data platform task scheduling: MR, Spark, SQL (MySQL, PostgreSQL, hive, spark SQL), Python, Procedure, Sub_Process

⁠User Interface

⁠How to contribute

Welcome to participate in contributing, please refer to this website to find out more: [How to contribute⁠]

⁠Get Help

Submit an github issue
Subscribe to the mail list: https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html⁠
Twitter: https://twitter.com/dolphinschedule⁠
Slack: https://asf-dolphinscheduler.slack.com⁠

Tag summary

Recent tags

Content type

Image

Digest

sha256:d657965e5…

Size

221.8 MB

Last updated

over 3 years ago

docker pull apache/dolphinscheduler:2.0.7

This week's pulls

Pulls:

168

Last week

Learn more⁠