Spark Standalone集群隐藏的REST API
配置
在spark-defaults.conf中配置，
1
spark.master.rest.enabled true
API
提交
使用/v1/submissions/create去提交任务，
curl -X POST http://master-0.spark.node:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '
{
"appResource": "/home/spark/spark/examples/jars/spark-examples_2.12-3.3.0.jar",
"sparkProperties": {
"spark.master": "spark://master-0.spark.node:7077,master-1.spark.node:7077",
"spark.app.name": "Spark REST API - PI",
"spark.submit.deployMode": "client",
"spark.eventLog.enabled": "true",
"spark.eventLog.dir": "hdfs://namenode:9000/shared/spark-logs",
"spark.jars": "/home/spark/spark/examples/jars/spark-examples_2.12-3.3.0.jar"
},
"clientSparkVersion": "3.3.0",
"mainClass": "org.apache.spark.examples.SparkPi",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"
},
"action": "CreateSubmissionRequest",
"appArgs": [
"1000"
]
}
'
提交成功会返回如下结果，
1
2
3
4
5
6
7
{
"action" : "CreateSubmissionResponse",
"message" : "Driver successfully submitted as driver-20220718051203-0008",
"serverSparkVersion" : "3.3.0",
"submissionId" : "driver-20220718051203-0008",
"success" : true
}
查看
使用/v1/submissions/status去查看任务状态，在status后加入submissionId，
curl http://master-0.spark.node:6066/v1/submissions/status/driver-20220718051203-0008
返回如下，
1
2
3
4
5
6
7
8
9
{
"action" : "SubmissionStatusResponse",
"driverState" : "RUNNING",
"serverSparkVersion" : "3.3.0",
"submissionId" : "driver-20220718051203-0008",
"success" : true,
"workerHostPort" : "192.168.8.101:44977",
"workerId" : "worker-20220718031455-192.168.8.101-44977"
}
kill
使用/v1/submissions/kill去关闭任务，仍然使用submissionId为参数，
1
curl -X POST http://master-0.spark.node:6066/v1/submissions/kill/driver-20220718051203-0008
返回，
1
2
3
4
5
6
7
{
"action" : "KillSubmissionResponse",
"message" : "Kill request for driver-20220718051203-0008 submitted",
"serverSparkVersion" : "3.3.0",
"submissionId" : "driver-20220718051203-0008",
"success" : true
}