API tools faq
paste
Advertisement
PeachLemonade

Airflow_bash_dag

PeachLemonade
Mar 14th, 2024
84
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Apache Log 24.12 KB | None | 0 0
raw download clone embed print report
  1. fhmfl12cg9vubrh80mui.auto.internal
  2. *** Found local files:
  3. ***   * /lessons/logs/dag_id=example_bash_dag/run_id=manual__2024-03-14T09:06:11.278350+00:00/task_id=print_date/attempt=1.log
  4. [2024-03-14, 09:06:11 UTC] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: example_bash_dag.print_date manual__2024-03-14T09:06:11.278350+00:00 [queued]>
  5. [2024-03-14, 09:06:11 UTC] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: example_bash_dag.print_date manual__2024-03-14T09:06:11.278350+00:00 [queued]>
  6. [2024-03-14, 09:06:11 UTC] {taskinstance.py:1308} INFO - Starting attempt 1 of 4
  7. [2024-03-14, 09:06:11 UTC] {taskinstance.py:1327} INFO - Executing <Task(BashOperator): print_date> on 2024-03-14 09:06:11.278350+00:00
  8. [2024-03-14, 09:06:11 UTC] {standard_task_runner.py:57} INFO - Started process 8157 to run task
  9. [2024-03-14, 09:06:11 UTC] {standard_task_runner.py:84} INFO - Running: ['airflow', 'tasks', 'run', 'example_bash_dag', 'print_date', 'manual__2024-03-14T09:06:11.278350+00:00', '--job-id', '53', '--raw', '--subdir', 'DAGS_FOLDER/bash_dag.py', '--cfg-path', '/tmp/tmphqgap3cu']
  10. [2024-03-14, 09:06:11 UTC] {standard_task_runner.py:85} INFO - Job 53: Subtask print_date
  11. [2024-03-14, 09:06:11 UTC] {task_command.py:410} INFO - Running <TaskInstance: example_bash_dag.print_date manual__2024-03-14T09:06:11.278350+00:00 [running]> on host fhmfl12cg9vubrh80mui.auto.internal
  12. [2024-03-14, 09:06:12 UTC] {taskinstance.py:1545} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='example_bash_dag' AIRFLOW_CTX_TASK_ID='print_date' AIRFLOW_CTX_EXECUTION_DATE='2024-03-14T09:06:11.278350+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-03-14T09:06:11.278350+00:00'
  13. [2024-03-14, 09:06:12 UTC] {subprocess.py:63} INFO - Tmp dir root location: /tmp
  14. [2024-03-14, 09:06:12 UTC] {subprocess.py:75} INFO - Running command: ['/usr/bin/bash', '-c', "/usr/lib/spark/bin/spark-submit --master yarn --deploy-mode cluster /lessons/partition.py '2022-05-31' '/user/master/data/events' '/user/kotlyarovb/data/events'"]
  15. [2024-03-14, 09:06:12 UTC] {subprocess.py:86} INFO - Output:
  16. [2024-03-14, 09:06:13 UTC] {subprocess.py:93} INFO - SLF4J: Class path contains multiple SLF4J bindings.
  17. [2024-03-14, 09:06:13 UTC] {subprocess.py:93} INFO - SLF4J: Found binding in [jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  18. [2024-03-14, 09:06:13 UTC] {subprocess.py:93} INFO - SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  19. [2024-03-14, 09:06:13 UTC] {subprocess.py:93} INFO - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  20. [2024-03-14, 09:06:13 UTC] {subprocess.py:93} INFO - SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  21. [2024-03-14, 09:06:13 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:13,492 WARN util.Utils: Your hostname, fhmfl12cg9vubrh80mui resolves to a loopback address: 127.0.1.1; using 172.16.0.24 instead (on interface eth0)
  22. [2024-03-14, 09:06:13 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:13,492 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
  23. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,154 INFO client.RMProxy: Connecting to ResourceManager at rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net/172.16.0.14:8032
  24. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,385 INFO client.AHSProxy: Connecting to Application History server at rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net/172.16.0.14:10200
  25. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,463 INFO yarn.Client: Requesting a new application from cluster with 5 NodeManagers
  26. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,917 INFO conf.Configuration: resource-types.xml not found
  27. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,918 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
  28. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,939 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (37888 MB per container)
  29. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,940 INFO yarn.Client: Will allocate AM container, with 1152 MB memory including 384 MB overhead
  30. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,940 INFO yarn.Client: Setting up container launch context for our AM
  31. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,945 INFO yarn.Client: Setting up the launch environment for our AM container
  32. [2024-03-14, 09:06:14 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:14,963 INFO yarn.Client: Preparing resources for our AM container
  33. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,025 INFO yarn.Client: Uploading resource file:/lessons/partition.py -> hdfs://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net/user/kotlyarovb/.sparkStaging/application_1692104774102_37396/partition.py
  34. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,241 INFO yarn.Client: Uploading resource file:/usr/lib/spark/python/lib/pyspark.zip -> hdfs://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net/user/kotlyarovb/.sparkStaging/application_1692104774102_37396/pyspark.zip
  35. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,684 INFO yarn.Client: Uploading resource file:/usr/lib/spark/python/lib/py4j-0.10.9-src.zip -> hdfs://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net/user/kotlyarovb/.sparkStaging/application_1692104774102_37396/py4j-0.10.9-src.zip
  36. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,925 INFO yarn.Client: Uploading resource file:/tmp/spark-0abfefbd-eedc-4409-946b-c04a6eca286d/__spark_conf__182705932882035646.zip -> hdfs://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net/user/kotlyarovb/.sparkStaging/application_1692104774102_37396/__spark_conf__.zip
  37. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,982 INFO spark.SecurityManager: Changing view acls to: kotlyarovb
  38. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,983 INFO spark.SecurityManager: Changing modify acls to: kotlyarovb
  39. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,983 INFO spark.SecurityManager: Changing view acls groups to:
  40. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,984 INFO spark.SecurityManager: Changing modify acls groups to:
  41. [2024-03-14, 09:06:15 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:15,984 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(kotlyarovb); groups with view permissions: Set(); users  with modify permissions: Set(kotlyarovb); groups with modify permissions: Set()
  42. [2024-03-14, 09:06:16 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:16,050 INFO yarn.Client: Submitting application application_1692104774102_37396 to ResourceManager
  43. [2024-03-14, 09:06:16 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:16,085 INFO impl.YarnClientImpl: Submitted application application_1692104774102_37396
  44. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:17,089 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  45. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:17,092 INFO yarn.Client:
  46. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     client token: N/A
  47. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     diagnostics: AM container is launched, waiting for AM container to Register with RM
  48. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     ApplicationMaster host: N/A
  49. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     ApplicationMaster RPC port: -1
  50. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     queue: root.kotlyarovb
  51. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     start time: 1710407176065
  52. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     final status: UNDEFINED
  53. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     tracking URL: http://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net:8088/proxy/application_1692104774102_37396/
  54. [2024-03-14, 09:06:17 UTC] {subprocess.py:93} INFO -     user: kotlyarovb
  55. [2024-03-14, 09:06:18 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:18,094 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  56. [2024-03-14, 09:06:19 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:19,096 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  57. [2024-03-14, 09:06:20 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:20,099 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  58. [2024-03-14, 09:06:21 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:21,101 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  59. [2024-03-14, 09:06:22 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:22,103 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  60. [2024-03-14, 09:06:23 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:23,105 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  61. [2024-03-14, 09:06:24 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:24,107 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  62. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:25,109 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  63. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:25,110 INFO yarn.Client:
  64. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     client token: N/A
  65. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     diagnostics: N/A
  66. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     ApplicationMaster host: rc1a-dataproc-d-knx1iehn79bn7eve.mdb.yandexcloud.net
  67. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     ApplicationMaster RPC port: 45359
  68. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     queue: root.kotlyarovb
  69. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     start time: 1710407176065
  70. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     final status: UNDEFINED
  71. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     tracking URL: http://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net:8088/proxy/application_1692104774102_37396/
  72. [2024-03-14, 09:06:25 UTC] {subprocess.py:93} INFO -     user: kotlyarovb
  73. [2024-03-14, 09:06:26 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:26,112 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  74. [2024-03-14, 09:06:27 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:27,114 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  75. [2024-03-14, 09:06:28 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:28,116 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  76. [2024-03-14, 09:06:29 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:29,118 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  77. [2024-03-14, 09:06:30 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:30,120 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  78. [2024-03-14, 09:06:31 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:31,124 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  79. [2024-03-14, 09:06:32 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:32,126 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  80. [2024-03-14, 09:06:33 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:33,128 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  81. [2024-03-14, 09:06:34 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:34,130 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  82. [2024-03-14, 09:06:35 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:35,132 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  83. [2024-03-14, 09:06:36 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:36,134 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  84. [2024-03-14, 09:06:37 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:37,136 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  85. [2024-03-14, 09:06:38 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:38,138 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  86. [2024-03-14, 09:06:39 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:39,140 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  87. [2024-03-14, 09:06:40 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:40,142 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  88. [2024-03-14, 09:06:41 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:41,144 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  89. [2024-03-14, 09:06:42 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:42,146 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  90. [2024-03-14, 09:06:43 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:43,148 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  91. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:44,151 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  92. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:44,151 INFO yarn.Client:
  93. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     client token: N/A
  94. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     diagnostics: N/A
  95. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     ApplicationMaster host: N/A
  96. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     ApplicationMaster RPC port: -1
  97. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     queue: root.kotlyarovb
  98. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     start time: 1710407176065
  99. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     final status: UNDEFINED
  100. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     tracking URL: http://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net:8088/proxy/application_1692104774102_37396/
  101. [2024-03-14, 09:06:44 UTC] {subprocess.py:93} INFO -     user: kotlyarovb
  102. [2024-03-14, 09:06:45 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:45,153 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  103. [2024-03-14, 09:06:46 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:46,155 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  104. [2024-03-14, 09:06:47 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:47,157 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  105. [2024-03-14, 09:06:48 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:48,160 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  106. [2024-03-14, 09:06:49 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:49,162 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  107. [2024-03-14, 09:06:50 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:50,164 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  108. [2024-03-14, 09:06:51 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:51,166 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  109. [2024-03-14, 09:06:52 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:52,168 INFO yarn.Client: Application report for application_1692104774102_37396 (state: ACCEPTED)
  110. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:53,170 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  111. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:53,170 INFO yarn.Client:
  112. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     client token: N/A
  113. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     diagnostics: N/A
  114. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     ApplicationMaster host: rc1a-dataproc-d-jysa8rey4rqmgffj.mdb.yandexcloud.net
  115. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     ApplicationMaster RPC port: 45199
  116. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     queue: root.kotlyarovb
  117. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     start time: 1710407176065
  118. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     final status: UNDEFINED
  119. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     tracking URL: http://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net:8088/proxy/application_1692104774102_37396/
  120. [2024-03-14, 09:06:53 UTC] {subprocess.py:93} INFO -     user: kotlyarovb
  121. [2024-03-14, 09:06:54 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:54,172 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  122. [2024-03-14, 09:06:55 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:55,174 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  123. [2024-03-14, 09:06:56 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:56,176 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  124. [2024-03-14, 09:06:57 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:57,178 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  125. [2024-03-14, 09:06:58 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:58,180 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  126. [2024-03-14, 09:06:59 UTC] {subprocess.py:93} INFO - 2024-03-14 09:06:59,182 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  127. [2024-03-14, 09:07:00 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:00,185 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  128. [2024-03-14, 09:07:01 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:01,187 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  129. [2024-03-14, 09:07:02 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:02,188 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  130. [2024-03-14, 09:07:03 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:03,190 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  131. [2024-03-14, 09:07:04 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:04,192 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  132. [2024-03-14, 09:07:05 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:05,195 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  133. [2024-03-14, 09:07:06 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:06,197 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  134. [2024-03-14, 09:07:07 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:07,199 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  135. [2024-03-14, 09:07:08 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:08,201 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  136. [2024-03-14, 09:07:09 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:09,203 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  137. [2024-03-14, 09:07:10 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:10,205 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  138. [2024-03-14, 09:07:11 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:11,207 INFO yarn.Client: Application report for application_1692104774102_37396 (state: RUNNING)
  139. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:12,209 INFO yarn.Client: Application report for application_1692104774102_37396 (state: FINISHED)
  140. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:12,209 INFO yarn.Client:
  141. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     client token: N/A
  142. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     diagnostics: User application exited with status 1
  143. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     ApplicationMaster host: rc1a-dataproc-d-jysa8rey4rqmgffj.mdb.yandexcloud.net
  144. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     ApplicationMaster RPC port: 45199
  145. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     queue: root.kotlyarovb
  146. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     start time: 1710407176065
  147. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     final status: FAILED
  148. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     tracking URL: http://rc1a-dataproc-m-dg5lgqqm7jju58f9.mdb.yandexcloud.net:8088/proxy/application_1692104774102_37396/
  149. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -     user: kotlyarovb
  150. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:12,216 ERROR yarn.Client: Application diagnostics message: User application exited with status 1
  151. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO - Exception in thread "main" org.apache.spark.SparkException: Application application_1692104774102_37396 finished with failed status
  152. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1192)
  153. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1583)
  154. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
  155. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
  156. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
  157. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
  158. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
  159. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
  160. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO -    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  161. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:12,219 INFO util.ShutdownHookManager: Shutdown hook called
  162. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:12,220 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-eae4916e-649b-4836-8f42-b5e2bce27bee
  163. [2024-03-14, 09:07:12 UTC] {subprocess.py:93} INFO - 2024-03-14 09:07:12,223 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0abfefbd-eedc-4409-946b-c04a6eca286d
  164. [2024-03-14, 09:07:12 UTC] {subprocess.py:97} INFO - Command exited with return code 1
  165. [2024-03-14, 09:07:12 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
  166. Traceback (most recent call last):
  167.   File "/usr/local/lib/python3.8/dist-packages/airflow/operators/bash.py", line 210, in execute
  168.     raise AirflowException(
  169. airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
  170. [2024-03-14, 09:07:12 UTC] {taskinstance.py:1345} INFO - Marking task as UP_FOR_RETRY. dag_id=example_bash_dag, task_id=print_date, execution_date=20240314T090611, start_date=20240314T090611, end_date=20240314T090712
  171. [2024-03-14, 09:07:12 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 53 for task print_date (Bash command failed. The command returned a non-zero exit code 1.; 8157)
  172. [2024-03-14, 09:07:12 UTC] {local_task_job_runner.py:225} INFO - Task exited with return code 1
  173. [2024-03-14, 09:07:12 UTC] {taskinstance.py:2653} INFO - 0 downstream tasks scheduled from follow-on schedule check
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!