{"id":26794,"date":"2019-04-01T09:00:28","date_gmt":"2019-04-01T16:00:28","guid":{"rendered":""},"modified":"2024-01-09T07:02:39","modified_gmt":"2024-01-09T15:02:39","slug":"how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/","title":{"rendered":"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ"},"content":{"rendered":"<p>We\u2019re delighted to release the Azure Toolkit for IntelliJ support for SQL Server Big Data Cluster Spark job development and submission. For first-time Spark developers, it can often be hard to get started and build their first application, with long and tedious development cycles in the integrated development environment (IDE). This toolkit empowers new users to get started with Spark in just a few minutes. Experienced Spark developers also find it faster and easier to iterate their development cycle.<\/p>\n<p>The toolkit extends IntelliJ support for the Spark job life cycle starting from creation, authoring, and debugging, through submission of jobs to SQL Server Big Data\u00a0Clusters. It\u00a0enables you to enjoy a native Scala and Java Spark application development experience and quickly start a project using built-in templates and sample code. The <a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/big-data-cluster\/spark-submit-job-intellij-tool-plugin?view=sqlallproducts-allversions&amp;branch=release-sqlseattle#link-sql-server-big-data-cluster\">integration with SQL Server Big Data Cluster<\/a> empowers you to quickly submit a job to the big data cluster as well as monitor its progress. The Spark console allows you to check schemas, preview data, and validate your code logic in a shell-like environment while you can develop Spark batch jobs within the same toolkit.<\/p>\n<p>The Azure Toolkit for IntelliJ offers the following capabilities:<\/p>\n<ul>\n<li>Connect to SQL Server Big Data Clusters and submit Spark jobs for execution.<\/li>\n<li>Create a Spark project using built-in templates, sample code, intelligent auto-creation of artifacts, and help locating assemblies.<\/li>\n<li>Integrate with Maven and SBT to automatically download library dependencies.<\/li>\n<li>Develop Spark applications with native authoring support (e.g. IntelliSense, auto format, and error checking) from IntelliJ.<\/li>\n<li>Locally run and debug Spark applications.<\/li>\n<li>Stop a running Spark application before its completion.<\/li>\n<li>Test and validate the Spark application locally or against a big data cluster.<\/li>\n<li>Integrate with Spark History Server to view job history, debug, and diagnose Spark jobs.<\/li>\n<\/ul>\n<h2>Highlights of Key Features<\/h2>\n<h3>Integration with SQL Server<\/h3>\n<p>This toolkit enables you to <a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/big-data-cluster\/spark-submit-job-intellij-tool-plugin?view=sqlallproducts-allversions&amp;branch=release-sqlseattle#link-sql-server-big-data-cluster\">connect<\/a> and submit Spark jobs to Azure <a href=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2018\/09\/25\/introducing-microsoft-sql-server-2019-big-data-clusters\/\">SQL Server Big Data Cluster<\/a><strong>, <\/strong>and navigate your SQL Server data and files.<\/p>\n<h3><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-26839 aligncenter\" src=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png\" alt=\"Link to SQL Server Big Data Cluster.\" width=\"791\" height=\"588\" srcset=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.webp 791w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-300x223.webp 300w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-768x571.webp 768w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-330x245.webp 330w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-400x297.webp 400w\" sizes=\"auto, (max-width: 791px) 100vw, 791px\" \/><\/h3>\n<h3>Create, author, submit, and stop a Spark application<\/h3>\n<p>To create a new Spark application using <a href=\"https:\/\/plugins.jetbrains.com\/plugin\/8053-azure-toolkit-for-intellij\">Azure toolkit for IntelliJ<\/a>, you can leverage the template to create and author a Spark job with sample code and built-in integrations with Maven and SBT.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-26842 aligncenter\" src=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-2.png\" alt=\"Create New Spark Project Template.\" width=\"795\" height=\"611\" srcset=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-2.webp 701w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-2-300x231.webp 300w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-2-325x250.webp 325w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-2-330x254.webp 330w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-2-400x308.webp 400w\" sizes=\"auto, (max-width: 795px) 100vw, 795px\" \/><\/p>\n<p>Next you can submit a Spark application and identify the job submission target cluster, job parameters, and add references in the\u00a0configuration dialog.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-large wp-image-26845 aligncenter\" src=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-3-1024x755.png\" alt=\"Run Spark Job Configuration.\" width=\"1024\" height=\"755\" \/><\/p>\n<p>Stop Spark applications from IntelliJ UI:\u00a0After your Spark job submission, you can stop running a Spark application in IntelliJ before its completion.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-large wp-image-26848 aligncenter\" src=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-4-1024x427.png\" alt=\"Stop Spark Project.\" width=\"1024\" height=\"427\" \/><\/p>\n<h2>Spark Console<\/h2>\n<p>This component facilitates Spark job authoring and enables you to run code interactively in a shell-like environment including <strong>Spark Local Console<\/strong> and <strong>Spark Livy Interactive Sessions<\/strong>.<\/p>\n<p>The Spark local console allows you to run your code interactively and validate your code logic locally. You can also check your programming variables and perform other scripting operations locally before submitting to the cluster. The Spark Livy interactive session establishes an interactive communication channel with your cluster so you can check on file schemas, preview data, and run ad-hoc queries while you\u2019re programming your Spark job.<\/p>\n<p>The Spark console has a <strong>Language Service <\/strong>built-in for Scala programming. You can leverage the language service features such as IntelliSense and autocomplete, to look up a Spark object (i.e., Spark context and Spark session) properties, query hive metadata, and check on function signatures. The <strong>Send Selection to Spark Console (Ctrl + Shift + S) <\/strong>feature simplifies the user experience for accessing the Spark console. You can send a highlighted single line of code or a block of code to the console from your main Scala project. This feature enables you to switch smoothly between contexts from coding and validation or testing code in the Spark console.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-26851 aligncenter\" src=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-5-1024x613.png\" alt=\"Send Code to Spark Console.\" width=\"1024\" height=\"613\" \/><\/p>\n<h2>Spark job debug &amp; diagnosis<\/h2>\n<p>In this release, Microsoft brings many of its learnings from running and debugging millions of its own big data jobs to the open source world of Apache Spark<em><strong><sup>TM<\/sup><\/strong><\/em>.\u00a0 Azure Toolkit integrates with <a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/big-data-cluster\/spark-history-server?view=sqlallproducts-allversions&amp;branch=release-sqlseattle\">the enhanced SQL Server Big Data Cluster Spark history server<\/a> with interactive visualization of job graphs, data flows, and job diagnosis. The new features assist SQL Server Big Data Spark developers to:<\/p>\n<ul>\n<li>View and play back Spark application\/job graph pivoting on execution progress, or data read and written.<\/li>\n<li>Identify the Spark application execution dependencies among stages and jobs for performance tuning.<\/li>\n<li>Simplify performance diagnosis through data read\/write heatmaps, and identify performance outliers as well as bottleneck stages and jobs.<\/li>\n<li>View Spark jobs and stage data input\/output size and time duration for performance optimization.<\/li>\n<li>Detect and view data or time skew for stage tasks.<\/li>\n<li>View executor allocation and utilization.<\/li>\n<li>Preview and download Spark job input and output data, as well as view Spark job table operations.<\/li>\n<\/ul>\n<h2 style=\"font-family: 'Segoe UI', SegoeUI, Tahoma, Arial, sans-serif\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-26857 aligncenter\" src=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-7.png\" alt=\"View Spark Job Skew.\" width=\"893\" height=\"618\" srcset=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-7.webp 650w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-7-300x208.webp 300w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-7-330x228.webp 330w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-7-400x277.webp 400w\" sizes=\"auto, (max-width: 893px) 100vw, 893px\" \/><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-26854 aligncenter\" src=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-6-1024x518.png\" alt=\"View Spark Job Graph.\" width=\"1024\" height=\"518\" \/><\/p>\n<h2>How to install<\/h2>\n<p>Please install IntelliJ <strong>2018.3<\/strong> version. You can get the latest details by going to the IntelliJ repository and searching <strong>Spark<\/strong> or <strong>Azure Toolkit for IntelliJ<\/strong>.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-26860 aligncenter\" src=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-8.png\" alt=\"Install Azure Toolkit for IntelliJ.\" width=\"841\" height=\"703\" srcset=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-8.webp 841w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-8-300x251.webp 300w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-8-768x642.webp 768w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-8-299x250.webp 299w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-8-330x276.webp 330w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-8-800x669.webp 800w, https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257-8-400x334.webp 400w\" sizes=\"auto, (max-width: 841px) 100vw, 841px\" \/><\/p>\n<p><strong>Read more: <\/strong><\/p>\n<ul>\n<li><a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/big-data-cluster\/big-data-cluster-overview?view=sqlallproducts-allversions\">What are SQL Server 2019 big data clusters<\/a>?<\/li>\n<li><a href=\"https:\/\/docs.microsoft.com\/en-us\/java\/azure\/intellij\/azure-toolkit-for-intellij-installation?view=azure-java-stable\">Install Azure Toolkit for IntelliJ<\/a><\/li>\n<li><a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/big-data-cluster\/spark-submit-job-intellij-tool-plugin?view=sqlallproducts-allversions&amp;branch=release-sqlseattle\">Submit Spark jobs on SQL Server big data clusters in IntelliJ <\/a><\/li>\n<li><a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/big-data-cluster\/spark-history-server?view=sqlallproducts-allversions&amp;branch=release-sqlseattle\">Debug and Diagnose Spark Applications on SQL Server big data clusters<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>We\u2019re delighted to release the Azure Toolkit for IntelliJ support for SQL Server Big Data Cluster Spark job development and submission. For first-time Spark developers, it can often be hard to get started and build their first application, with long and tedious development cycles in the integrated development environment (IDE).<\/p>\n","protected":false},"author":5562,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","footnotes":""},"post_tag":[],"product":[],"content-type":[2424],"topic":[2451,2466],"coauthors":[2605],"class_list":["post-26794","post","type-post","status-publish","format-standard","hentry","content-type-best-practices","topic-big-data","topic-developer"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ - Microsoft SQL Server Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ - Microsoft SQL Server Blog\" \/>\n<meta property=\"og:description\" content=\"We\u2019re delighted to release the Azure Toolkit for IntelliJ support for SQL Server Big Data Cluster Spark job development and submission. For first-time Spark developers, it can often be hard to get started and build their first application, with long and tedious development cycles in the integrated development environment (IDE).\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft SQL Server Blog\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/sqlserver\" \/>\n<meta property=\"article:published_time\" content=\"2019-04-01T16:00:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-01-09T15:02:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png\" \/>\n<meta name=\"author\" content=\"Jenny Jiang\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@SQLServer\" \/>\n<meta name=\"twitter:site\" content=\"@SQLServer\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jenny Jiang\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 min read\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/\"},\"author\":[{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/author\/jenny-jiang\/\",\"@type\":\"Person\",\"@name\":\"Jenny Jiang\"}],\"headline\":\"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ\",\"datePublished\":\"2019-04-01T16:00:28+00:00\",\"dateModified\":\"2024-01-09T15:02:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/\"},\"wordCount\":861,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/\",\"name\":\"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ - Microsoft SQL Server Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png\",\"datePublished\":\"2019-04-01T16:00:28+00:00\",\"dateModified\":\"2024-01-09T15:02:39+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#primaryimage\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png\",\"contentUrl\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#website\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/\",\"name\":\"Microsoft SQL Server Blog\",\"description\":\"Official News from Microsoft\u2019s Information Platform\",\"publisher\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#organization\",\"name\":\"Microsoft SQL Server Blog\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png\",\"contentUrl\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png\",\"width\":259,\"height\":194,\"caption\":\"Microsoft SQL Server Blog\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"http:\/\/www.facebook.com\/sqlserver\",\"https:\/\/x.com\/SQLServer\",\"https:\/\/www.youtube.com\/user\/MSCloudOS\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ - Microsoft SQL Server Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/","og_locale":"en_US","og_type":"article","og_title":"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ - Microsoft SQL Server Blog","og_description":"We\u2019re delighted to release the Azure Toolkit for IntelliJ support for SQL Server Big Data Cluster Spark job development and submission. For first-time Spark developers, it can often be hard to get started and build their first application, with long and tedious development cycles in the integrated development environment (IDE).","og_url":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/","og_site_name":"Microsoft SQL Server Blog","article_publisher":"http:\/\/www.facebook.com\/sqlserver","article_published_time":"2019-04-01T16:00:28+00:00","article_modified_time":"2024-01-09T15:02:39+00:00","og_image":[{"url":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png","type":"","width":"","height":""}],"author":"Jenny Jiang","twitter_card":"summary_large_image","twitter_creator":"@SQLServer","twitter_site":"@SQLServer","twitter_misc":{"Written by":"Jenny Jiang","Est. reading time":"3 min read"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#article","isPartOf":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/"},"author":[{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/author\/jenny-jiang\/","@type":"Person","@name":"Jenny Jiang"}],"headline":"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ","datePublished":"2019-04-01T16:00:28+00:00","dateModified":"2024-01-09T15:02:39+00:00","mainEntityOfPage":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/"},"wordCount":861,"commentCount":0,"publisher":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#organization"},"image":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#primaryimage"},"thumbnailUrl":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/","url":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/","name":"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ - Microsoft SQL Server Blog","isPartOf":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#primaryimage"},"image":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#primaryimage"},"thumbnailUrl":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png","datePublished":"2019-04-01T16:00:28+00:00","dateModified":"2024-01-09T15:02:39+00:00","breadcrumb":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#primaryimage","url":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png","contentUrl":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/03\/257.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/2019\/04\/01\/how-to-develop-and-submit-spark-jobs-to-sql-server-big-data-clusters-in-intellij\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/"},{"@type":"ListItem","position":2,"name":"How to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ"}]},{"@type":"WebSite","@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#website","url":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/","name":"Microsoft SQL Server Blog","description":"Official News from Microsoft\u2019s Information Platform","publisher":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#organization","name":"Microsoft SQL Server Blog","url":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png","contentUrl":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png","width":259,"height":194,"caption":"Microsoft SQL Server Blog"},"image":{"@id":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/#\/schema\/logo\/image\/"},"sameAs":["http:\/\/www.facebook.com\/sqlserver","https:\/\/x.com\/SQLServer","https:\/\/www.youtube.com\/user\/MSCloudOS"]}]}},"msxcm_display_generated_audio":false,"msxcm_animated_featured_image":null,"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/posts\/26794","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/users\/5562"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/comments?post=26794"}],"version-history":[{"count":0,"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/posts\/26794\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/media?parent=26794"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/post_tag?post=26794"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/product?post=26794"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/content-type?post=26794"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/topic?post=26794"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/sql-server\/blog\/wp-json\/wp\/v2\/coauthors?post=26794"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}