[roelhogervorst]

This PR adds a spark_R operator which will allow you to schedule R, and sparkR jobs on a dataproc cluster. The functionality to run that kind of job is already in dataproc, but for some reason there is no operator in Airflow.

---

Issue link: WILL BE INSERTED BY boring-cyborg

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://apache-airflow-slack.herokuapp.com/

[turbaszek]

Do we need custom operator for R jobs? Can't we use the DataprocSubmitJobOperator?

[turbaszek]

The DataprocJobBaseOperator as well as other XJobOperators will be deleted in future:

        # TODO: Remove one day
        warnings.warn(
            "The `{cls}` operator is deprecated, please use `DataprocSubmitJobOperator` instead. You can use"
            " `generate_job` method of `{cls}` to generate dictionary representing your job"
            " and use it with the new operator.".format(cls=type(self).__name__),
            DeprecationWarning,
            stacklevel=1
        )

[roelhogervorst]

Is there an example of how to use the generic DataprocJobBaseOperator for all the jobs yet, @turbaszek?

[mik-laj]

https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_dataproc.py#L141-L166

[roelhogervorst]

oh that is way easier, I'll create a new PR

[turbaszek]

Hi @roelhogervorst any progress? I am happy to help with running system tests for the new SparkR job :)

[roelhogervorst]

I'm so sorry @turbaszek , I completely missed your message. Are you saying we can submit jobs without specifying the type of job?